How to Discourage Search Engines from Indexing Your WordPress Site

Are you trying to discourage search engines from indexing your WordPress site?
While most website owners want search engines to index their sites, there are cases when you might want to keep your WordPress site private.
This guide covers how to prevent search engines from indexing your site using different methods, from simple WordPress settings to more advanced techniques.
Why Prevent Search Engines from Indexing Your WordPress Site?
Most website owners consider search engines like Google or bing to be their best friend. After all, search engines are often the most significant source of traffic.
So, why would anyone want to disable search engine indexing for a WordPress site?
There are actually a few good reasons to do such thing. For exemple, this might the case if you’re still building your website on a publicly accessible domain. In that case, you might not want Google or any other search engine to index your “Under Construction” or “Coming Soon” pages. Other reasons could include:
- Creating an invite-only or test website.
- Transferring a domain that still has outdated links pointing to it.
- Managing a private blog or portfolio that you don’t want to appear in search engine results.
Remember, even if you think no one knows about your site yet, there are various ways it could accidentally appear in search engine results. Old links from a previous domain owner, random domain directories, or even automated crawls can make your site more public than you’d like.
Warning
Do not forget to untick the “Discourage Search Engines from Indexing Your WordPress Site” box when you put your website in production. You can also monitor your websites with WP Umbrella to be alerted when Search engine visibility is not activated in WordPress.
Ready to boost your productivity, impress your clients and grow your WordPress agency?
Install WP Umbrella on your websites in a minute and discover a new way to manage multiple WordPress sites.
Get Started for free
How Do Search Engines Index Websites?
Before we get into how discourage search engines, let’s briefly discuss how they work. Search engines use bots or spiders to crawl websites and index their content.
Once indexed, your site starts appearing in search engine results pages (SERPs).
Discouraging Search Engines From Indexing Your WordPress Site
There are two ways you can discourage search engines from indexing your WordPress site.
Method 1: Using WordPress Settings to Discourage Search Engine Indexing

This method lets site owners prevent search engines from crawling or indexing their websites by using an inbuilt feature of WordPress.
- Click on ‘Settings’ in the WordPress admin area.
- Select ‘Reading’ from the drop-down menu below ‘Settings’.
- To confirm your changes, click on the ‘save’ button.
And that’s it! However, keep in mind that it’s up to search engines to honor this request.
Method 2: Manually Edit the Robots.txt File
For those who prefer to take matters into their own hands, you can manually edit the robots.txt file to prevent search engines from crawling your site.
Connect to your website using your FTP client and go to the WordPress root directory folder and find the Robots.txt file. Create a new blank file if you can’t find it.
Add this code at the top of the file:
User-agent:*
Disallow: /
This way, search engine robots should not index your website.
By using the code above, search engines will not index your entire site. In order to disallow a specific page, you must provide the subdirectory and slug.
Disallow: /private/my-article
Method 3: Server-Level Protection (The Secure Option)

You can instruct search engines not to index your website using the default WordPress visibility settings or the robots.txt file.
Even so, search engines may still crawl and index your website’s pages, files, or images.
The only way to ensure that search engines will not index or crawl your website, at all, is to password protect it on the server level.
If you want to be absolutely sure that your site stays private, you could password protect your WordPress site on the server level. Plugins like Password Protected can easily do this for you.
Install and activate the plugin, then navigate to Settings -> Password Protected and customize the settings according to your needs.
What If You Change Your Mind? How to Enable Indexing Again
If you decide to make your site publicly accessible:
- Uncheck “Discourage Search Engines from Indexing This Site”
Go to Settings > Reading and uncheck the box in the Search Engine Visibility section.
- Edit Robots.txt
Update robots.txt to remove the Disallow rules, or delete the file entirely.
- Submit to Google Search Console
Submit again your site to Google Search Console. This helps search engines begin crawling your content immediately.
Conclusion
Discouraging search engines from indexing your WordPress site is a strategy you might adopt if your site is under construction, you’re running a private blog, or for other privacy-related reasons.
WordPress provides built-in settings to request that search engines ignore your site, or you can manually edit the robots.txt file for more specific control.
If you really want to make sure that no one can access your website, you can enable a server-level protection with basic HTTP authentication.
For ongoing monitoring and sites maintenance, WP Umbrella offers everything you need to manage your sites from a single place.