How to Discourage Search Engines from Indexing Your WordPress Site

Are you trying to discourage search engines from indexing your WordPress site?

While most website owners want search engines to index their sites, there are cases when you might want to keep your WordPress site private.

This guide covers how to prevent search engines from indexing your site using different methods, from simple WordPress settings to more advanced techniques.

Why Prevent Search Engines from Indexing Your WordPress Site?

Most website owners consider search engines like Google or bing to be their best friend. After all, search engines are often the most significant source of traffic.

So, why would anyone want to disable search engine indexing for a WordPress site?

There are actually a few good reasons to do such thing. For exemple, this might the case if you’re still building your website on a publicly accessible domain. In that case, you might not want Google or any other search engine to index your “Under Construction” or “Coming Soon” pages. Other reasons could include:

1. Creating an invite-only or test website.
2. Transferring a domain that still has outdated links pointing to it.
3. Managing a private blog or portfolio that you don’t want to appear in search engine results.

Remember, even if you think no one knows about your site yet, there are various ways it could accidentally appear in search engine results. Old links from a previous domain owner, random domain directories, or even automated crawls can make your site more public than you’d like.

How Do Search Engines Index Websites?

Before we get into how discourage search engines, let’s briefly discuss how they work. Search engines use bots or spiders to crawl websites and index their content.

Once indexed, your site starts appearing in search engine results pages (SERPs).

Discouraging Search Engines From Indexing Your WordPress Site

There are two ways you can discourage search engines from indexing your WordPress site.

Method 1: Using WordPress Settings to Discourage Search Engine Indexing

This method lets site owners prevent search engines from crawling or indexing their websites by using an inbuilt feature of WordPress.

1. Click on ‘Settings’ in the WordPress admin area. 
2. Select ‘Reading’ from the drop-down menu below ‘Settings’.
3. To confirm your changes, click on the ‘save’ button.

And that’s it! However, keep in mind that it’s up to search engines to honor this request.

Method 2: Manually Edit the Robots.txt File

For those who prefer to take matters into their own hands, you can manually edit the robots.txt file to prevent search engines from crawling your site.

Connect to your website using your FTP client and go to the WordPress root directory folder and find the Robots.txt file. Create a new blank file if you can’t find it.

Add this code at the top of the file:

User-agent:*
Disallow: /

This way, search engine robots should not index your website.

By using the code above, search engines will not index your entire site. In order to disallow a specific page, you must provide the subdirectory and slug.

Disallow: /private/my-article

Method 3: Server-Level Protection (The Secure Option)

You can instruct search engines not to index your website using the default WordPress visibility settings or the robots.txt file.

Even so, search engines may still crawl and index your website’s pages, files, or images.

The only way to ensure that search engines will not index or crawl your website, at all, is to password protect it on the server level.

If you want to be absolutely sure that your site stays private, you could password protect your WordPress site on the server level. Plugins like Password Protected can easily do this for you.

Install and activate the plugin, then navigate to Settings -> Password Protected and customize the settings according to your needs.

What If You Change Your Mind? How to Enable Indexing Again

If you decide to make your site publicly accessible:

1. Uncheck “Discourage Search Engines from Indexing This Site”

Go to Settings > Reading and uncheck the box in the Search Engine Visibility section.

2. Edit Robots.txt

Update robots.txt to remove the Disallow rules, or delete the file entirely.

3. Submit to Google Search Console

Submit again your site to Google Search Console. This helps search engines begin crawling your content immediately.