How to Discourage Search Engines from Indexing Your WordPress Site

Why would you want to disable search engine visibility for your WordPress site?

Most of the people who own websites would do whatever it takes to get them indexed.

However, If you are still building your website, you may not want search engines to crawl it. In such situation, you should discourage search engines from indexing your site. 

In this article, we will show you how to prevent search engines from crawling your WordPress site.

Why Would You Want to Disable Search Engine Visibility for Your WordPress Site?

Most website owners consider search engines like Google or bing to be their best friend. After all, search engines are often the most significant source of traffic.

So, why would anyone want to disable search engine indexing for a WordPress site?

There are actually a few good reasons to do such thing. For exemple, this might the case if you’re still building your website on a publicly accessible domain. In that case, you might not want Google or any other search engine to index your “Under Construction” or “Coming Soon” pages. Other reasons could include:

1. Creating an invite-only or test website.
2. Transferring a domain that still has outdated links pointing to it.
3. Managing a private blog or portfolio that you don’t want to appear in search engine results.

Remember, even if you think no one knows about your site yet, there are various ways it could accidentally appear in search engine results. Old links from a previous domain owner, random domain directories, or even automated crawls can make your site more public than you’d like.

How Do Search Engines Index Websites?

Before we get into how discourage search engines, let’s briefly discuss how they work. Search engines use bots or spiders to crawl websites and index their content.

Once indexed, your site starts appearing in search engine results pages (SERPs).

This is generally what you want, but not always.

Discouraging Search Engines From Indexing Your WordPress Site

There are two ways you can discourage search engines from indexing your WordPress site. Let’s explore them in detail.

Method 1: WordPress Search Engine Visibility Settings

This method lets site owners prevent search engines from crawling or indexing their websites by using an inbuilt feature of WordPress.

1. Click on ‘Settings’ in the WordPress admin area. 
2. Select ‘Reading’ from the drop-down menu below ‘Settings’.
3. To confirm your changes, click on the ‘save’ button.

And that’s it! However, keep in mind that it’s up to search engines to honor this request.

Method 2: Manually Edit the Robots.txt File

For those who prefer to take matters into their own hands, you can manually edit the robots.txt file to prevent search engines from crawling your site.

Connect to your website using your FTP client and go to the WordPress root directory folder and find the Robots.txt file. Create a new blank file if you can’t find it.

Add this code at the top of the file:

User-agent:*
Disallow: /

This way, search engine robots should not index your website.

By using the code above, search engines will not index your entire site. In order to disallow a specific page, you must provide the subdirectory and slug.

Disallow: /private/my-article

Method 3: Server-Level Protection (The Nuclear Option)

You can instruct search engines not to index your website using the default WordPress visibility settings or the robots.txt file.

Even so, search engines may still crawl and index your website’s pages, files, or images.

The only way to ensure that search engines will not index or crawl your website, at all, is to password protect it on the server level.

If you want to be absolutely sure that your site stays private, you could password protect your WordPress site on the server level. Plugins like Password Protected can easily do this for you.

Install and activate the plugin, then navigate to Settings -> Password Protected and customize the settings according to your needs.

What If I Change My Mind? How to Index WordPress Site in Google

If you later decide that you want to get your WordPress site indexed in Google, simply reverse the steps taken above. Google and other search engines should naturally start crawling and indexing your site over time.