How to Discourage Search Engines from Indexing Your WordPress Site
Those who own websites will do whatever it takes to get them indexed.
However, If you are still building your website, you may not want search engines to crawl it. In such situation, you should discourage search engines from indexing your site.
In this article, we will show you how to prevent search engines from crawling your WordPress site.
Why Would Want to Stop Search Engines From Indexing your WordPress Site?
The most important source of traffic for most websites is search engines. So why would anyone block search engines?
In the case of a publicly accessible domain name on which you’re developing your website, you probably don’t want Google to index your under construction or maintenance mode page.
It’s also possible that you want to have an invite-only website or a test website and want it to get listed on SERPs.
Many people believe that search engines won’t find their website if there are no links to it. Not quite.
- You may have inherited your domain name from someone else, and they may still have links pointing to your website.
- Your link may appear in some domain search sites’ results.
- There are literally thousands of pages that list domain names. There is a chance that your site will appear on one of those pages.
Discouraging Search Engines From Indexing WordPress Sites
Method 1: Utilizing the inbuilt functionality of a WordPress website
This method lets site owners prevent search engines from crawling or indexing their websites by using an inbuilt feature of WordPress.
- Click on ‘Settings’ in the WordPress admin area.
- Select ‘Reading’ from the drop-down menu below ‘Settings’.
- To confirm your changes, click on the ‘save’ button.
And that’s it!
Method 2: Editing the robots.txt file
If you prefer the manual approach, you can edit the robots.txt file with an FTP client.
Connect to your website using your FTP client and go to the WordPress root directory folder and find the Robots.txt file. Create a new blank file if you can’t find it.
Add this code at the top of the file:
User-agent:*
Disallow: /This way, search engine robots should not index your website.
By using the code above, search engines will not index your entire site. In order to disallow a specific page, you must provide the subdirectory and slug.
Disallow: /private/my-articleHow to Make Your Site Not Show Up in Search Results?
You can instruct search engines not to index your website using the default WordPress visibility settings or the robots.txt file.
Even so, search engines may still crawl and index your website’s pages, files, or images.
The only way to ensure that search engines will not index or crawl your website, at all, is to password protect it on the server level.
Hopefully, many plugins can help you password protect your WordPress site. In our opinion, Password Protected is the best plugin. Using it is pretty simple and you shouldn’t need further explanation.
Install and activate the plugin, then navigate to Settings -> Password Protected and customize the settings according to your needs.
We hope that you have found this article useful in stopping search engines from crawling your site. You might also be interested in our article about protecting WordPress files.