Those who own websites will do whatever it takes to get them indexed.
However, If you are still building your website, you may not want search engines to crawl it. In such situation, you should discourage search engines from indexing your site.
In this article, we will show you how to prevent search engines from crawling your WordPress site.
Table of contents
The most important source of traffic for most websites is search engines. So why would anyone block search engines?
In the case of a publicly accessible domain name on which you’re developing your website, you probably don’t want Google to index your under construction or maintenance mode page.
It’s also possible that you want to have an invite-only website or a test website and want it to get listed on SERPs.
Many people believe that search engines won’t find their website if there are no links to it. Not quite.
This method lets site owners prevent search engines from crawling or indexing their websites by using an inbuilt feature of WordPress.
And that’s it!
If you prefer the manual approach, you can edit the robots.txt file with an FTP client.
Connect to your website using your FTP client and go to the WordPress root directory folder and find the Robots.txt file. Create a new blank file if you can’t find it.
Add this code at the top of the file:
User-agent:*
Disallow: /
This way, search engine robots should not index your website.
By using the code above, search engines will not index your entire site. In order to disallow a specific page, you must provide the subdirectory and slug.
Disallow: /private/my-article
You can instruct search engines not to index your website using the default WordPress visibility settings or the robots.txt file.
Even so, search engines may still crawl and index your website’s pages, files, or images.
The only way to ensure that search engines will not index or crawl your website, at all, is to password protect it on the server level.
Hopefully, many plugins can help you password protect your WordPress site. In our opinion, Password Protected is the best plugin. Using it is pretty simple and you shouldn’t need further explanation.
Install and activate the plugin, then navigate to Settings -> Password Protected and customize the settings according to your needs.
We hope that you have found this article useful in stopping search engines from crawling your site. You might also be interested in our article about protecting WordPress files.
This guide is designed to describe all major aspects of WP_Debug & WP_Debug_log so you can enable WordPress debugging.
Learn how to add favicon in WordPress.
Learn how to easily create professional-looking WordPress 404 error pages.