Discouraging search engines from indexing or crawling WordPress

WordPress has a feature which informs search engines and crawling bots that the website should not be indexed or crawled. When it is enabled, the "noindex, nofollow" meta tags will be added to your website header, and the sitemap will be removed from the virtual robots.txt file of WordPress. Additional details about the changes applied with this feature are available at the official WordPress documentation. The discouraging of search engine indexing is mostly used for test, staging, and private WordPress installations.

Note: It is up to the search engines or web crawlers to honor such requests, so this solution may prove to be ineffective in certain scenarios. If you wish to restrict public access to your WordPress website, you can enable password protection for the website, so only authorized parties will have access to it. For more details, please refer to the Password protection section from our online documentation.

Enabling this feature after your WordPress website has been indexed will not get the indexing for your website removed. You will need to send a request to the search engine or crawler to remove the indexing.

The feature that allows you to advise search engines and crawling bots to ignore your WordPress website can be managed from the WordPress Dashboard, through the hosting Control Panel > WordPress Manager, and via WP-CLI.

Request search engines to not index the website via the WordPress Dashboard

To request search engines to not index your website via the WordPress Dashboard, you need to log into your WordPress website as an administrator and navigate to the Settings > Readings section. Under the Search engine visibility option, select the checkbox next to Discourage search engines from indexing this site, and press the Save Changes button.

Discouraging search engines from indexing or crawling WordPress via the hosting Control Panel > WordPress Manager

You can enable the Discourage search engines from indexing this site option for your WordPress website very easily through the hosting Control Panel. All you need to do is open the management page for your WordPress installation through the hosting Control Panel > WordPress Manager and click on the Enable button next to Discourage search engine indexing from the Tools section.

Enable the Discourage search engine indexing option for WordPress via WP-CLI

To send information to search engines or crawlers that your WordPress website should not be indexed via WP-CLI (the command-line interface for the management of WordPress websites), you need to enable SSH access for your account. After that, connect to it through SSH, and navigate to the directory of your WordPress installation. For example, you will need to go to the ~/www/www directory of your account if WordPress is installed on your main domain. The exact command to do that is:

cd ~/www/www
To enable the Discourage search engine indexing option for your WordPress blog, use this WP-CLI command:

wp option set blog_public 0