How to disable automatic creation of the robots.txt file

By default, on shared and reseller servers, if a robots.txt file does not exist in the document root (public_html) directory, the server automatically creates a new robots.txt file at midnight.

This article describes how to disable this behavior. You may want to do this, for example, if you do not need to specify how search engines and web crawlers index your site.

Disabling automatic generation of the robots.txt file

To prevent the server from automatically generating a robots.txt file in the document root directory, follow these steps:

  1. Log in to cPanel.
    If you do not know how to log in to your cPanel account, please see this article.
  2. In the FILES section of the cPanel home screen, click the File Manager icon:

    cPanel - File Manager icon

  3. In the left sidebar, click the public_html folder.
  4. On the top menu bar, click + File:

    cPanel - File Manager - Create file

  5. In the New File dialog box, in the New File Name text box, type robots.txt.ignore:

    cPanel - File Manager - New File dialog box

  6. Confirm that the New file will be created in text box is set to /public_html.
  7. Click Create New File. Automatic generation of the robots.txt file is now disabled.

More Information

For more information about the robots.txt file, please vist http://www.robotstxt.org.

Did you find this article helpful? Then you'll love our support. Experience the A2 Hosting difference today and get a pre-secured, pre-optimized website. Check out our web hosting plans today.

Usamos cookies con el fin de personalizar para usted nuestro sitio Web y analizar el uso de éste. Usted acepta esto haciendo clic en “Acepto” o al continuar usando el sitio Web. Hay más información sobre cookies en nuestra política de privacidad.