Submit a Ticket
Support Center » Knowledgebase » Web Server (Apache + IIS) » Automatic robots.txt inclusion

Automatic robots.txt inclusion

Our performance monitoring and optimization system may automatically add "robots.txt" to one or all of your domain names hosted with us (shared/VPS/Cloud packages), if your domain name lacks robots.txt file when initially deployed.  Our system will not alter or replace any existing robots.txt files, regardless if they are properly structured or search engine friendly.


Our default robots.txt is search engine safe and friendly (does not block any specific search engines or user agents), and it helps mitigate load caused by automatic search engine crawlers, especially if you are hosting applications such as WordPress or Joomla. For more information on robots and their impact on your account performance please review:

http://www.mochasupport.com/kayako/index.php?_m=knowledgebase&_a=viewarticle&kbarticleid=750

Please, note that additional restrictions to how search engine crawlers scan your account maybe enforced if your sites are getting slowed down due to search engine crawlers or your server is running out of resources to handle all incoming requests.

 This answer was helpful  This answer was not helpful
 Back