Please note that aboutwebsite does not interpret a 401/403 response ("Unauthorized"/"Forbidden") to a robots.txt fetch as a request not to crawl any pages on the site. If your server is case sentitive, your filename must be all lower case.
Example: http://www.yoursite.com/robots.txt
To remove your site from search engines and prevent all robots from crawling it in the future, place the following robots.txt file in your server root: |
