2nd
April
2004
Robots.txt Validator
“Search engines will look in your root domain for a special file named ‘robots.txt’ (http://www.mydomain.com/robots.txt). The file tells the robot (spider) which files it may spider (download). This system is called, The Robots Exclusion Standard. The format for the robots.txt file is special. It consists of records. Each record consists of two fields : a User-agent line and one or more Disallow: lines…”