Robots.txt Validator

Posted by
Creative Director

Friday, April 2nd, 2004 at 8:21 am

Archived in

& April 2004

“Search engines will look in your root domain for a special file named ‘robots.txt’ (http://www.mydomain.com/robots.txt). The file tells the robot (spider) which files it may spider (download). This system is called, The Robots Exclusion Standard. The format for the robots.txt file is special. It consists of records. Each record consists of two fields : a User-agent line and one or more Disallow: lines…”

Post to Twitter

Link: 

http://www.searchengineworld.com/cgi-bin/robotcheck.cgi
Both comments and trackbacks are currently closed.

More from xBlog


Kronos video

Sample visual
Check out this video we made for Kronos to help celebrate International Women's Day, 2011. Learn more in this xBlog post or jump over to YouTube and watch it there.

Azure poster

Sample visual
XPLANE | Dachis Group developed a A vibrant, engaging poster showing how Microsoft Azure enables developers to run applications and store data on Microsoft servers. The poster recently took top honors in the American Business Awards.

Tweets & Flickrs