Robots.txt

Text file placed in a websites root directory and linked in the html code.

Allows for SEO's to control the actions of search engine spiders on the site or even deny them access.