robots.txt

A robots.txt file is a root-level file declaring which crawlers may access which parts of a site. Unlike meta directives, it operates before a page is fetched rather than after.