Robots.txt

Robots.txt is a plain file at the root of a domain, such as example.com/robots.txt. It tells well behaved crawlers which parts of the site they shouldn't request. Site owners use it to keep crawlers away from admin areas, search result pages and other low value sections.

It controls crawling, not indexing. A page blocked in robots.txt can still appear in results if other sites link to it, usually without a description. To keep a page out of results, use noindex and let the page be crawled. Robots.txt is also public, so don't use it to hide anything private.

Practical point: check yours after every site change. One wrong line, like a blanket disallow left over from staging, can block the whole site. Use the robots.txt report in Google Search Console to test it. Also think about whether you want to allow or block AI crawlers, which is a separate choice.