Robots.txt

Robots.txt is a text file placed at the root of a website that tells search engine crawlers which pages or sections of the site they are and are not allowed to crawl. It is used to prevent search engines from indexing duplicate content, staging pages, private areas, or low-value pages that could dilute search performance. Robots.txt controls crawling but does not prevent a page from being indexed if other sites link to it.

See also: Crawlability · Indexing · Technical SEO · Canonical URL