What robots.txt Actually Controls (and Doesn't)
A robots.txt file tells well-behaved crawlers which parts of a site they're welcome to crawl — but it's fundamentally an honor-system convention, not an access control mechanism. Any crawler can simply ignore it, and in fact malicious scrapers routinely do; robots.txt only affects the behavior of crawlers that voluntarily choose to respect it, which major search engines generally do, but nothing enforces compliance.
This distinction matters because robots.txt is frequently misunderstood as a security or privacy tool — disallowing a path in robots.txt doesn't password-protect it, encrypt it, or hide it from anyone who already has (or guesses) the URL; it only asks compliant crawlers not to index it. A truly private path needs actual authentication, not a robots.txt entry. Blocking a path also doesn't guarantee it stays out of search results either — if other pages link to a disallowed URL, search engines can still show that URL (without crawling its content) based on the link alone, which is a common source of "why is this blocked page still showing up in search" confusion.
Different crawlers (Googlebot, Bingbot, AI-training crawlers like GPTBot) can be given different rules via separate User-agent blocks, which is increasingly relevant now that many sites want to allow search indexing while opting out of AI training crawls specifically.