robots.txt controls crawling, not indexing. A blocked URL can still appear in search results if other sites link to it; use noindex to keep a page out.
It can also point crawlers to your sitemap and set rules per bot, including AI crawlers like GPTBot and ClaudeBot.
Why it matters when you launch
Check that your production robots.txt isn't still blocking everything from staging. It's one of the most common launch-day SEO mistakes.