What robots.txt controls, and what it does not
A robots.txt file controls crawling, not indexing, and the distinction is the source of most confusion about it. Disallow tells a well-behaved crawler not to fetch a URL. It does not tell search engines to keep that URL out of their index, and a disallowed page can still appear in results — typically with no description, listed on the strength of links pointing at it from elsewhere.
The consequence is counter-intuitive: to keep a page out of search results you must let crawlers fetch it, so they can see a noindex directive in the page's meta tags or HTTP headers. Blocking it in robots.txt actively prevents that, because the crawler never reads the page and never learns it should be excluded.
It is also entirely advisory. The file is a public request that cooperative crawlers honour; it has no enforcement whatsoever, and scrapers ignore it freely. Anything requiring actual protection needs authentication, not a line in a text file.
The rules that trip people up
The file must live at the domain root — /robots.txt — and it applies only to that exact host and protocol. A subdomain needs its own file, and as far as the standard is concerned http and https are separate origins.
Matching is by prefix, which regularly surprises. Disallow: /admin blocks /admin, /admin/, and also /administrator and /admin-guide. Add a trailing slash to limit it to the directory. Paths are case-sensitive. Disallow: / blocks the entire site, and Disallow: with nothing after it blocks nothing — a one-character difference between a normal site and an invisible one.
Rules are grouped under User-agent lines, and a crawler obeys only the most specific group matching it, ignoring the rest entirely. A bot with its own named section does not also apply the * rules, so anything meant to apply universally must be repeated in each named group.
A file that publishes your secrets
Because robots.txt is public by design, listing sensitive paths in it advertises exactly what you hoped to hide. A Disallow: /internal-admin-panel/ line is a signpost, and reading robots.txt is a routine early step in reconnaissance.
Use broad patterns rather than specific ones where you can, and rely on authentication for anything that matters. If a path must not be discovered, it must not be discoverable — and a public file naming it defeats that entirely.
Two additions are worth making regardless: a Sitemap: line with the absolute URL of your sitemap, which is the standard way to advertise it and is honoured by all major engines, and a Crawl-delay only if a specific crawler is genuinely overloading your server — Google ignores it, and crawl rate is better managed in Search Console. The sitemap generator produces the file to point at.