E-mail:
jsem@patrikpilous.cz
Robots.txt is a simple text file placed in the root of a website that gives search engine robots instructions on where they may and may not go on the site. Think of it as a gatehouse at the entrance. A robot that arrives looks at it first and follows the rules. It is mainly used to stop robots wasting time and the so-called crawl budget on unimportant parts of the website, such as the admin area, the bin, internal search or overly complicated (overengineered) filters.
Despite its simplicity, robots.txt is one of the most sensitive files on a website, because a single wrong line can hide the entire site from search engines.
Where the word comes from
The name combines robot and the txt extension, meaning a plain text file for robots. A well-known fact is that the word robot itself is a Czech creation. Karel Čapek first used it in his 1920 play R.U.R., and his brother Josef derived it from the word robota, meaning hard, forced labour.
A brief history
Robots.txt was created in 1994 out of practical necessity. Dutch developer Martijn Koster was looking for a way to stop robots overloading servers and crawling where they should not. He proposed a simple protocol that became known as the Robots Exclusion Protocol. It was not an official standard but a voluntary agreement that “well-behaved” robots respected. It worked so well that it became a de facto standard that still holds today. Only in 2022 was the protocol officially codified as the internet standard RFC 9309. So for almost thirty years, one of the pillars of the web rested solely on everyone voluntarily following it.
How robots.txt works from an SEO perspective
The file contains simple rules. The User-agent line specifies which robot the rules apply to, Disallow blocks access to a given path and Allow, on the other hand, permits it. Robots.txt most often also includes a link to the sitemap, which is where all the URLs we want in search engine indexes are stored.
It is essential to understand one thing that trips up a lot of people. Robots.txt controls crawling, not indexing. If you block a page in robots.txt, the robot will not crawl it, but if links point to it, it can still appear in the index, just without a description. A different tool is used to remove a page from the index: the noindex tag.
Why robots.txt matters
A correctly configured robots.txt helps search engines focus on what matters. Importantly, this applies mainly to really large websites. On large sites, where a robot does not have time to crawl absolutely everything, directing the crawl budget towards valuable pages is a big advantage. At the same time, it is a tool that must be handled with care. The best-known disaster in SEO is a line that accidentally blocks the entire website, which happens mainly when moving from a staging version to the live site. That is why I check the contents of robots.txt at the start of every SEO audit. On WordPress, it is also often managed by an SEO plugin, so it is good to know what it contains.
Where it goes further
Robots.txt is a lovely example of how the internet is held together more by agreements and trust than by hard rules. For almost thirty years, it was enough that well-behaved robots voluntarily followed a plain text file that could not enforce anything. Anyone who wanted to could ignore it. But this fragile balance is cracking right now. Many robots that collect data for artificial intelligence interpret the rules in their own way or simply bypass them. The fight over who may access your content and what they do with it is back to square one. And robots.txt, that old gatehouse, has suddenly become one of the battlegrounds of the AI era.
Most common mistakes
- A Disallow slash line that accidentally blocks the entire website, typically after launching from staging.
- Blocking a page in robots.txt in the belief that this removes it from the index. That is what noindex is for.
- Blocking folders with images or scripts that search engines need to render the page.
- A missing link to the sitemap, which should be listed in robots.txt.
Questions about robots.txt
What is robots.txt?
A text file in the root of a website that gives search engine robots rules on where they may and may not go on the site.
Does robots.txt block indexing?
No. Robots.txt controls crawling, not indexing. A blocked page can still get into the index if links point to it. The noindex tag is used to remove it.
Where can I find robots.txt?
Always in the root of the domain at the address slash robots.txt. On WordPress, it is usually managed by an SEO plugin.
