E-mail:
jsem@patrikpilous.cz
Crawling, procházení in Czech (literally plazení, i.e. creeping), is the process in which a search engine robot (or crawler) goes through a website and reads one page after another. For search engines, the main purpose is to find new or updated pages.
The robot, which Google calls Googlebot, starts with known addresses and works its way further by following links, page by page, link by link. Hyperlinks are how crawlers move through websites.
However, as an SEO consultant, I also use various crawlers such as Screaming Frog or the Ahrefs crawler to get data about a website’s pages and its overall structure. This way I can, for example, find
What the crawler (robot) finds is the basis for the next step, which is indexing. Crawling is therefore the very first condition for visibility. If the robot never reaches your page or cannot access it, it cannot store it or show it in the results. Simply put, without crawling there is no organic traffic.
You can nudge Google in Google Search Console to visit individual pages or the whole website.
Where the word comes from
The English verb to crawl means to creep, to move on hands and knees or to move slowly. That is exactly how the first creators of search engines imagined their robots: as spiders slowly making their way across the web along threads of links. Hence another common term, web spider. In Czech, the plain word procházení has taken hold, but among SEO people you are more likely to hear the English crawling or crawl.
The metaphor of a creeping creature systematically exploring the terrain still fits perfectly today, at a time when SEO is far more complex. In fact, even modern AI models rely on pages crawled and indexed by search engines.
Even so, the best-known tool for SEO consultants, Screaming Frog, has a frog in its logo instead of a spider.
A short history of crawling
The first web crawlers appeared in the early 1990s, even before Google. Their task was simple: go through as many pages as possible and build a list, a catalogue, a library from them. Few people today can imagine the internet without a search engine. You had to know a specific URL, try things out, and you were mostly discovering something someone had already told you about. Search engines connected and unified the internet.
Google perfected the whole principle and combined crawling with assessing the importance of pages.
Today, the robot does not just read text. It can also render a page much like a browser and process content loaded with JavaScript. The development of crawling goes hand in hand with how the whole of SEO and GEO is changing, with websites now also crawled by language model bots.
How crawling works from an SEO perspective
You can make crawling much easier for the robot, or seriously complicate it. The robots.txt file tells it where it may and may not go, and it finds a list of important addresses in the sitemap. A clear website architecture and structure is also key, as it lets the robot reach every page within a few clicks.
On larger websites, people deal with the so-called crawl budget, meaning how many pages the robot crawls in a given time. It can be wasted on a slow website, which poor Core Web Vitals also hold back, or on unnecessary redirect chains. The more efficiently the robot crawls, the faster new content appears in the results.
Why crawling matters
Crawling is the quiet foundation everything else stands on. You can have the best content in the world, but if the robot cannot reach it, it is as if it did not exist. That is why checking crawling is at the core of every technical SEO audit.
On large websites, especially e-shops, crawling becomes a real discipline. Thousands of filters and product variants can overwhelm the robot so much that there is no time or capacity left for the important pages. Sensible management of the crawl budget then directly affects how quickly changes to the website show up in the search results.
Where it goes further
Crawling is essentially how a search engine maps a constantly changing world. The web changes every second, and the robot has to keep going round it again and again so that its picture of reality does not go out of date. How often it stops by your website also depends on how important your website is, which is reflected in its domain authority. The robot visits strong, frequently updated websites far more often than those that stand still. The robot’s attention is limited, and it allocates it by importance, just like a person.
Most common mistakes
- Accidentally blocking important parts of the website in the robots.txt file, or noindex on newly created pages.
- Wasting crawl budget on endless filters, duplicates and redirect chains.
- A forgotten or outdated sitemap, so the robot does not even know about some pages.
- Confusing crawling with indexing. The fact that the robot crawls a page does not mean it will also index it. If it does not index it, you can see this in GSC under Discovered – currently not indexed.
Questions about crawling
What is crawling?
The process of a search engine robot going through a website, reading pages and following the links between them. It is the first step, without which a page cannot get into the index or the search results. So it gets neither impressions nor clicks.
What is the difference between crawling and indexing?
Crawling means going through and reading a page. Indexing means storing it in the search engine’s database. A page can be crawled and still not be indexed.
What is crawl budget?
The number of pages a robot crawls on a given website within a certain time. It mainly matters for large websites and e-shops, where it can easily be wasted on unimportant addresses.
