Crawl rate is the speed at which a search engine’s bot requests pages from your server, usually expressed as requests per second or requests per day. For Google, the crawl rate limit is the ceiling Googlebot sets for itself so that crawling never slows your site down for real visitors.
How crawl rate works
Googlebot decides how fast to crawl by watching how your server responds. When pages come back quickly and without errors, it gradually opens more simultaneous connections and fetches more often. When responses slow down, or the server returns errors such as 500, 503 or 429 (too many requests), it backs off. The process is automatic and continuous.
Crawl rate is one half of crawl budget. It sets how much crawling your server can bear; crawl demand sets how much Google actually wants to do. A fast server does not guarantee heavy crawling if Google has little reason to revisit your pages.
Google used to offer a setting in Search Console to cap the crawl rate manually. It retired that tool in early 2024, saying its own systems now adjust well enough. Other search engines differ: Bing offers crawl control settings in Bing Webmaster Tools and honours the crawl-delay directive in robots.txt, which Google ignores.
Why it matters
For most sites, crawl rate takes care of itself and needs no attention. It becomes relevant in two situations.
The first is when crawling is too slow. If Googlebot has reduced its rate because your server is struggling, new and updated pages take longer to be picked up. A sudden fall in crawl requests is often an early warning of hosting trouble, such as an overloaded shared server or a firewall that has started rate-limiting bots.
The second is when crawling is too heavy. On cheap shared hosting, a burst of bot traffic, not only from Googlebot but from SEO tools and AI crawlers, can slow the site for customers. A small online shop in Norwich running a sale does not want its checkout slowing down because bots are working through every product filter.
Common mistakes
- Blocking Googlebot in robots.txt to reduce load. It stops crawling entirely for those paths, and pages can drop out of the index. It is a blunt tool for a capacity problem.
- Serving errors for days. Returning 503 or 429 slows Googlebot down quickly, but Google advises using this only for short periods. Leave it in place for long and pages may be dropped.
- Blaming Googlebot without checking. Many requests claiming to be Googlebot are fake. Verify by reverse DNS lookup before acting on log data.
- Mistaking a crawl-rate drop for a ranking problem. Lower crawling is usually a symptom of server trouble or reduced demand, not a cause of lower rankings in itself.
How to act on it
Open the Crawl stats report under Settings in Google Search Console. It shows total crawl requests per day, average response time and the share of responses that were errors, along with any host problems Google encountered. A rising response time alongside falling requests points to a server issue worth raising with your host.
If bots are genuinely straining the server, improve capacity first: caching, a CDN, a better hosting plan. For a short emergency, temporarily serving 503 responses to bots is the method Google itself recommends. Diagnosing crawl problems from Search Console data and server logs is part of my technical SEO work.