Crawl-delay is an unofficial line in a robots.txt file that asks a crawler to wait a set number of seconds between requests to your site. It is a request, not a control, and search engines differ in whether they respect it.
How crawl-delay works
The directive sits inside a user-agent group in robots.txt, for example:
User-agent: bingbot
Crawl-delay: 5
That asks Bing’s crawler to leave roughly five seconds between fetches. The arithmetic is worth doing before you set a value. A delay of 10 seconds limits a crawler to at most 8,640 requests in a day; a delay of 30 seconds brings that down to 2,880. On a shop with tens of thousands of product and category URLs, that can mean weeks before every page is revisited.
Support varies:
- Googlebot ignores it. Google never adopted the directive and sets its own pace from how your server responds. Its documented standard for robots.txt does not include crawl-delay.
- Bingbot honours it. Bing also offers crawl control settings in Bing Webmaster Tools, which give finer control over the hours when it crawls hardest.
- Other bots vary. Some SEO tools and AI crawlers respect it; others do not. Check each crawler’s own documentation rather than assuming.
Why it matters
Crawl-delay mostly matters by accident. Many sites carry a crawl-delay line copied from an old template or added by a hosting company years ago, and nobody remembers why. If the value is high, it can quietly slow how quickly Bing finds new pages and picks up changes. Bing’s index also feeds other services, so slow Bing crawling can reach further than Bing’s own results.
It can also be genuinely useful. A small business on modest shared hosting that sees its site slow down when several bots arrive at once may use a short delay for specific crawlers to spread the load. That buys time while the underlying capacity problem is addressed, and it can be removed again once the hosting is sorted.
What it cannot do is manage Google. If Googlebot is putting strain on your server, crawl-delay will make no difference, and the answer lies in server capacity or, for a short emergency, temporary 503 or 429 responses as Google recommends. The broader picture is covered under crawl rate.
Common mistakes
- Setting it under User-agent: * and expecting Google to obey. Google will ignore it; everyone else who honours it will be slowed.
- Choosing a large value without doing the sums. A delay that looks small in seconds can cap a crawler at a few thousand pages a day.
- Leaving an inherited value in place. Old robots.txt files often carry crawl-delay lines nobody can justify.
- Using it instead of fixing hosting. If ordinary bot traffic slows your site, real visitors at busy times are probably suffering too.
How to act on it
Open yoursite.co.uk/robots.txt and check for any crawl-delay lines. If you find one, ask what problem it was meant to solve and whether that problem still exists. If the site is not under strain, remove it. If it is, set the delay only for the specific bots causing trouble, with a short value, and use Bing Webmaster Tools crawl control for Bing.
Then look at the server side: caching, a CDN and adequate hosting usually solve bot load far better than delays. Reviewing robots.txt and crawler behaviour is part of every technical SEO engagement I run.