Googlebot is the web crawler Google uses to find and fetch pages for its search index: an automated program that requests your pages, reads them and follows their links to discover more. If Googlebot cannot reach a page, or sees something different from what visitors see, that page cannot rank properly in Google Search.
How Googlebot works
There are two main versions, Googlebot Smartphone and Googlebot Desktop. Since Google moved to mobile-first indexing, almost every site is crawled mainly by the smartphone version, so the mobile version of your page is the one that counts.
Crawling starts from URLs Google already knows about, from links on other pages and from sitemaps you submit. For each URL, Googlebot checks your robots.txt file first, then requests the page and records the status code and the HTML. Pages are then queued for rendering in an up-to-date version of Chromium, which runs the JavaScript and produces the version Google indexes. Links found on the rendered page join the queue.
Googlebot decides for itself how fast and how often to crawl. It slows down if your server responds slowly or returns errors, and visits important, frequently changing pages more often than stable ones. It ignores the crawl-delay directive in robots.txt. Google also runs other crawlers with their own names, such as Googlebot-Image and Googlebot-Video, plus special-purpose fetchers for products like ads and feeds.
Why it matters
Everything else in SEO depends on Googlebot getting through. A UK online shop with thousands of filter combinations can waste much of its crawl budget on near-duplicate URLs, so new products take weeks to appear. A site whose main content loads only after a script runs may show Googlebot an empty shell. A firewall or security plugin that treats Googlebot as an attacker can quietly block a whole site.
It also matters for security and data. Many bots claim to be Googlebot in their user agent string to slip past defences or scrape content. Treating every request labelled “Googlebot” as genuine can let scrapers through, while blocking real Googlebot by mistake can take a site out of search.
Common mistakes
- Blocking CSS and JavaScript. Old robots.txt rules that block theme or script folders stop Googlebot rendering the page as visitors see it.
- Using robots.txt to remove pages. Disallowing a URL stops crawling but not necessarily indexing. Use noindex on a crawlable page instead.
- Serving Googlebot different content. Showing it a special version is cloaking, which breaks Google’s spam policies.
- Trusting the user agent alone. Verify suspected Googlebot traffic before allowing or blocking it.
- Ignoring server errors. Repeated 5xx responses make Googlebot crawl less, slowing down every update.
How to act on it
Use the URL Inspection tool in Google Search Console on your most important pages, then click “Test live URL” and “View tested page” to see the rendered HTML and screenshot Googlebot produced. If the content is missing there, Google is not seeing it either.
Check the Crawl stats report under Settings in Search Console for response times, error rates and which file types are being fetched. To confirm real Googlebot visits in your server logs, run a reverse DNS lookup on the IP address: genuine requests resolve to googlebot.com, google.com or googleusercontent.com, and a forward lookup should return the same IP. Google also publishes its crawler IP ranges.
When Googlebot is struggling with a site, the cause is usually in templates, server configuration or JavaScript rather than in individual pages, which is why it sits at the centre of technical SEO work.