Screaming Frog is the common name for the Screaming Frog SEO Spider, a desktop program that crawls a website the way a search engine bot does and reports what it finds on every URL. It is built by Screaming Frog, a UK search marketing company, and is one of the most widely used technical SEO tools.
How Screaming Frog works
You enter a starting URL and the SEO Spider follows every link it can find, recording each page, image, script and stylesheet. For each URL it collects the status code, title tag, meta description, headings, canonical tag, robots directives, word count, response time, links in and out and much more. The results appear in tabs and filters, so you can jump straight to “missing titles”, “redirect chains” or “pages blocked by robots.txt”, and export any list to a spreadsheet.
It runs on Windows, macOS and Linux and uses your own computer’s memory, so very large sites need a well-specified machine or its database storage mode. At the time of writing (October 2026), the free version crawls up to 500 URLs with core features. A paid annual licence removes the limit and adds features such as JavaScript rendering, custom extraction of any element on the page, scheduled crawls, crawl comparison and connections to Google Search Console, GA4 and PageSpeed Insights. The same company makes a separate Log File Analyser for log file analysis.
Why it matters
Most SEO problems that cost UK businesses traffic are not visible from the front of the site. A page can look fine in a browser while carrying a noindex tag, canonicalising to the wrong URL, sitting behind three redirects or having no internal links pointing to it. A crawler shows these problems across every page at once, which is how you find the template fault affecting 800 product pages rather than the one page someone noticed.
Because it crawls like a bot, it is also the quickest way to test changes before and after a redesign or website migration, comparing the old and new site URL by URL.
Common mistakes
- Treating every warning as urgent. The tool flags anything that might be an issue. A meta description a few characters too long is not equal to 200 pages returning 404.
- Crawling without JavaScript rendering on a JavaScript site. On sites built with React, Vue or other script-heavy frameworks, the default text-only crawl can miss content and links that only appear after scripts run.
- Hammering a fragile server. The default speed can strain cheap shared hosting. Lower the crawl speed for small sites and tell the host if you are crawling a large one.
- Forgetting orphan pages. A crawl only finds what is linked. Connect your XML sitemap, Search Console and analytics to see pages that exist but have no internal links.
- Crawling the wrong version. Confirm you are auditing the live site, not staging, and that no robots.txt rule or password is cutting the crawl short.
How to act on it
Run a full crawl of your live site with your XML sitemap and Search Console connected. Start with the issues that stop pages being found or indexed: error status codes, unintended noindex tags, wrong canonicals, redirect chains and orphan pages. Then move to duplicate or missing titles and descriptions, thin pages and internal linking depth. Save the crawl, fix in batches, then recrawl and compare.
If you would rather have the crawl interpreted and turned into a ranked list of fixes, that is a core part of my technical SEO work.