If your website does not appear on Google at all, the cause is nearly always one of six faults: the site is asking search engines to leave it out, a file on the site is turning Google away, Google cannot load the pages, Google has never found them, the pages point Google at a different address, or the site is new and Google has not caught up yet. If it does appear, just not for the searches you care about, that is a ranking problem rather than an indexing problem, and it has different fixes. Below is how to tell which one you have and how to check each cause yourself, in the order I would check them.
Indexed, missing or not ranking: the first check
Before Google can show a page, it has to find it, fetch it and store it in its index, the collection of pages it chooses from whenever someone searches. Each stage can fail on its own, and each failure has its own fix, so spend five minutes finding out which one you are dealing with.
Type site:yourdomain.co.uk into Google, using your own domain with nothing between the colon and the address. The results are pages Google has stored for that domain.
- Nothing comes back. Google holds none of the site. Work through the causes below from the top.
- Some pages appear and others do not. Google has the site but is missing particular pages. The page-level causes, numbers 1, 4 and 5 below, are the likely ones.
- Your pages are listed, but an ordinary search never finds them. They are indexed and not ranking. Skip to the section on indexed pages that still cannot be found.
Treat the site: result as a rough guide. It is not a complete list and sometimes leaves out pages Google does hold. For a definite answer about a single page, use the URL Inspection tool in Google Search Console, which I come back to further down.
Six causes to check, in this order
The order matters. The first three can hide an entire site and take minutes to check, so there is no sense working on content or links until they are ruled out.
1. The site is asking to be left out
A page can carry a short instruction telling search engines not to include it in their results. This is a noindex tag, and Google follows it. Developers often switch it on while a site is being built, so that unfinished pages stay out of Google, and then forget to switch it off at launch.
On WordPress, open Settings, then Reading, and find Search engine visibility. If the box labelled “Discourage search engines from indexing this site” is ticked, WordPress adds noindex to every page on the site. Untick it and save. I check this before anything else, because it takes ten seconds and it blocks everything when it is wrong.
Wix, Squarespace and Shopify each have their own settings for hiding pages, or a whole site, from search or behind a password, and a new Shopify shop stays password-protected until you remove the password. SEO plugins can also apply noindex to whole groups of pages, such as tag archives or a product category. To check a single page, right-click it, choose View page source and look for the word “noindex”. The same instruction can be sent in the server’s response headers, where the source will not show it, but URL Inspection reports it either way.
2. robots.txt is keeping Google’s crawler out
Robots.txt is a plain text file at the root of a site that tells crawlers which areas they may visit. Open yourdomain.co.uk/robots.txt in your browser. If it contains Disallow: / under User-agent: *, it is telling every crawler, Google’s included, to stay out of the whole site. That line is a frequent leftover from a staging copy that was moved onto the live domain. Delete it, and keep only rules for areas you genuinely want left alone, such as an admin login.
Two details catch people out. A robots.txt block stops Google reading a page, not listing it, so a blocked address can still turn up in results with no description if other sites link to it. And Google cannot see a noindex tag on a page it is not allowed to fetch. To keep a page out of Google, let it be crawled and use noindex; to get it in, make sure neither stands in the way.
3. Google cannot load the pages
Google fetches your pages much as a visitor’s browser does, so anything that stops a stranger reaching your content stops Google too. The usual suspects are:
- a password on the whole site, left over from development or set in the hosting control panel;
- a “coming soon” or maintenance-mode plugin that is still active;
- server errors, where pages fail to load some of the time, for example when a hosting plan runs short of resources;
- a firewall or bot-protection service that treats Google’s crawler as an intruder and blocks or challenges it.
A quick test: open the site in a private window on your phone, using mobile data rather than the office wifi. If you hit a login box, a holding page or an error, Google is probably hitting it too. In Search Console, the live test in URL Inspection fetches the page as Google and shows you what came back, which settles the question.
4. Google has never found the pages
Google discovers most pages by following links, both between pages on your own site and from other websites. A new domain that nobody links to, or a page your own site never links to, can sit unnoticed for a long time. So can pages reached only through a search box, a form, or menu items built with JavaScript that are not proper links.
Fix it from both ends. Link each page you want found from at least one other relevant page, using words that describe what is on it. Then submit an XML sitemap, a file that lists your pages, in Search Console, so Google has a list to work through. WordPress creates one at /wp-sitemap.xml unless an SEO plugin has replaced it with its own. Keep the sitemap to pages you want indexed: one padded with redirects and noindexed pages sends Google mixed messages.
5. The page points Google somewhere else
A canonical tag tells Google which address is the main version of a page. Used well, it stops near-identical pages competing with each other. Set wrongly, it tells Google to index a different page instead of the one it sits on. After a launch, I look for canonicals that still name the staging address, or for every page naming the homepage. View the source, search for “canonical” and check that the address it gives is the page itself, on the live domain, in the right https and www form.
Redirects can do the same damage. If the old domain, or the www or non-www version, redirects in a loop or to the wrong page, Google may never settle on the address you want it to show.
6. The site is new and Google has not caught up
If none of the above applies and the site launched recently, it may simply be in the queue. Google gives no fixed timescale for indexing a new site: it depends on how easily Google can find the site and how often other pages link to it. To help, verify the site in Search Console, submit the sitemap, use Request indexing on the handful of pages that matter most, and add your web address wherever the business is already listed, such as your Google Business Profile or a trade association’s member directory. Getting indexed is only the first step; ranking takes longer, as I explain in how long SEO takes to show results.
My rule of thumb: if a site is still completely absent a few weeks after launch, with a sitemap submitted, stop waiting and go back through causes 1 to 5.
Ask Google directly in Search Console
Search Console is Google’s free service for site owners, and it is the one place Google tells you directly how it sees your pages. If you have not set it up yet, start there. Choose a domain property, which you verify by adding a TXT record to your domain’s DNS settings at your registrar. It covers every version of the site at once: http and https, with and without www, and any subdomains.
Three parts of it answer most of these questions:
- URL Inspection. Paste in any address on your site to see whether the page is in Google’s index, when Google last crawled it, which canonical Google picked and whether anything blocked it. Once you have fixed the problem, the Request indexing button asks Google to come back.
- The Page indexing report. It sorts every page Google knows about by status, with a reason for each one left out of the index. The reasons line up with the causes above: “Excluded by ‘noindex’ tag”, “Blocked by robots.txt”, “Server error (5xx)”, “Duplicate, Google chose different canonical than user” and so on.
- Manual actions and Security issues. Rarely relevant to a small business, but if a Google reviewer has penalised the site, or Google has found it hacked, these two reports say so and nothing else will.
Two statuses in the Page indexing report are not settings you can switch off. “Discovered – currently not indexed” means Google knows the page exists but has not fetched it yet, which is common on new sites and on very large ones. “Crawled – currently not indexed” means Google fetched the page and chose to leave it out for now. That usually points at the page itself: thin, too close to another page, or adding nothing to what Google already holds. The fix is a better page, not a technical change.
Indexed, but you still cannot find it
If URL Inspection says a page is on Google, the site is showing. It is not ranking where you are looking, which is a different problem with different fixes.
Search for your exact business name first. If you come up for that, Google has the site and knows whose it is. Not appearing for a broad phrase such as “accountant” or “kitchen fitter London” is normal for a new or small site, because you are up against businesses that have spent years building pages and links around those phrases. Changing that takes a page written for the phrase that answers the searcher better than the pages above it. When your question has widened from “why can I not find my site” to “why is it not doing better”, my SEO audit checklist covers the full set of checks.
When you can see it and your customers cannot
Sometimes it runs the other way: you search and find yourself near the top, but a customer says they could not find you. Google’s results vary with the searcher’s location, device and language, and to some extent with what they have searched for before. You have opened your own site countless times from your own desk. A customer searching on a phone in Croydon may be shown different results from someone at a laptop in Camden.
A private browser window strips out some of that, but not your location. For a fairer view, open the Performance report in Search Console. It shows the average position your pages held for each search across real people’s searches, and you can filter it by country and device. Searching on google.co.uk instead of google.com makes no difference, because Google goes by where you are, not by which address you typed.
Stop it happening on the next launch
Most of the faults above are introduced at one of three moments: when a site is launched, when it is rebuilt, or when it moves to a new host. Settings that are correct on a staging copy are wrong on the live site, and nobody notices because the site looks perfect to the people who built it. My website launch SEO checklist covers what to check on the day. If a developer is building your site, ask for a screenshot of the search engine visibility setting, a copy of the live robots.txt and a URL Inspection result for the homepage before they hand it over.
If you have ruled all of this out
If the settings are clean, robots.txt allows crawling, the site loads for a stranger, the sitemap is in and URL Inspection still shows important pages out of the index, the cause is harder to see. Content that only appears once JavaScript runs, a second copy of the site on another address, a domain with a history Google distrusts, or quality problems across many pages can all produce the same symptom. Finding which one means reading crawl data, server responses and Search Console together, and that is the work I do in technical SEO.
If you would like a second pair of eyes first, request a free SEO audit. Send me your web address and I will reply with whether Google can index your pages and, if it cannot, what I would fix first.