When someone tells us their site does not rank, the first thing we check is not their content or their backlinks. It is whether Google has the pages at all.

More often than people expect, it does not. And a page that is not in the index cannot rank for anything, no matter how good it is. You are not losing to a competitor — you were never in the race.

Here is how to check, and the six things that are usually wrong.

First, get the actual number

Open Google Search Console, go to Indexing → Pages, and compare two numbers: how many pages are indexed, and how many pages your site actually has.

If you do not have Search Console set up, that is your first problem and it takes ten minutes to fix. As a rough substitute in the meantime, search Google for site:yourdomain.com. The result count is approximate and Google says not to rely on it, but if your forty-page site returns three results, you do not need precision to know something is wrong.

Now, the six causes, roughly in order of how often we find them.

1. robots.txt is blocking the site

The classic. A site is built on a staging server with a robots.txt that says:

User-agent: *
Disallow: /

That is correct behaviour for staging — you do not want a half-finished site indexed. Then the site is deployed to production and the file goes with it. Every crawler is now politely declining to look at anything.

Check: visit yourdomain.com/robots.txt and read it. If you see Disallow: / under User-agent: *, that is the whole problem.

This is the single most expensive two-line file in web development, and it can sit there for a year because the site looks completely normal to a human.

2. A stray noindex tag

Same story, different mechanism. A <meta name="robots" content="noindex"> in the page head, or an X-Robots-Tag: noindex HTTP header, tells search engines to drop the page even though they can crawl it.

WordPress has a checkbox in Settings → Reading labelled “Discourage search engines from indexing this site”. It is meant for development. It gets left on constantly.

Check: view source and search for noindex. In Search Console, the Pages report lists these explicitly under “Excluded by ‘noindex’ tag”.

Worth knowing: a page blocked in robots.txt cannot be seen to have a noindex tag, because the crawler never fetches it to read the tag. If you want a page removed from the index, you must let it be crawled so the noindex can be seen. Blocking it in robots.txt instead can leave it indexed indefinitely.

3. Everything canonicalises to the homepage

A canonical tag tells Google “this is the authoritative URL for this content”. It is the correct fix for genuine duplicates.

It is also frequently applied wrongly — a template puts <link rel="canonical" href="https://example.com/"> in a shared header, and now every page on the site declares that it is really the homepage. Google consolidates them all and indexes one URL.

Check: view source on any interior page and find the canonical tag. It should point at that page’s own URL. Not the homepage, not a trailing-slash variant, not http:// when you serve https://.

4. The content only exists after JavaScript runs

Google can render JavaScript. That is true and people repeat it as though it settles the matter. It does not.

Rendering is a second pass. The crawler fetches the HTML, and if the content is not there it queues the page for rendering, which happens at some later point that you do not control. On a large or frequently changing site, that delay compounds. On a small site with a low crawl budget it can mean pages sit unrendered for weeks.

Other crawlers are worse. Many social preview bots and some search engines do not execute JavaScript at all, which is why a link to a client-rendered page often unfurls with no title or description.

Check: disable JavaScript in your browser and reload your page. What remains is roughly what a crawler reads on the first pass. If it is a blank div, that is your problem. Or use the URL Inspection tool in Search Console and look at the rendered HTML it reports.

Fix: server-side rendering or static generation. Every page on this site ships complete HTML — you can turn JavaScript off entirely and read all of it.

Google finds pages by following links and by reading your sitemap. A page with no internal links pointing at it is discoverable only through the sitemap, which is a weak signal — it says the page exists, not that it matters.

Orphans accumulate quietly: a landing page built for a campaign, a service page dropped from the navigation during a redesign, an old location page.

Check: crawl your own site with a tool like Screaming Frog and compare the URLs it discovers against your sitemap. Anything in the sitemap that the crawl never reached is an orphan.

Fix: link to it from somewhere relevant, or accept that it does not matter and remove it. Both are fine. Leaving it in the sitemap unlinked is the only bad option.

6. Crawl budget spent on pages that do not matter

Google allocates a finite amount of crawling to your site. Most small sites never hit that limit — but sites that generate URLs automatically absolutely do.

Faceted navigation is the usual culprit. A shop with filters for colour, size, and price generates a distinct URL per combination, and thousands of near-identical pages get crawled while your actual service pages wait. Calendars are another: an “next month” link that goes on forever produces an infinite crawl space.

Check: in Search Console, Settings → Crawl stats. Look at what Google is spending its requests on. If most crawls are hitting URLs with query parameters, that is where the budget is going.

Fix: robots.txt disallow rules for parameter patterns, canonical tags pointing filtered views at the unfiltered page, and nofollow on links that generate infinite spaces.

The fifteen-minute version

If you do one thing after reading this:

  1. Open yourdomain.com/robots.txt. Look for Disallow: /.
  2. Open Search Console → Indexing → Pages. Compare indexed pages to actual pages.
  3. View source on an interior page. Check the canonical points at itself and there is no noindex.
  4. Disable JavaScript and reload. Confirm the content is still there.

That catches the overwhelming majority of cases, and every one of them is a plumbing problem rather than a competitive one. Which is good news: plumbing problems have definite fixes, and the recovery when you fix them is often dramatic, because the pages were fine all along. Nobody could see them.

If you would rather someone else did this, our free visibility audit covers all six checks and comes back in writing, whether or not you hire us.