Crawlability & Indexing
Before a page can rank in search, it has to be found, crawled, and indexed — and if any of those steps fails, the best content in the world is invisible to search engines. Crawlability and indexing are the technical foundation of SEO: they govern whether search engines can access your pages, understand their structure, and add them to the index they search. Get this wrong (a stray robots.txt rule, an accidental noindex, a canonical pointing at the wrong page) and traffic can vanish overnight, no matter how good your content or links are.
The key distinction many people blur: crawling (a bot fetching the page) and indexing (the page being stored and made eligible to appear in results) are different steps. A page can be crawled but not indexed, or blocked from crawling yet still indexed from external links. Understanding the controls for each — robots.txt, sitemaps, canonical tags, and noindex — is what lets you steer search engines deliberately instead of hoping.
TL;DR
- Discover → crawl → index → rank. Each step can break and hide your pages.
- robots.txt controls crawling;
noindexcontrols indexing — they're different. - XML sitemaps help discovery; canonical tags resolve duplicate content.
- One wrong directive (
Disallow: /, straynoindex) can deindex a whole site.
Quick Example
The two most consequential controls — and they do different things:
Crawling vs Indexing
- Crawling — a search bot (Googlebot) fetches the page's content by following links and sitemaps.
- Indexing — the crawled content is analyzed and stored in the search index, making it eligible to appear in results.
They're sequential but distinct: blocking crawling doesn't guarantee a page won't be indexed (it can be indexed from links pointing to it, without content), and a crawlable page isn't automatically indexed (the engine decides). Controlling each requires the right tool.
The Key Controls
- robots.txt — a file at your domain root telling crawlers which paths they may or may not crawl (
Disallow). It controls crawling, not indexing — don't rely on it to keep pages out of search. noindex(meta tag or HTTP header) — tells search engines not to index a page (it must be crawlable for the bot to see this directive).- XML sitemap — a list of your important URLs submitted to search engines to aid discovery, especially for large or poorly-linked sites.
- Canonical tag (
rel="canonical") — tells search engines which URL is the primary version when duplicate or similar content exists across multiple URLs, consolidating ranking signals.
Crawl Budget
For large sites, search engines allocate a finite crawl budget — how many pages they'll crawl in a given time. Wasting it on low-value, duplicate, or infinite URL spaces (faceted navigation, session ids) means important pages get crawled less often. Manage it by blocking crawl traps, fixing duplicate URLs with canonicals, and keeping a clean internal link structure. Small sites rarely need to worry; large ones must.
Common Indexing Problems
- Accidental
Disallow: /in robots.txt — blocks the whole site from crawling. - Stray
noindex— often a staging directive left in production, deindexing pages. - Wrong canonical — pointing pages at the wrong canonical URL, hiding them.
- Duplicate content without canonicals — splits ranking signals across URLs.
- Orphan pages — not linked internally, so never discovered.
- JS rendering issues — content only appearing after JavaScript the crawler doesn't execute (see SEO for SPAs).
Best Practices
- Submit an XML sitemap and keep it current (canonical URLs only).
- Use
noindex(not robots.txt) to keep pages out of the index — and don't block a page you're trying to noindex. - Set canonical tags to resolve duplicate content and consolidate signals.
- Monitor Search Console — the Index Coverage / Page Indexing report flags what's not indexed and why.
- Maintain clean internal linking so pages are discoverable.
Common Mistakes
Blocking a page in robots.txt to keep it out of search
Leaving a staging noindex in production
FAQ
What's the difference between crawling and indexing?
Crawling is when a search engine's bot fetches a page's content by following links and sitemaps. Indexing is when that content is analyzed and stored in the search index, making the page eligible to appear in results. They're separate steps: a page can be crawled but not indexed (the engine decides it's not worth indexing), and — counterintuitively — a page blocked from crawling can still be indexed based on links pointing to it (just without its content). Because they're distinct, you control them with different tools: robots.txt for crawling, noindex for indexing.
How do I stop a page from appearing in search results?
Use a noindex directive (a meta robots tag or an X-Robots-Tag HTTP header) — and crucially, make sure the page is still crawlable so the bot can actually read that directive. The common mistake is blocking the page in robots.txt instead, which prevents crawling: the bot never sees the noindex, and the page can remain indexed from external links. So to reliably remove a page from the index, allow crawling and add noindex; don't block it in robots.txt at the same time. For already-indexed pages, also request removal in Search Console.
What is a canonical tag and when do I need one?
A canonical tag (<link rel="canonical" href="...">) tells search engines which URL is the authoritative version when the same or very similar content is accessible at multiple URLs — for example, with tracking parameters, http/https, www/non-www, or print/mobile variants. It consolidates ranking signals (links, relevance) onto the canonical URL instead of splitting them across duplicates, and prevents duplicate-content dilution. You need it whenever content is reachable via more than one URL, which is extremely common; setting canonicals correctly is a core technical-SEO task.
Why isn't my page getting indexed?
Many possible causes, best diagnosed via Search Console's Page Indexing report. Common ones: it's blocked by robots.txt or carries a noindex directive; a canonical tag points elsewhere (so Google indexes the canonical instead); it's an orphan page with no internal links, so it was never discovered; the content is low-value or duplicate and Google chose not to index it; it relies on JavaScript the crawler didn't render (see SEO for SPAs); or it's simply new and not yet crawled. Check the indexing report for the specific reason, fix it, submit the URL, and request reindexing.
Related Topics
- Structured Data & Schema — Helping engines understand content
- Meta Tags & Open Graph — Page-level directives
- SEO for SPAs — Indexing JavaScript apps
- Core Web Vitals — Page experience signals
- Web Performance — Fast pages crawl better