Robots.txt vs Noindex: Which Keeps a Page Out of Search

Robots.txt keeps a crawler off the page. noindex keeps the page out of the index. Only one of those is a removal tool, and it's noindex.

If you want a page gone from search results, serve noindex and let the crawler reach it. If you want to stop wasting crawl activity on URLs that were never going to be indexed anyway, use robots.txt. If you apply both to the same URL, you get the worst outcome on the menu: the page can stay in the index, and you've made it impossible for the engine to ever learn otherwise.

That table is the whole argument. The rest of this post is why each row is true and what to do about it. If you want the wider context these two directives sit inside, start with what technical SEO covers and come back.

Every confusing behavior in this topic falls out of a single fact: the two directives act at different moments, and robots.txt acts first.

Row 1 is the one everybody skips, and it's the one that matters. A search engine can learn a URL exists without ever fetching it: an inbound link from another site, an internal link, a sitemap entry, a redirect target, a mention in a public document. Robots.txt has no effect on discovery. It only refuses the fetch.

So a disallowed URL is a URL the engine knows about and is forbidden from examining. Google's own documentation states the consequence plainly: robots.txt is not a mechanism for keeping a page out of search results, and a blocked URL that other pages link to can still be indexed, just without a description, because the crawler was never allowed to read one.

Two directives, two states each, four outcomes. Only one of them is a removal.

Row four is where sites get stuck. The directive that would remove the page is sitting inside a document the crawler has been forbidden to open.

The failure is common enough that Search Console has a dedicated status for it. Someone finds an internal search results page or a thin tag archive in the index, wants it gone, and does the thorough thing: noindex on the template and a Disallow in robots.txt. Both directives, maximum coverage.

The crawler now stops fetching the URL, so it never sees the noindex. The page was already indexed, so it stays indexed and can keep appearing as a bare listing. Weeks later it's still there, so the team adds a broader disallow, which makes the problem harder rather than easier.

Google Search Console distinguishes these cases in the Pages report, and the labels are precise:

"Indexed, though blocked by robots.txt" is the alarm. The fix is counterintuitive and it is the whole point of this post: remove the block so the crawler can come in and read the noindex you already wrote.

The question is never which directive is better. It's what you're actually trying to prevent.

The canonical row is the one that quietly costs the most. Teams reach for noindex on duplicate product or location pages when what they wanted was consolidation. noindex throws the page away. A canonical folds it into the version you kept.

Free, and it takes about twenty minutes. Every number below is an illustrative example, not SparkCliks data.

Step 1: pull the three statuses that matter. In Google Search Console, open Indexing, then Pages, and read the "Why pages aren't indexed" table. Find these three rows and write down each count:

• Blocked by robots.txt
• Indexed, though blocked by robots.txt
• Excluded by noindex tag

Say you see 2,140 blocked, 61 indexed though blocked, and 380 excluded by noindex. The 61 is your work list. It's small, it's specific, and it's the only one of the three that represents pages doing something you did not ask for.

Step 2: export and read the 61. Click the status, then use the export button. Ask one question per URL: did I ever want this indexed? If yes, remove the disallow and leave it alone. If no, remove the disallow and serve noindex. Both answers start with removing the disallow.

Step 3: confirm what the crawler sees, not what your browser sees. Paste a sample URL into the search bar at the top of Search Console to open URL Inspection, then click Test live URL. Read the two fields named Crawl allowed? and Indexing allowed?. Those are the two directives, reported separately, for that exact URL. A page you intend to remove should read crawl allowed Yes and indexing allowed No. Any other combination is a misconfiguration.

Step 4: read the robots.txt report. Under Settings, open the robots.txt report to see the fetched file, its last fetch time and any parse errors. The old robots.txt Tester was retired in December 2023, so guides pointing you at it are out of date.

Step 5: run the intersection check almost nobody runs. Take the URL list from your XML sitemap and the disallow rules from robots.txt, and find the overlap. Every URL that appears in both is a direct contradiction: you're asking the engine to crawl a page you've forbidden it to crawl. Then do the same for noindex, since a noindex URL sitting permanently in your sitemap is the same mixed message pointed the other way. Two lists, one comparison, and it surfaces conflicts no single report will show you.

The order is the part that goes wrong. Run it top to bottom.

1. Decide the destination state first. Gone forever (return 404 or 410), live for users but out of search (noindex), or merged into another URL (301 or canonical). These need different work and produce different reports.
2. Remove any robots.txt disallow covering the URL. Do this before anything else. Every step below depends on the crawler being able to fetch the page.
3. Serve the noindex. Verify it with a raw fetch or view-source, not by trusting your CMS toggle. Check the response headers as well as the HTML, because a stray X-Robots-Tag at the server can contradict your meta tag.
4. Leave the URL in your sitemap for now. It's a mixed signal long term, and a deliberate accelerator short term: the sitemap entry helps the engine come back and see the new directive sooner. Remove the entry once the page has dropped.
5. Request indexing via URL Inspection for your handful of highest-priority URLs. It's a queue nudge, not a guarantee, and it doesn't scale past a few pages.
6. If it's urgent, add the Removals tool on top. It hides the URL for about six months, which buys time while noindex does the permanent work. Used alone it's a snooze button, and the page returns when it expires.
7. Verify the status flipped. In the Pages report, the URL should move to "Excluded by noindex tag". That's confirmation the engine fetched the page and obeyed.
8. Only now, consider re-adding a robots.txt disallow, and usually don't. It's defensible for a large set of URLs where crawl waste is real and you've confirmed every one has dropped. For a handful of pages it buys nothing and re-arms the trap.

Timing is the part to set expectations on. Recrawl frequency varies from about a day for a heavily linked page to months for a deep, rarely linked one. There's no setting that changes this. If a page hasn't been rechecked, no directive on it has taken effect yet.

A PDF has no . Neither does an image, a CSV export or a JSON endpoint. All of them can be indexed. The HTTP header is how you apply robots directives to them.

Nginx, applied to PDFs:

location ~* \.pdf$ {
    add_header X-Robots-Tag "noindex, nofollow";
}

Apache, the same rule:

<FilesMatch "\.pdf$">
    Header set X-Robots-Tag "noindex, nofollow"
</FilesMatch>

Two cautions. The header carries exactly the same crawl requirement as the meta tag, so a blocked URL means an unread header and the trap applies identically. And because these rules are set at the server, they're easy to scope too broadly. A server-level noindex meant for a staging environment, shipped to production by a config copy, is one of the fastest ways to remove an entire site from search. Grep your server config for X-Robots-Tag the same way you'd grep your templates for noindex.

This topic has an unusually long tail of stale guidance still ranking.

The first row is the expensive one. A noindex line in robots.txt has done nothing since September 2019, and it fails silently: the file parses, the report shows no error, and the page stays indexed.

Robots.txt is a public file. Anyone can read yours, and people do. Listing a path there announces that the path exists and that you'd rather nobody looked at it, which is the opposite of concealment.

Look back at the real file quoted earlier: it disallows /admin and /api. That's fine, because those routes are protected by authentication. The disallow is a crawl-efficiency measure, and the authentication is the actual control. The failure mode is a site that has the disallow and not the authentication, treating the robots file as the lock.

noindex hides nothing either. The page still returns 200 to anyone holding the URL, it still renders, and it can still be shared, archived and scraped. It's an instruction about search results, not about access.

If content genuinely must not be reachable, the controls are HTTP authentication, a login wall, an IP allowlist, or not publishing it. Use robots.txt and noindex for what they are: crawl management and index management, in that order.

Indexed or not indexed is a binary, and it sits upstream of everything else. Whether anyone then picks your listing out of ten results is a separate question with a far less settled answer: what search engines say about click data as a ranking signal. SERP Clicks states the same limit in its own FAQ, because no click behavior of any kind moves a page that isn't in the index to begin with.