Skip to the indexed answers
Digital Marketing Index

usdigitalmarketingindex.com SEO

Why can a URL blocked by robots.txt still appear in Google search results?

Because robots.txt controls crawling, and appearing in search results depends on indexing, which is a different step. A disallow rule tells Google not to fetch the page. It does not tell Google to forget that the URL exists. If other pages link to the blocked URL, Google can learn the address from those links and list it, without ever reading the page. Google’s documentation says so directly: a robots.txt file “is not a mechanism for keeping a web page out of Google”, and “A page that’s disallowed in robots.txt can still be indexed if linked to from other sites” (Google’s introduction to robots.txt).

How robots directives work

A robots.txt file sits at the root of a host and lists which paths crawlers may request. An example:

User-agent: *
Disallow: /internal-reports/

This asks every compliant crawler not to request any URL whose path begins with /internal-reports/. That is the whole of its effect. It is an instruction about fetching. Google’s documentation describes the file’s main purpose as avoiding overloading a site with requests.

Two things it does not do. It does not hide the URL, since the file itself is public and lists the path. And it does not carry any instruction about the index.

Why blocked URLs can be known

A URL becomes known to Google through links and sitemaps, not only through crawling the URL itself.

The documentation explains the consequence: “While Google won’t crawl or index the content blocked by a robots.txt file, we might still find and index a disallowed URL if it is linked from other places on the web”. The result can then appear with its address and, as the same passage notes, other public information such as the anchor text of links pointing to it.

A hypothetical scenario shows the sequence.

  1. A site blocks /internal-reports/ in robots.txt.
  2. A partner’s public page links to /internal-reports/q3-summary/ with the link text Q3 summary report.
  3. Google crawls the partner’s page and finds the link.
  4. Google does not fetch the report, because of the disallow rule.
  5. Google knows the URL and the words used to link to it. For a search matching those words, the URL may be listed, without a description drawn from the page.

The block worked as designed: the page was not crawled. The listing happened anyway.

There is a second effect that catches site owners. Because the page cannot be fetched, Google cannot see a noindex rule placed on it. Google’s documentation on robots meta tags states that such settings “can be read and followed only if crawlers are allowed to access the pages that include these settings” (Google’s robots meta tag documentation). Adding noindex to a page that is also disallowed achieves nothing.

Appropriate removal mechanisms

Choose the tool by the goal.

Goal Mechanism Note
Reduce crawler requests to a section robots.txt disallow Does not prevent listing
Keep a page out of search results noindex, in a meta tag or HTTP header The page must remain crawlable so the rule can be read
Keep content private Authentication Content behind a login cannot be fetched or shown
Take a page down Remove it and return 404 or 410 The URL drops out after it is recrawled
Hide a result quickly while a permanent fix is made Search Console's removals tool Temporary; pair it with one of the above

The documentation names the first two alternatives itself: to keep a page out of Google, “block indexing with noindex or password-protect the page”.

For a URL that is already listed and blocked, the order of operations matters.

  1. Remove the disallow rule for that URL, so it can be crawled.
  2. Add noindex, or remove the page, or put it behind a login.
  3. Wait for Google to recrawl and process the change.
  4. Only then, if crawling still needs limiting, consider restoring a disallow rule.

Skipping step 1 defeats the fix. The noindex is added, the block stays, and nothing changes.

Safe verification

Check the state of the URL with tools that report what Google sees.

  • Use the URL Inspection tool in Search Console. It reports whether crawling is allowed and whether the page is indexed. For a blocked URL, the indexing report category to look for is the one describing pages indexed though blocked by robots.txt.
  • Test the URL against the live robots.txt file to confirm which rule applies.
  • After making changes, request indexing and check again later. Nothing changes until the URL is recrawled.
  • Search for the exact URL to see how it is listed, and record the date.

Do not verify by assuming. A rule that looks right in the file may not match the URL as written, because of case, a trailing slash, or a more specific rule elsewhere in the file.

No method here promises a date. Removal depends on recrawling, and Google gives no schedule for a single URL.

A blocked URL that is merely unwanted in results is a tidiness problem. A URL exposing confidential information is an incident, with further steps and more urgency, set out in the guide to permanently removing a confidential URL from organic search results.

The distinction to remember: robots.txt says do not fetch. Noindex says do not list. Google has to be able to fetch a page to be told not to list it.

Link trail

Reader comments on this answer

Add your comment to this answer

Your email address will not be published. Required fields are marked *