usdigitalmarketingindex.com SEO
How can you verify a noindex directive delivered through an HTTP X-Robots-Tag header?
Updated November 15, 2026
Request the URL and read the response headers for a line beginning X-Robots-Tag that contains noindex, confirm that the same header is sent on a normal GET request and to Googlebot, and then confirm in Search Console that Google has seen it. A header cannot be checked by viewing a page’s source, because it is not in the page. It travels with the response. The header is the only way to apply noindex to files that have no HTML head, such as PDFs and images, and it works only if crawlers are allowed to fetch the URL.
Where X-Robots-Tag appears
Google’s documentation describes two equivalent ways to deliver noindex: a meta tag in the HTML, or an HTTP response header. Of the header it says “you can return an X-Robots-Tag HTTP header with a value of either noindex or none in your response”, and that “A response header can be used for non-HTML resources, such as PDFs, video files, and image files” (Google’s documentation on blocking indexing with noindex).
The header is set in server or application configuration, so it can apply to a single file, a directory, or a file type. That reach is also the risk. A rule written for one folder can match more than intended, and nothing visible on the pages will show it.
Inspecting headers
A response carrying the directive looks like this. The response is an illustration.
HTTP/1.1 200 OK
Content-Type: application/pdf
X-Robots-Tag: noindex
Content-Length: 482113
To check a real URL from the command line, request the headers:
$ curl -sI https://www.example.com/files/price-list.pdf
That sends a HEAD request. A server is not obliged to send identical headers for HEAD and GET, and some configurations apply rules to one and not the other. Confirm with a GET request that discards the body and prints the headers:
$ curl -s -o /dev/null -D - https://www.example.com/files/price-list.pdf
Then check what a Google crawler would be sent, by setting the user agent. This shows whether the server varies its response by client:
$ curl -s -o /dev/null -D - -A "Googlebot" https://www.example.com/files/price-list.pdf
A request from your machine with that user agent is not treated by every server as Googlebot, since some sites verify crawlers by IP address. Use it to catch user-agent rules, and rely on Search Console for the definitive view.
What to read in the output:
- The status line. The header matters on a 200 response. On a redirect, the header that counts is the one on the final URL.
- The X-Robots-Tag line or lines. There may be more than one.
- The value. noindex, or none, which combines noindex and nofollow.
- Any user-agent prefix. A value such as googlebot: noindex applies to that crawler only. Without a prefix it applies to all.
- Other robots directives. Google’s documentation says that where rules conflict, the more restrictive applies (Google’s robots meta tag documentation).
Browser developer tools show the same information: open the network panel, load the URL, select the request, and read the response headers.
Crawler access requirements
A header is read only when the URL is fetched. Google states the condition: “For the noindex rule to be effective, the page or resource must not be blocked by a robots.txt file, and it has to be otherwise accessible to the crawler”.
Check three things.
- robots.txt does not disallow the URL. A blocked file is never requested, so its header is never seen, and the URL can still be listed if other pages link to it.
- The URL does not require a login or return an error to crawlers.
- No firewall or bot rule refuses Google’s requests.
One mistake to avoid is adding the header and blocking the path in robots.txt for good measure. The two cancel: the block prevents the header from working. Note also that the same documentation says specifying noindex inside robots.txt is not supported.
Post-change validation
After the header is deployed, confirm with the six checks below.
| Level | Check | Pass condition |
|---|---|---|
| Server | GET request, read headers | X-Robots-Tag: noindex present on the intended URLs |
| Scope | Same request against URLs that should stay indexed | Header absent |
| Crawler view | URL Inspection in Search Console, live test | Indexing allowed reads No, with noindex detected in the X-Robots-Tag HTTP header |
| Access | robots.txt test for the URL | Crawling allowed |
| Result over time | Page indexing report | URL moves to excluded by noindex after it is recrawled |
| Search | Search for the exact URL after some weeks | No longer listed |
The scope row deserves equal care. Test a sample of URLs outside the intended set, including the home page and a main template, to be sure the rule has not spread.
Timing: nothing changes in search results until Google recrawls the URL. For a file that is fetched infrequently this can take a while. URL Inspection can request a recrawl for individual URLs.
To reverse the directive later, remove the header and confirm its absence with the same GET request. A cached configuration at a CDN can keep sending the old header after the origin has changed, so test the public URL, not the origin.
Keep a record: which rule was added, where, the URLs it is meant to match, the date, and the result of each check. Header-level directives are invisible in content reviews, and the record is the only place a later investigator will find them.
Checking one URL by hand confirms a directive that was set on purpose. Watching for one that appears by accident, across a whole site, is a monitoring task, described in the guide to how alerts can detect accidental noindex tags added to production webpages.
The summary: read the headers on a real GET response, make sure crawlers can fetch the URL, and let Search Console confirm what Google received.
Add your comment to this answer