Skip to the indexed answers
Digital Marketing Index

usdigitalmarketingindex.com SEO

How can alerts detect accidental noindex tags added to production webpages?

By fetching a watchlist of important URLs on a schedule, reading the indexing directives each one returns, in the HTML and in the response headers, and raising an alert the moment a page that should be indexable carries a noindex. A noindex instruction is one short string, and it can reach production in several ordinary ways: a staging setting left switched on at launch, a CMS checkbox ticked on a template, a plugin update that changes a default, a header rule added at the server. Nothing on the page looks different to a visitor. Google’s documentation states the consequence: when Googlebot crawls the page and extracts the rule, “Google will drop that page entirely from Google Search results, regardless of whether other sites link to it” (Google’s documentation on blocking indexing with noindex). A monitor that checks daily or hourly finds the directive before it has been acted on across the site.

Critical URL watchlist

Monitoring every URL continuously is unnecessary. Monitoring the right sample catches template-level faults at once.

Include Why
The home page Highest consequence
One or more URLs from every template A template fault appears on its sample
The highest-traffic landing pages Direct revenue exposure
One URL from each section and language Section and locale settings
Key non-HTML files that should be indexed, such as PDFs Header directives apply to these
A few URLs that should carry noindex To detect the opposite fault: a directive removed

A watchlist entry:

URL Template Expected directive Priority
/ Home Indexable Critical
/boiler-repair/ Service Indexable Critical
/guides/boiler-pressure/ Guide Indexable High
/products/thermostat-t200/ Product Indexable Critical
/blog/ Listing Indexable High
/downloads/installation-guide.pdf Document Indexable Medium
/search?q=test Internal search noindex Medium
/account/login/ Account noindex Low

Keep the list in version control. Add an entry whenever a new template launches.

Supplement the watchlist with a full crawl at a lower frequency, to catch faults on pages that share no template with any watched URL.

Rendered and header directives

A noindex can arrive by more than one route, and the monitor must read each.

Google’s documentation describes the two forms: noindex is “a rule set with either” a meta tag “or HTTP response header”. It adds that the header form “can be used for non-HTML resources, such as PDFs, video files, and image files”.

Location Form How to check
HTML head, served A meta element named robots, or named for a specific crawler such as googlebot, with noindex in its content Fetch the page; parse the head
HTML head, rendered The same element inserted or changed by JavaScript Render the page; parse the resulting DOM
HTTP response header An X-Robots-Tag header containing noindex Read the response headers
Header for a specific crawler X-Robots-Tag naming a user agent Read the header value in full

Details the check must handle:

  • More than one robots meta element can be present. Any one containing noindex counts.
  • The value “none” is equivalent to noindex with nofollow.
  • Directives can be addressed to a named crawler. Check the general robots name and the search engine names.
  • A script can add a noindex after load. The served HTML is then clean and the rendered HTML is not.
  • Headers can be added by the application, the web server, or a CDN. Fetch through the public address, not the origin.
  • Responses can differ by user agent. Fetch with an ordinary browser user agent and with a search crawler user agent, and compare.

Related states to detect in the same pass, since they have similar effect:

State Check
Canonical pointing to a different URL Compare the canonical with the URL fetched
Status code not 200 Read the status
Blocked by robots.txt Test the path against the live file
Redirect Read the status and location

One interaction is worth a rule of its own. The documentation says that for a noindex rule to be effective, the page “must not be blocked by a robots.txt file”, since a blocked page is never fetched and the rule is never seen. So a page that is both blocked and marked noindex is in an inconsistent state, and the monitor should report it.

The reverse task, keeping a staging copy out of search on purpose, is covered in the guide to how a public staging website can be prevented from entering Google’s index.

Detection cadence

Check Frequency Scope
Watchlist fetch: status, meta robots, X-Robots-Tag, canonical Hourly, or every few minutes for the critical tier Watchlist
Rendered check Daily, and after each release Watchlist
Post-deployment check Immediately after every release Watchlist
Full-site crawl for directives Weekly All URLs
Search Console page indexing report, "Excluded by noindex tag" Weekly Site
Sitemap cross-check: any sitemap URL returning noindex Daily Sitemap URLs

The release-triggered check matters as much as the schedule. A deployment is a moment at which an accidental directive can arrive, and a check wired into the pipeline’s final step finds it within minutes.

A sample alert rule:

ALERT unexpected-noindex
  FOR   each url in watchlist where expected = indexable
  FETCH url (public address, no cache), user agents: browser, Googlebot
  WHEN  meta robots (served or rendered) contains "noindex" or "none"
     OR X-Robots-Tag header contains "noindex" or "none"
  THEN  severity = url.priority
        notify: on-call developer, SEO owner
        include: url, template, where found (meta served / meta rendered /
                 header), exact value, time first seen, last deployment ID
  REPEAT every 30 minutes until cleared

The alert names where the directive was found. That detail directs the fix: a served meta tag points to the template or CMS setting, a rendered-only tag to a script, a header to server or CDN configuration.

Avoid false alarms by keeping expected-noindex pages in the list with their expected state, so they do not fire, and by requiring two consecutive detections for lower-priority URLs.

Incident response

An escalation playbook for a single affected template.

Step Action Owner
1 Confirm: fetch the alerted URL manually; read source, rendered DOM, and headers On-call developer
2 Scope: fetch other URLs of the same template, and one from each other template On-call developer
3 Identify the route: served meta, rendered meta, or header Developer
4 Identify the cause: compare with the last deployment, CMS settings, plugin updates, server and CDN rules Developer
5 Fix or roll back Developer, with release manager
6 Verify: refetch the watchlist; all expected-indexable URLs clean SEO owner
7 Purge caches so no cached copy still carries the directive Developer
8 Establish exposure: when did the directive first appear? Check deployment time and monitor history SEO owner
9 Check Google's state: URL Inspection on sample URLs; page indexing report for the template's URLs SEO owner
10 Request recrawling: resubmit the sitemap; use URL Inspection for the most important URLs SEO owner
11 Monitor: indexed count for the template over the following weeks SEO owner
12 Record: change log entry and incident note; add any missed template to the watchlist SEO owner

Escalation times, to be set by the organization: how long a critical alert may stand before it goes to the next person, and who that is at each level.

Exposure determines how much recovery is needed. A directive live for minutes may never have been fetched. One live for days will have been seen on the pages Google crawled in that time. Pages removed from the index return after Google recrawls them without the directive, and how long that takes varies by page.

Prevent the routes found. Environment-specific settings should come from configuration that cannot be promoted from staging to production. Staging is better protected by authentication than by noindex, which removes the setting that gets carried over. The release checklist includes the watchlist check.

Confirming the header form of the directive on a single URL, by hand, has its own procedure, described in the guide to how to verify a noindex directive delivered through an HTTP X-Robots-Tag header.

The rule: watch one URL from every template and every top landing page, read both the page and its headers, served and rendered, alert on any unexpected noindex, and check again after every release.

Link trail

Reader comments on this answer

Add your comment to this answer

Your email address will not be published. Required fields are marked *