Skip to the indexed answers
Digital Marketing Index

usdigitalmarketingindex.com SEO

How can a release pipeline detect robots.txt changes that unexpectedly block site sections?

By testing the candidate robots.txt against a fixed list of important URLs before it is deployed, and failing the release if any URL’s crawl permission differs from what is expected. The test evaluates the file the way a crawler does: it selects the rule group that applies to a given crawler, finds the rule that matches each URL path, and reports allowed or blocked. A plain text comparison of old and new files is not enough. One added line can block thousands of pages, and a harmless-looking reordering can change which group a crawler follows. The pipeline needs two things: fixtures, meaning URLs with their expected results, and a parser that applies the real matching rules. With those in place, a blocking change is caught as a failed build instead of as a traffic decline weeks later.

Baselining allowed and blocked routes

Write down what the site intends. The baseline is a list of representative URL paths, one or more for every section and template, each with the expected result for the crawlers that matter.

Test fixtures, for a hypothetical site:

# path                                   user-agent     expected
/                                        Googlebot      allow
/boiler-repair/                          Googlebot      allow
/guides/boiler-pressure/                 Googlebot      allow
/products/thermostat-t200/               Googlebot      allow
/products/thermostat-t200/?color=white   Googlebot      allow
/blog/                                   Googlebot      allow
/blog/page/2/                            Googlebot      allow
/images/boiler-gauge.jpg                 Googlebot-Image allow
/assets/app.css                          Googlebot      allow
/assets/app.js                           Googlebot      allow
/sitemap.xml                             Googlebot      allow
/cart/                                   Googlebot      block
/checkout/                               Googlebot      block
/search?q=boiler                         Googlebot      block
/account/                                Googlebot      block
/admin/                                  *              block

Build the list from:

  • one URL per template and per top-level section;
  • URLs with query parameters that should be crawlable;
  • static resources needed for rendering, such as CSS and JavaScript;
  • images and other media;
  • paths that must stay blocked, so that an accidental unblocking is caught too;
  • the highest-traffic landing pages, from analytics or Search Console.

Include each crawler the file addresses by name. A rule under a named group applies to that crawler only.

Keep the fixtures in version control beside the robots.txt source. A change to the intended policy is then a change to the fixtures, reviewed with the change to the file.

Automated robots diff checks

Two checks run on every build that touches the file or the template that generates it.

Check 1: semantic test. Parse the candidate file with a library that implements the robots exclusion protocol’s matching rules, evaluate every fixture, and compare with the expected column.

The matching rules the test must respect, from Google’s documentation (Google’s robots.txt specification):

Rule Detail
Group selection "Only one group is valid for a particular crawler", chosen as the group with the most specific matching user agent. "Other groups are ignored"
Rule precedence Crawlers "use the most specific rule based on the length of the rule path"
Conflicts "In case of conflicting rules, including those with wildcards, Google uses the least restrictive rule"
Case Path matching is case-sensitive
Wildcards An asterisk matches any sequence of characters; a dollar sign marks the end of the URL
Merging Multiple groups for the same user agent are merged

These are why string matching fails. Three examples:

# 1. A new group changes which rules Googlebot follows.
User-agent: *
Disallow: /admin/

User-agent: Googlebot
Disallow: /tmp/
# Googlebot now follows only its own group. /admin/ is no longer
# blocked for Googlebot, though the first line is unchanged.

# 2. A short rule that matches far more than intended.
Disallow: /p
# Blocks /products/, /pricing/, /press/ and every path beginning /p.

# 3. A longer Allow overrides a shorter Disallow.
Disallow: /blog/
Allow: /blog/guides/
# /blog/guides/x is allowed; /blog/news/x is blocked.

A diff shows that lines changed. Only evaluation against real paths shows what the change does.

Check 2: textual diff, for human review. Show the reviewer the lines added and removed, with the fixture results beside them.

Further automated checks:

Check Fails when
File present and non-empty Missing or zero length
Syntax Unknown directives or malformed lines
Site-wide disallow A <!–INLINECODE0–> applies to the general group or to a named search crawler
Environment The staging file, which blocks everything, is about to be deployed to production
Sitemap line Missing, or pointing to a non-production host
Size Larger than the limit crawlers will read
Fixture coverage A new top-level section exists with no fixture

The sitemap deserves the same treatment as robots.txt: a stored baseline and a comparison after each release. That check is described in the guide to detecting unexpected XML sitemap changes after a CMS update.

Release rejection rules

Decide in advance what blocks a release.

Condition Action
Any fixture expected allow evaluates as block Reject
Any fixture expected block evaluates as allow Reject, unless the fixture was updated in the same change with approval
Site-wide disallow for a search crawler Reject
Rendering resources blocked Reject
File missing or unparsable Reject
Staging file detected in a production build Reject
File changed and no fixtures changed Pass if all fixtures pass; flag for review
Fixtures changed Require approval from the SEO owner

An example alert condition:

ALERT robots-blocking-change
  WHEN  build.robots_txt changed
  AND   any fixture where expected = allow AND result = block
  THEN  fail build
        notify: SEO owner, release manager
        message: "robots.txt change blocks {count} expected-allow paths:
                  {paths}. Matching rule: {rule} in group {group}."

The message names the paths, the rule that matched, and the group, so that the author can see at once what to fix.

Approvals: a change to robots.txt or its fixtures should need a named reviewer who understands the file. It is a small file with site-wide effect.

Production verification

The pipeline tests the candidate. Production needs its own check, because the live file can differ from what was built: a CDN rule, a server override, a plugin that generates the file dynamically, or a manual edit.

After each deployment:

  1. Fetch the live robots.txt from the production host, on every hostname and protocol in use.
  2. Confirm the response status is 200 and the content type is plain text.
  3. Compare the fetched content with the deployed version.
  4. Run the fixtures against the fetched content.
  5. Record the result.

Continuous monitoring:

Monitor Frequency Alert when
Live file content hash Every few minutes to hourly Changes outside a release
Fixtures against the live file Hourly Any expected-allow path is blocked
HTTP status of robots.txt Every few minutes Not 200
Search Console robots.txt report After changes Fetch errors, or a version Google fetched that differs from intended
Page indexing report, "blocked by robots.txt" Weekly Count rises
URL Inspection on a sample of priority URLs After changes Crawl not allowed

The status check matters as much as the content. Google’s documentation describes how server errors on the robots.txt file affect crawling, so an unavailable file is an incident in itself.

Timing: the documentation says Google “generally caches the contents of robots.txt file for up to 24 hours”. A bad file may be acted on for up to a day after it is corrected, and a correction is not instantaneous. That is the argument for catching the change before release.

If a blocking change reaches production:

  1. Restore the previous file immediately.
  2. Verify by fetching it.
  3. Check which URLs were affected, using the fixtures and server logs.
  4. Watch the page indexing report over the following weeks.
  5. Add a fixture for the path that was missed.
  6. Record the incident in the change log.

Remember what the file does. A disallow stops crawling. It does not remove a URL from the index, and a blocked URL cannot have its noindex tag read. Detecting that directive when it appears by accident is a separate monitoring task, described in the guide to how alerts can detect accidental noindex tags added to production webpages.

The rule: keep a list of URLs with their expected crawl permission, evaluate every candidate robots.txt against it with a real parser, reject any release that changes a result unintentionally, and run the same test against the live file.

Link trail

Reader comments on this answer

Add your comment to this answer

Your email address will not be published. Required fields are marked *