Skip to the indexed answers
Digital Marketing Index

usdigitalmarketingindex.com SEO

How can two site crawl snapshots reveal unintended changes after a code release?

By crawling the site with identical settings before and after the release, joining the two results on URL, and listing every page whose recorded attributes differ. A release is meant to change certain things. A crawl comparison shows everything that changed, including what nobody intended: a canonical tag that now points elsewhere on one template, titles regenerated from a different field, a block of internal links that disappeared, pages that began redirecting. None of these raises an error or shows in a visual check of a few pages. In a diff they appear as rows. The work then is to separate the intended changes from the regressions, since a difference is a finding only once it has been compared with what the release was supposed to do.

Matched URL universe

A comparison is meaningful only if the two crawls examined the same things in the same way.

Setting Keep identical
Start points Same seed URLs or the same URL list
Scope Same hostnames, same include and exclude rules
User agent Same
JavaScript rendering Same mode in both
Robots handling Same
Crawl depth and limits Same
Authentication and cookies Same
Location Same origin, if content varies by location
Time As close to the release as practical on both sides

Two modes are useful together.

Mode Shows
List crawl: a fixed list of URLs fetched in both snapshots Attribute changes on known pages, unaffected by link structure
Spider crawl: following links from the home page Changes in what is reachable, and in depth

After crawling, classify every URL.

Class Meaning First question
In both Present before and after Did its attributes change?
Only before No longer found Removed, redirected, or orphaned?
Only after Newly found Intended new page, or accidental URL such as a parameter or duplicate?

“Only before” in a spider crawl does not mean the page is gone. It may exist with nothing linking to it. Request those URLs directly to find out.

Take the “before” snapshot from production just ahead of the release, and where possible a second from staging with the release applied, so that differences are seen before launch.

Status and canonical diffs

Compare the attributes that govern crawling and indexing first.

Attribute A change means
HTTP status Pages now erroring or redirecting, or errors now returning 200
Redirect target Redirects pointing somewhere new
Canonical tag The declared preferred URL changed
Meta robots and X-Robots-Tag Indexing instructions changed
robots.txt allow or block Crawl access changed
Indexability, derived The combined effect of the above
hreflang Language annotations changed
Pagination links Series structure changed
Content type and response headers Delivery changed
Response time Performance changed

Summarize by template, not only by URL. A difference on 4,000 URLs is one defect if they share a template. Group rows by URL pattern and count.

A change-diff table design, with examples:

URL pattern Attribute Before After URLs affected Intended? Severity
<!–INLINECODE0–> Canonical Self Self, with <!–INLINECODE1–> appended 1,840 No Critical
<!–INLINECODE2–> Meta robots index, follow noindex, follow 212 No Critical
<!–INLINECODE3–> Status 200 301 to <!–INLINECODE4–> 640 Yes, planned move None; verify targets
<!–INLINECODE5–> Title Category and brand Category only 55 No Major
<!–INLINECODE6–> Structured data Product present Product present, new property added 1,840 Yes None
/about/ Word count 420 415 1 Yes, copy edit None
<!–INLINECODE7–> Found in crawl No Yes 3,100 No Major: new crawlable URLs
All Response time Baseline 5% slower All Unknown Minor; monitor

The table holds benign rows beside critical ones on purpose. A planned move and a copy edit are differences too, and the report must not present them as failures.

Broken links and metadata changes

Next, the attributes that describe content and connections.

Attribute A change means
Title Regenerated, truncated, emptied, or duplicated
Meta description Changed or emptied
H1 and heading outline Structure changed
Word count or content hash Body content added or removed
Inlinks Pages gaining or losing internal links
Outlinks Links removed from templates
Anchor text Link wording changed
Click depth Pages moved further from the home page
Broken internal links Links now leading to errors
Links to redirects Internal links not updated
Images Missing images, lost alt text
Structured data types Markup added, dropped, or altered
Rendered against served Content now dependent on scripts

Signs of a template-level regression:

  • the same attribute changed on every URL of a pattern;
  • titles or H1s that became identical across pages;
  • word counts falling by a similar amount everywhere on a template, suggesting a removed block;
  • inlink counts collapsing for a group of pages, suggesting a removed navigation element;
  • duplicate counts rising.

Thresholds reduce noise. Small differences in word count or response time occur between any two crawls. Flag changes above a set size, and always flag changes in status, canonical, and robots directives whatever their size.

Issue triage

Each difference gets a decision. When the list is longer than the team can fix in one cycle, the order comes from a separate exercise, covered in the guide to prioritizing technical SEO tasks when engineering capacity is limited.

  1. Compare with the release notes and the change log. Is the difference explained by a planned change?
  2. If intended, verify it was done correctly, then close.
  3. If not intended, classify severity.
  4. Find the cause: which commit, template, or configuration.
  5. Assign, fix, and re-crawl the affected pattern.

Severity guide:

Severity Examples Response
Critical <!–INLINECODE8–> added; canonical pointing to wrong URLs; important pages returning errors; robots.txt blocking Fix or roll back immediately
Major Titles or H1s lost; internal links removed from templates; structured data dropped; new crawlable duplicate URLs Fix in the next release, days not weeks
Minor Small content shifts; slight performance change; cosmetic metadata differences Schedule or accept
None Intended and correct Record

A triage record:

Difference Intended Severity Cause Owner Action Verified
Canonical with parameter on products No Critical Navigation component appends tracking parameter, canonical built from request URL Developer Build canonical from clean path Re-crawl of /products/

Reporting rules:

  • Report regressions and intended changes in separate sections.
  • State the crawl settings and dates.
  • Give counts by template, with example URLs.
  • Do not report a difference as a failure until it has been checked against the planned changes.
  • Note what the crawl could not see: pages behind logins, content varying by location, and anything outside the crawl’s scope.

Make it routine. Keep a saved snapshot before every release. The comparison takes little time when the previous crawl exists, and it cannot be done at all when it does not.

A crawl shows the site as a crawler fetched it on those two occasions. It does not show how search engines responded. Follow critical findings into Search Console’s indexing reports.

Structured data regressions introduced by a plugin update are one class of release defect with its own diagnosis, described in the guide to how a plugin update can introduce structured data errors across an entire site.

The method in order: crawl the same URLs the same way before and after, join on URL, group differences by template, check each against what was planned, and treat as defects only the ones nobody intended.

Link trail

Reader comments on this answer

Add your comment to this answer

Your email address will not be published. Required fields are marked *