usdigitalmarketingindex.com SEO
How can two site crawl snapshots reveal unintended changes after a code release?
Updated November 15, 2026
By crawling the site with identical settings before and after the release, joining the two results on URL, and listing every page whose recorded attributes differ. A release is meant to change certain things. A crawl comparison shows everything that changed, including what nobody intended: a canonical tag that now points elsewhere on one template, titles regenerated from a different field, a block of internal links that disappeared, pages that began redirecting. None of these raises an error or shows in a visual check of a few pages. In a diff they appear as rows. The work then is to separate the intended changes from the regressions, since a difference is a finding only once it has been compared with what the release was supposed to do.
Matched URL universe
A comparison is meaningful only if the two crawls examined the same things in the same way.
| Setting | Keep identical |
|---|---|
| Start points | Same seed URLs or the same URL list |
| Scope | Same hostnames, same include and exclude rules |
| User agent | Same |
| JavaScript rendering | Same mode in both |
| Robots handling | Same |
| Crawl depth and limits | Same |
| Authentication and cookies | Same |
| Location | Same origin, if content varies by location |
| Time | As close to the release as practical on both sides |
Two modes are useful together.
| Mode | Shows |
|---|---|
| List crawl: a fixed list of URLs fetched in both snapshots | Attribute changes on known pages, unaffected by link structure |
| Spider crawl: following links from the home page | Changes in what is reachable, and in depth |
After crawling, classify every URL.
| Class | Meaning | First question |
|---|---|---|
| In both | Present before and after | Did its attributes change? |
| Only before | No longer found | Removed, redirected, or orphaned? |
| Only after | Newly found | Intended new page, or accidental URL such as a parameter or duplicate? |
“Only before” in a spider crawl does not mean the page is gone. It may exist with nothing linking to it. Request those URLs directly to find out.
Take the “before” snapshot from production just ahead of the release, and where possible a second from staging with the release applied, so that differences are seen before launch.
Status and canonical diffs
Compare the attributes that govern crawling and indexing first.
| Attribute | A change means |
|---|---|
| HTTP status | Pages now erroring or redirecting, or errors now returning 200 |
| Redirect target | Redirects pointing somewhere new |
| Canonical tag | The declared preferred URL changed |
| Meta robots and X-Robots-Tag | Indexing instructions changed |
| robots.txt allow or block | Crawl access changed |
| Indexability, derived | The combined effect of the above |
| hreflang | Language annotations changed |
| Pagination links | Series structure changed |
| Content type and response headers | Delivery changed |
| Response time | Performance changed |
Summarize by template, not only by URL. A difference on 4,000 URLs is one defect if they share a template. Group rows by URL pattern and count.
A change-diff table design, with examples:
| URL pattern | Attribute | Before | After | URLs affected | Intended? | Severity |
|---|---|---|---|---|---|---|
| <!–INLINECODE0–> | Canonical | Self | Self, with <!–INLINECODE1–> appended | 1,840 | No | Critical |
| <!–INLINECODE2–> | Meta robots | index, follow | noindex, follow | 212 | No | Critical |
| <!–INLINECODE3–> | Status | 200 | 301 to <!–INLINECODE4–> | 640 | Yes, planned move | None; verify targets |
| <!–INLINECODE5–> | Title | Category and brand | Category only | 55 | No | Major |
| <!–INLINECODE6–> | Structured data | Product present | Product present, new property added | 1,840 | Yes | None |
| /about/ | Word count | 420 | 415 | 1 | Yes, copy edit | None |
| <!–INLINECODE7–> | Found in crawl | No | Yes | 3,100 | No | Major: new crawlable URLs |
| All | Response time | Baseline | 5% slower | All | Unknown | Minor; monitor |
The table holds benign rows beside critical ones on purpose. A planned move and a copy edit are differences too, and the report must not present them as failures.
Broken links and metadata changes
Next, the attributes that describe content and connections.
| Attribute | A change means |
|---|---|
| Title | Regenerated, truncated, emptied, or duplicated |
| Meta description | Changed or emptied |
| H1 and heading outline | Structure changed |
| Word count or content hash | Body content added or removed |
| Inlinks | Pages gaining or losing internal links |
| Outlinks | Links removed from templates |
| Anchor text | Link wording changed |
| Click depth | Pages moved further from the home page |
| Broken internal links | Links now leading to errors |
| Links to redirects | Internal links not updated |
| Images | Missing images, lost alt text |
| Structured data types | Markup added, dropped, or altered |
| Rendered against served | Content now dependent on scripts |
Signs of a template-level regression:
- the same attribute changed on every URL of a pattern;
- titles or H1s that became identical across pages;
- word counts falling by a similar amount everywhere on a template, suggesting a removed block;
- inlink counts collapsing for a group of pages, suggesting a removed navigation element;
- duplicate counts rising.
Thresholds reduce noise. Small differences in word count or response time occur between any two crawls. Flag changes above a set size, and always flag changes in status, canonical, and robots directives whatever their size.
Issue triage
Each difference gets a decision. When the list is longer than the team can fix in one cycle, the order comes from a separate exercise, covered in the guide to prioritizing technical SEO tasks when engineering capacity is limited.
- Compare with the release notes and the change log. Is the difference explained by a planned change?
- If intended, verify it was done correctly, then close.
- If not intended, classify severity.
- Find the cause: which commit, template, or configuration.
- Assign, fix, and re-crawl the affected pattern.
Severity guide:
| Severity | Examples | Response |
|---|---|---|
| Critical | <!–INLINECODE8–> added; canonical pointing to wrong URLs; important pages returning errors; robots.txt blocking | Fix or roll back immediately |
| Major | Titles or H1s lost; internal links removed from templates; structured data dropped; new crawlable duplicate URLs | Fix in the next release, days not weeks |
| Minor | Small content shifts; slight performance change; cosmetic metadata differences | Schedule or accept |
| None | Intended and correct | Record |
A triage record:
| Difference | Intended | Severity | Cause | Owner | Action | Verified |
|---|---|---|---|---|---|---|
| Canonical with parameter on products | No | Critical | Navigation component appends tracking parameter, canonical built from request URL | Developer | Build canonical from clean path | Re-crawl of /products/ |
Reporting rules:
- Report regressions and intended changes in separate sections.
- State the crawl settings and dates.
- Give counts by template, with example URLs.
- Do not report a difference as a failure until it has been checked against the planned changes.
- Note what the crawl could not see: pages behind logins, content varying by location, and anything outside the crawl’s scope.
Make it routine. Keep a saved snapshot before every release. The comparison takes little time when the previous crawl exists, and it cannot be done at all when it does not.
A crawl shows the site as a crawler fetched it on those two occasions. It does not show how search engines responded. Follow critical findings into Search Console’s indexing reports.
Structured data regressions introduced by a plugin update are one class of release defect with its own diagnosis, described in the guide to how a plugin update can introduce structured data errors across an entire site.
The method in order: crawl the same URLs the same way before and after, join on URL, group differences by template, check each against what was planned, and treat as defects only the ones nobody intended.
Add your comment to this answer