usdigitalmarketingindex.com SEO
How can untreated control pages be used to evaluate an SEO change?
Updated November 15, 2026
By applying the change to one group of pages, leaving a comparable group alone, and reading the result as the difference between how the two groups moved over the same period. Organic traffic changes for reasons unrelated to any edit: seasons, search updates, competitors, news. A before-and-after comparison of the changed pages alone credits all of that to the change. A control group experiences the same outside influences without the change, so its movement is an estimate of what the changed pages would have done anyway. The difference between the two is the estimate of the change’s effect. It is an estimate with real limits. Pages are not interchangeable, groups are small, and anything that touches only one group, other than the change, contaminates the result.
Selection of controls
Controls should resemble the test pages in everything that affects search performance.
| Attribute | Why it must match |
|---|---|
| Template or page type | Different types respond differently to the same events |
| Topic and intent | Seasonal and market effects differ by topic |
| Traffic level | Small pages are noisy; large ones dominate totals |
| Age | New pages are still settling |
| Position range | Pages near the top have less room to move |
| Device and country mix | External effects vary by both |
| Internal linking | Structural importance |
Ways to form the groups:
| Method | Description | Strength |
|---|---|---|
| Random assignment | From a pool of similar pages, assign each to test or control at random | Best protection against choosing, even unconsciously, pages that were about to improve |
| Matched pairs | Pair pages by traffic and topic; assign one of each pair to test | Works with small numbers |
| Convenience | Change the pages someone wanted to change; use others as control | Weak; the choice itself may explain the result |
Random assignment within a set of similar pages is the method to prefer. Record the full list of both groups before the change is made.
Group size matters. With a handful of pages per group, one page’s unrelated jump or fall decides the outcome. More pages, each with meaningful traffic, give a steadier comparison. Where the site has only a few comparable pages, say that the test is indicative at best.
Title tags are one change this design is used for, with details of their own, such as how the two groups are formed and when to stop. They are covered in the guide to testing title-tag variations on matched sets of comparable pages.
Matching traffic patterns
Before the change, the two groups should move together. Check it.
- Take a pre-period of several weeks, long enough to include ordinary variation.
- Plot weekly clicks for the test group and the control group.
- Compare their week-to-week movement.
| Pre-period finding | Meaning |
|---|---|
| The lines rise and fall together | The control is a reasonable stand-in |
| The lines diverge before anything was changed | The groups differ; re-select |
| One group has a spike from a single page | Remove or replace that page, in both groups' terms, before starting |
| The ratio between the groups is stable | Good; the ratio is the baseline |
If the groups did not track each other before the change, a difference afterward proves nothing.
Use the same metric throughout, for instance clicks from Search Console by page, with the same country, device, and search type filters. Sum old and new URLs if any URL changed.
Avoiding contamination
Contamination is anything, other than the change under test, that affects one group and not the other, or that carries the change into the control.
| Source | Example | Safeguard |
|---|---|---|
| Sitewide changes during the test | A template release, navigation change, or speed fix that alters every page | Freeze other changes, or note them; they affect both groups, though not always equally |
| Changes to control pages | Someone edits a control page | Mark control pages as frozen |
| Internal links | Test pages gain links that take prominence from control pages | Keep link changes out of the test unless they are the test |
| Shared queries | Test and control pages compete for the same queries, so a gain for one is a loss for the other | Choose pages that do not rank for the same queries |
| Spillover | The change on test pages affects the site as a whole, including controls | Recognize that the difference then understates the effect |
| External events | News or a seasonal peak affecting one topic | Match topics across groups |
| Search updates | A broad update during the test | Affects both groups, but possibly unequally; extend or repeat |
| Tracking changes | Tags or consent altered on some pages | Use Search Console clicks, which do not depend on site tags |
| Different timing | Test pages changed over several weeks | Change all test pages on one date |
Keep a log of everything that happened on the site during the test. It is the first thing to consult when a result looks odd.
Calculating directional impact
A test-versus-control example, with invented numbers.
| Group | Pages | Clicks, 4 weeks before | Clicks, 4 weeks after | Change |
|---|---|---|---|---|
| Test: titles rewritten | 30 | 10,000 | 11,500 | +15% |
| Control: untouched | 30 | 8,000 | 8,400 | +5% |
Calculation:
Control change: 8,400 / 8,000 = 1.05 (+5%)
Expected test clicks without the change:
10,000 x 1.05 = 10,500
Observed test clicks: 11,500
Difference: 11,500 - 10,500 = 1,000 clicks
Relative to expected: 1,000 / 10,500 = 9.5%
Reading: the test pages rose 15%, of which about 5 points is what comparable pages did with no change. The estimate of the change’s effect is about 1,000 clicks over four weeks, or roughly 9.5% above what was expected.
A naive before-and-after reading would have claimed the full 15%. If the control had fallen 10% and the test had stayed flat, the same method would show a benefit where the naive reading shows nothing.
Assumptions behind the causal reading:
- Without the change, the test group would have moved as the control group did. This is assumed, supported by the pre-period check, and never proven.
- Nothing else affected one group and not the other.
- The control group was not affected by the change.
- The groups are large enough that the difference is not one page’s accident.
- The periods are comparable in length and in weekdays.
Checks on the result:
| Check | Purpose |
|---|---|
| Per-page view | Is the gain spread across test pages, or due to one or two? |
| Remove the largest page from each group and recalculate | Does the result survive? |
| Pre-period difference | How much did the two groups differ by chance before the change? A post-change difference no larger than that is not evidence |
| Other metrics | Do impressions, position, and CTR move in a way consistent with the change? A title change should show in CTR at similar positions |
| Longer window | Does the difference persist? |
| Repeat | Apply the change to the control group later; does it respond similarly? |
State the conclusion as directional. “Test pages outperformed controls by about 9.5% over four weeks; the pre-period difference between the groups was within 2%; the gain was spread across most test pages” is a supported statement. “The change increased traffic by 9.5%” claims more precision and certainty than a page-level comparison on one site provides.
Limits to acknowledge every time:
- This is not a randomized experiment on searchers. Pages, not people, are assigned, and pages differ.
- Search results are a competitive system. Other sites changed during the test too.
- A result on one template may not transfer to another.
- No effect detected is not proof of no effect, especially with small groups.
This guide covers the design and the calculation. Deciding what to conclude from a test that showed no gain, or a loss, and when to stop one, is taken up in the guide to what criteria should determine whether a completed SEO experiment has failed.
The method in order: choose comparable pages, assign at random, confirm they moved together beforehand, change one group on one date, keep everything else equal, and report the difference in movement with its assumptions.
Add your comment to this answer