usdigitalmarketingindex.com SEO
What criteria should determine whether a completed SEO experiment has failed?
Updated November 15, 2026
The criteria are the ones written down before the experiment started. A test has failed when its primary metric, measured over the planned period against the planned comparison, did not reach the threshold that was set in advance, and nothing happened during the test that invalidates the comparison. It is inconclusive when the comparison was disturbed, the data were too thin to separate an effect from ordinary variation, or the test did not run as designed. Without criteria fixed beforehand, a completed experiment cannot fail or succeed. It can only be argued about.
Hypothesis and primary metric
Three things must be recorded before any change is made.
- The hypothesis, as a statement that could turn out false: changing X on these pages will raise Y.
- One primary metric. Clicks, impressions, or click-through rate for a defined set of pages and queries. Secondary metrics can be watched, and they do not decide the outcome.
- The threshold. The smallest change in the primary metric that would justify rolling the change out. This comes from the cost of the rollout, not from statistics.
Record also which pages change and which comparable pages stay as they are. A before-and-after reading on the changed pages alone cannot separate the change from everything else that happened in those weeks.
Test duration and confounders
Set the duration in advance and do not stop early because the numbers look good or bad.
The period has to be long enough for search engines to process the change. Google’s starter guide advises that “you likely want to wait a few weeks to assess whether your work had beneficial effects” (Google’s SEO starter guide). It also has to cover whole weeks, so that weekday patterns do not distort the comparison.
List in advance the events that would disturb the test, and check for them at the end.
- An announced Google update during the period
- A seasonal shift in demand for the topic
- Other changes to the same pages or templates
- A site outage, a tracking change, or a marketing campaign
- A change in what appears on the results page for the queries involved
Criteria for failure versus noise
Apply the pre-set rules in order.
| Finding at the end of the test | Verdict |
|---|---|
| Test pages beat comparison pages by at least the threshold, with no confounder | Successful |
| Test pages did not reach the threshold, comparison pages were stable, no confounder | Failed |
| Test pages performed worse than comparison pages by a clear margin, no confounder | Failed, and the change is reverted |
| Test and comparison pages moved together | Inconclusive: an outside factor moved both |
| A listed confounder occurred | Inconclusive |
| The number of clicks or impressions was too small for the difference to exceed normal week-to-week variation | Inconclusive |
| The change was not deployed as specified | Invalid: rerun |
To judge normal variation, look at how much the primary metric for the comparison pages moved from week to week before the test. A difference smaller than that movement cannot be distinguished from it. This is a practical screen. It does not replace a statistical test, and no claim of statistical significance should be made unless a proper method was chosen in advance and its conditions were met.
A failed test is a valid result. It means the hypothesis was tested fairly and was not supported.
What to do with ambiguous results
A sample decision log shows how the record reads. The entries are fabricated for illustration.
| Date | Entry |
|---|---|
| Before launch | Hypothesis: adding a comparison table to 20 category pages will raise clicks. Primary metric: clicks per page per week. Threshold: 10 percent above comparison pages. Duration: six full weeks. Comparison: 20 similar category pages, unchanged. |
| Before launch | Confounders to check: Google updates, seasonal demand, other template changes. |
| Launch | Change deployed to the 20 test pages. Verified on all 20. |
| End of week 6 | Test pages up 4 percent. Comparison pages up 3 percent. Week-to-week variation before the test was about 6 percent. No listed confounder occurred. |
| Decision | Failed by the pre-set rule: the test pages did not reach the 10 percent threshold, the comparison pages were stable, and no listed confounder occurred. The 1-point difference is also inside normal variation. Not rolled out. |
| Limitations | Twenty pages per group. Low click counts on eight of the test pages. |
For an inconclusive result there are three legitimate next steps, and they should be chosen by rule.
- Extend, if the plan allowed for one extension and no confounder occurred. Decide the extension length before looking again.
- Rerun with more pages or on pages with more traffic, if the data were too thin.
- Close and record, if the change is cheap to keep and harmless, or expensive and unproven.
What is not legitimate is to change the primary metric after the fact, to select the subset of pages that improved, or to keep extending until the number crosses the line.
Choosing the unchanged comparison pages well is what makes the verdict possible. That method is covered in the guide to using untreated control pages to evaluate an SEO change.
The criterion in one sentence: a test fails by the rule you set before it began, or it does not have a verdict.
Add your comment to this answer