Skip to the indexed answers
Digital Marketing Index

usdigitalmarketingindex.com SEO

How can title-tag variations be tested on matched sets of comparable pages?

By dividing a set of similar pages into two groups, applying the new title pattern to one group and leaving the other as it is, and comparing how the two groups move over the same weeks. A page has one title at a time, so two titles cannot be shown to searchers of the same page side by side. The comparison is between groups of pages instead. That makes it a quasi-experiment: pages are not identical, search results shift for reasons outside the test, and Google may display a title link different from the title element. Run with care, the test indicates whether the new pattern helps. It does not produce a precise uplift figure, and the plan should say so before it starts.

Selecting comparable pages

The test needs many pages of one kind.

Requirement Reason
Same template Titles follow one pattern, so a pattern can be tested
Same intent Product pages with product pages; guides with guides
Enough pages With few pages, one page's unrelated movement decides the result
Enough clicks per page Pages with a handful of clicks add noise
Stable history Exclude pages published or substantially edited in recent weeks
No pending changes Exclude pages due for other work

Assignment:

  1. List the eligible pages with their clicks and impressions for the previous weeks.
  2. Sort by clicks, and pair adjacent pages.
  3. Within each pair, assign one page to the variant group and one to the control group at random.
  4. Confirm that the two groups have similar totals for clicks, impressions, and average position before the test.
  5. Record both lists.

Random assignment within matched pairs is what makes the groups comparable. Choosing the variant pages by hand, for instance the ones that “need it most”, builds a difference into the groups before the test begins.

Define the variant as a pattern, not as individually crafted titles. “Add the key specification after the product name” can be applied to every variant page and later to the whole template. Titles hand-written per page test the writer, not the pattern.

Controlling concurrent changes

During the test, the only intended difference between the groups is the title.

Control Detail
Freeze other changes on both groups No content edits, no link changes, no template work on these pages
Change all variant titles at the same time One start date
Leave H1s and content unchanged Otherwise the test is of several things
Hold the control titles fixed No edits
Log sitewide events Releases, outages, search updates, campaigns, with dates
Check for shared queries Variant and control pages should not compete for the same queries
Keep internal links constant No new links to either group

Verify the treatment. Changing the title element does not guarantee what searchers see. Google’s documentation explains that title links are generated automatically and can draw on several sources on the page (Google’s documentation on title links). After the change is recrawled, search for a sample of variant pages and record the title link displayed. If Google is showing something other than the new title on a large share of them, the test is not measuring what was intended.

Allow time for recrawling before the measurement period starts. Pages changed on day one are not all reprocessed on day one. Confirm with URL Inspection on a sample that the new title has been seen.

Measuring CTR and position

A title’s direct effect should appear in click-through rate at a given position. Measure position alongside it, since a change in position changes CTR for reasons unrelated to the wording.

For each group, for the pre-period and the test period, from Search Console’s performance report filtered to the group’s pages:

Metric Use
Clicks Outcome
Impressions Exposure
CTR Primary measure
Average position To check that the groups did not diverge in ranking

Compare the groups’ changes, not the variant’s change alone.

A results layout, with invented figures to show the form:

Group Pre-period impressions Pre-period clicks Pre-period CTR Test impressions Test clicks Test CTR Pre position Test position
Variant, 60 pages 200,000 6,000 3.0% 204,000 6,936 3.4% 6.1 6.0
Control, 60 pages 198,000 5,940 3.0% 202,000 6,060 3.0% 6.2 6.2

Reading: both groups began at 3.0%. The control stayed at 3.0%. The variant rose to 3.4%, with position nearly unchanged in both. The difference between the groups is 0.4 percentage points of CTR.

Before treating that as an effect, check:

Check Purpose
Pre-period difference How far apart were the groups in earlier periods of the same length, with no treatment? A test difference no larger than that is noise
Per-page distribution Is the gain spread across variant pages, or carried by two or three?
Remove the largest pages and recompute Does the result hold?
Matched queries For queries present in both periods, did CTR rise at similar positions?
Position Did the variant group's position change? If it did, CTR moved for more than one reason
Device split Is the effect present on each device?
Title link displayed Was the new title shown?

Query-level data in Search Console omits anonymized queries, so query-level checks cover part of the clicks.

Stopping and interpreting results

Fix the rules before the test starts.

A test plan:

Item Content
Hypothesis The variant pattern raises CTR on product pages at unchanged positions
Pages Eligible list; variant and control lists attached
Variant The exact pattern
Start Date all variant titles change
Wait Period for recrawling, confirmed by inspection
Measurement period Fixed length in whole weeks, set in advance
Primary metric Difference between groups in CTR change
Secondary Clicks; position
Minimum data Clicks per group below which the test is declared inconclusive
Noise benchmark The between-group difference seen in pre-periods
Stop early if The variant group's clicks fall well beyond the control's, checked weekly
Decision rule Roll out if the difference exceeds the noise benchmark, is spread across pages, and position is stable; otherwise do not
Frozen changes List
Owner Name

Do not stop the test the first time the variant is ahead. Looking daily and stopping on a good day finds effects that are not there. Run the full planned period.

Limits of the quasi-experiment, to print with the result:

  • Pages were assigned, not searchers. The two groups of pages differ in ways matching cannot remove.
  • Searches for the two groups are different searches, on different results pages, against different competitors.
  • Google may rewrite title links, differently for different queries.
  • Anything that affected one group only during the period is mixed into the result.
  • A result on one template and one period may not hold on another.
  • With modest numbers of pages, real effects can be missed and chance differences can look real.

State the conclusion in those terms. “Over four weeks the variant group’s CTR rose 0.4 points more than the control’s, a difference larger than the 0.1 points seen between the groups in the three preceding periods, with position stable and the gain spread across most pages” describes what was observed. A claim that the new titles increase CTR by a fixed percentage goes beyond it.

Possible outcomes:

Outcome Action
Variant clearly ahead on the agreed rule Apply the pattern to the control group; watch whether it responds similarly, which serves as a replication
No difference beyond noise Keep the original; record the test
Variant behind Revert; record
Mixed by device or page type Consider a narrower pattern, and a new test
Titles not displayed as written Investigate why before testing wording

Record the test and its result in the change log either way. A pattern that made no difference is worth knowing about.

This method suits a decision about a pattern across many pages. The prior question, whether a single page that already performs well should have its title touched at all, is weighed in the guide to when it is worth changing the title tag of a page that already performs well.

The method in order: pair similar pages, assign at random, change only the title on one group, confirm what Google displays, measure the difference in CTR change over a fixed period, and report it with the limits of the design.

Link trail

Reader comments on this answer

Add your comment to this answer

Your email address will not be published. Required fields are marked *