How to Run a Title Tag Test and Read CTR Honestly

You cannot randomize organic search traffic, so you cannot run a true split test on a title. What you can run is a quasi-experiment, and it is readable if you build four things into it.

1. Verify the displayed title first. Sample the actual title link in the results before and after the change. If the search engine is rewriting it, the test is void before it starts.
2. Hold position roughly constant. Click-through rate is mostly a function of where you sit and what surrounds you. A page that moved from 8.2 to 6.4 will show a CTR rise that has nothing to do with your sentence.
3. Keep a control group. Pick comparable pages you deliberately do not touch. They absorb seasonality, demand swings and algorithm updates, so a lift that appears in both groups is not your lift.
4. Decide the run length from impressions, not from the calendar. Two weeks on a page with 400 impressions tells you nothing. The table in the duration section gives the floors.

Everything else in this post is the detail behind those four points.

A landing page A/B test works because you own the door. Your server decides which visitor sees which variant, the assignment is random, and both variants run at the same time under the same conditions. Organic search gives you none of that. The search engine decides who sees your listing, when, in what position, and next to what. You get one title per URL, shown to everybody, and the only way to change it is to change it for the whole population at once.

That is the whole problem in one paragraph, and it is why the two disciplines produce evidence of very different strength.

Two workarounds get proposed and both have real limits. URL splitting (publishing two versions at two URLs) breaks because the two URLs will not rank identically, so you have introduced the exact confounder you were trying to remove. Template-level testing across many pages (change the title pattern on half your product pages, leave half alone) is the strongest version available, because assignment to the two halves can be genuinely arbitrary and both halves run in the same weeks. It only works when you have enough near-identical pages, which rules out most sites below a few hundred URLs on a common template.

For everyone else, the honest label is quasi-experiment. That is not a reason to skip it. It is a reason to write the conclusion carefully.

Before you touch anything, find out whether the sentence you publish is the sentence searchers read. This step gets skipped almost universally and it invalidates more title tests than every statistical problem combined.

Independent measurements of rewrite frequency disagree on the headline number because they sample differently. Zyppy, in an analysis by Cyrus Shepard of 80,959 URLs across 2,370 sites using Q1 2022 data, found 61.6% of title links changed. A Search Engine Land analysis by John McAlpin, published May 1, 2025, covering roughly 30,000 keywords across the top 20 results, put it at 76%. Shepard's data also shows written length predicting the rewrite rate, with the low point of the curve falling around 51 to 60 characters and titles over 70 characters rewritten almost always. On March 20, 2026, Danny Goodwin reported at Search Engine Land that Google confirmed a test of AI-generated headline rewrites in Search, described by the company as small and narrow.

Search Console will not answer this for you. The Performance report stores queries, clicks, impressions, CTR and position. It does not store the title link that was displayed, and URL Inspection reports indexing status rather than the headline shown beside your result. So sample it by hand, before the change and again after it.

If the after sample shows a rewrite that the before sample did not, stop and report that instead of a CTR number. It is a more useful finding than the one you were chasing. Our post on how to match a title tag to a landing page headline has the full sampling method and what to do about each classification.

Write the design down before you make the change. A test designed after the data arrives is not a test.

Hypothesis. One sentence, with a direction and a mechanism. "Adding the year to the title of our nine comparison pages will raise CTR because the queries carry recency intent." Not "we think the new titles are better."

Unit. Pages, pooled. A single page is almost never big enough to read on its own, so a title tag test is usually a test of a title pattern applied to a group.

Baseline window. 28 days, ending three days before the change to allow for the Search Console data delay. Export clicks, impressions, CTR and average position per page, and per query for the top queries, and save the file. Do this before you edit anything.

Change. One class of change only. Titles, on the test group, all on the same day. If you also update the meta description, the H1 and the publish date, you have run one test and learned nothing about any of the four.

Start of the change window. Not the day you deployed. The day the new title is actually being shown, which you confirm by sampling the results. Reindexing lag varies by site and by page, and counting from the deploy date quietly contaminates the first days of your change window with old-title data.

Change window. The same length as the baseline, 28 days, so day-of-week effects cancel. Do not compare a 28 day window against a 14 day window and scale it up.

Control group. Named in advance. See the control group section.

Decision rule. Written before you look: the minimum control-adjusted CTR difference you will treat as signal, and what you will do at each outcome. Deciding this afterwards is how a null result becomes a case study.

Ranked roughly by how often each one turns out to be the real explanation for a reported lift.

CTR without position is not a metric, it is a coincidence. A listing at position 3 and the same listing at position 8 have different achievable click rates no matter what the title says, which is why a before and after CTR comparison taken across a position change tells you nothing about the change you made.

Search Console gives you average position, and it needs handling with care. It is an average across every impression in the window, and it moves when the query mix moves even if your rank for every individual query is identical. There is also no position filter in the Search Console interface. Filters cover search type, date, query, page, country, device and search appearance, so position handling happens in your spreadsheet after export, not in the report.

A workable rule set:

The stronger version of this check works at query level. Export the Queries report for the test pages in both windows, keep only the queries present in both, and compare CTR query by query inside the same position bucket. It is more work and it removes the single biggest source of fake lifts. If a title change is real, it should show up on queries whose position did not move.

For the mechanics of pulling these exports cleanly, including the privacy filter that makes the export disagree with the chart, see how to measure organic CTR in Google Search Console.

The control group is the cheapest upgrade available to a title test and the step almost everyone skips. Its job is simple: anything that happens to your site for reasons unrelated to your change also happens to the control pages, so you can subtract it.

Choosing one well matters more than making it large.

• Comparable position band. Control pages should sit in roughly the same position range as the test pages. A control set ranking 1 to 3 cannot tell you anything about drift affecting pages ranking 6 to 10.
• Comparable intent and template. Blog posts control for blog posts. Product pages control for product pages. Mixing them imports a difference you did not intend to measure.
• Comparable impression volume. Ideally the control group's total impressions are at least as large as the test group's, so the adjustment you subtract is measured more precisely than the effect you are measuring.
• Genuinely untouched. No title edits, no content refresh, no internal link changes, no new schema, for the whole test.
• Enough of them. Eight to fifteen pages is a practical range. Fewer than five and one odd page dominates the control average.

Then there is the assignment problem. You will be tempted to put your worst-performing pages in the test group, because those are the ones you want to fix. Do that and regression to the mean will hand you a lift for free: pages selected for having an unusually bad window tend to look better in the next window regardless of what you do. If you have enough comparable pages, assign them to test and control arbitrarily, for example by sorting on page ID and alternating. If you must test only the underperformers, choose control pages that underperformed by a similar margin in the same baseline window.

"Run it for a month" is a calendar rule pretending to be a statistical one. What actually decides whether you can read the result is how many impressions accumulate in each window, and how big an effect you are trying to detect.

CTR is a proportion, and the uncertainty on a proportion shrinks with the square root of the sample. A standard two-proportion power calculation gives a planning floor. The table below assumes a 3.0% baseline CTR, 80% power and a 5% two-sided significance level, and shows impressions needed per window, so the baseline window and the change window each need that many.

Baseline CTR changes the answer too, because the variance of a proportion depends on the proportion itself. Holding the target at a 20% relative change:

Read the top row of the first table and let it change your plans. Detecting a 5% relative improvement needs around 208,000 impressions in each window, which most sites will never have on a single page set. That is not a reason to give up on testing. It is a reason to stop reporting 5% movements as results.

Three caveats, stated rather than buried. Impressions are not independent draws: the same person searches twice, a query trends for a week, and the results page changes mid-window. Every figure above therefore understates the real uncertainty. Second, this is a planning tool for deciding whether a test is worth running, not a significance test to quote in a client report. Third, if you cannot reach the floor, pool more pages rather than extending the window. Extending buys impressions at the price of more seasonality and more ranking drift, and past about eight weeks that trade goes bad. Click signal measurement limits covers what else the two reports quietly cannot see.

Every figure below is an illustrative example, not SparkCliks data and not a study result. The arithmetic is real and you can run it on your own exports.

Setup. Five comparison pages under /compare/ get new titles that lead with the object noun and drop a brand suffix. Ten pages under /guides/, in a similar position range, are the control and stay untouched. The baseline window is 28 days ending three days before the change. The change window is the 28 days beginning the day the new titles were confirmed live in the results, which was four days after deploy.

The naive read. Test CTR went from 2.88% to 3.27%. That is a 13.5% relative rise, and it is the number that ends up in the case study. Run the two-proportion check on it:

SE = sqrt( p1(1-p1)/n1 + p2(1-p2)/n2 )
   = sqrt( 0.0288*0.9712/41300 + 0.0327*0.9673/43100 )
   = 0.00119

z  = (0.0327 - 0.0288) / 0.00119 = 3.28

A z of 3.28 looks decisive. It is also wrong, because it credits the title with everything that happened to the site during those 28 days.

The control-adjusted read. Subtract what the untouched pages did over the same period. This is a difference in differences:

Attributable change
  = (test after - test before) - (control after - control before)
  = (3.27% - 2.88%) - (3.08% - 3.00%)
  = 0.39 points - 0.08 points
  = 0.31 points

That is about a 10.8% relative improvement on the test group's baseline, not 13.5%. Now the uncertainty, which has to combine both groups:

SE(difference in differences) = sqrt( SE_test^2 + SE_control^2 )
                              = sqrt( 0.00119^2 + 0.00104^2 )
                              = 0.00158

z = 0.0031 / 0.00158 = 1.96
95% interval on the attributable change: 0.00 to 0.62 points

The verdict. The interval touches zero. Position was stable in both groups, at 0.1 of movement, and the displayed titles were confirmed unchanged, so the test is at least valid. The honest reading is "a small improvement, most likely real, possibly as small as nothing, plausibly as large as 0.6 points." That is a keep-the-change result, not a case study. Notice how far it sits from the headline the naive calculation offered. The gap between those two numbers is the entire subject of this article.

Had the test group been twelve pages instead of five, that same effect would have cleared the floor comfortably. The fix for this test was more pages, decided in advance.

Match your outcome to a row before you write anything down.

Three claims to keep straight, because they get collapsed into one constantly.

You may claim: the pages whose titles you changed showed a CTR difference of X points relative to a set of comparable pages you did not change, over a defined window, at roughly stable position.

You may not claim: that the title change caused a ranking improvement. Nothing in a CTR test measures ranking, and the relationship between click behavior and ranking is contested rather than settled. Search engines have publicly denied using raw click-through rate as a direct ranking factor, and our post on click data as a ranking signal walks through what has actually been said and shown.

You may not claim: a percentage lift with no interval attached. "CTR rose 13.5%" and "CTR rose somewhere between 0% and 22%" describe the same data, and only the second one is honest. If you report a single number, put the impression counts beside it so a reader can check you.

One more discipline that costs nothing: write up your null results internally. A team that only records its winners builds a library of noise, and after a year nobody can tell which title conventions actually work on their site. Knowing which patterns you tested and could not distinguish from zero is a real asset, and it is what stops the next person rerunning the same test. To decide which pages deserve a test at all, grade them against your own organic CTR benchmark rather than a published click curve.

Run this before the change, not after.

1. Hypothesis written down with a direction and a mechanism.
2. Test pages named. Control pages named, comparable in position band, template and volume, and at least as large in total impressions.
3. Assignment to test and control is arbitrary, or the two groups underperformed by a similar margin in the baseline window.
4. Displayed titles sampled by hand for the top query of every test page, and recorded.
5. Baseline exported: pages and queries, with clicks, impressions, CTR and position, 28 days ending three days back.
6. Brand queries split out with a regex query filter if brand traffic is a meaningful share.
7. Impression volume checked against the duration table. If the test group cannot reach the floor, add pages before starting.
8. Minimum readable difference written down, along with the action you will take at each outcome.
9. Any active click campaign on the test pages either paused, or extended to the control pages at the same rate.
10. Only the title changes. Nothing else ships to these pages during the test.
11. Change window starts on the confirmed live date, not the deploy date.
12. After the window closes, sample the displayed titles again before you calculate anything.

Steps 4 and 12 are the two people cut for time, and they are the two that decide whether the other ten meant anything.