Incrementality Testing: How Much of Your Paid Search Is Buying Clicks You Already Own?

Incrementality testing answers the one question about paid search that a channel report never will: how much of that spend bought customers, and how much bought clicks the business was already going to get for free. The distinction sounds academic until you see it priced. In a case study published by Search Engine Land on August 17, 2026, SEO director Anna Crowe matched a company’s Google Ads search terms against its Search Console queries and found that of $36,129 spent in one branded campaign, only $3,945, or 10.9%, appeared to buy clicks organic could not have captured. The other 89.1% was brand defense. This article is the repeatable version of that analysis: how to build the match, what counts as dependent rather than cannibalized, how long a pause test needs to run, and the conditions under which the right answer is to leave brand search on.

What is incrementality testing in paid search?

Incrementality testing measures the conversions or clicks that would not have happened without the ad, by removing the ad for a controlled group and comparing the result against a group that kept it. It is the difference between what a platform attributes to a campaign and what the campaign actually caused. A last-click report gives paid search credit for every conversion it touched. An incrementality test asks whether the touch was necessary.

Three designs get used in practice. A geo holdout switches ads off in a random half of markets and leaves them on in the rest, which is the design Google itself recommends for search. A time-based pause turns a campaign off everywhere and reads the before-and-after, which is simpler and cheaper but exposed to seasonality. A query-level match pairs each paid search term with the same query in Search Console to see whether organic was already winning the click. Crowe’s case study used the third to size the problem and the second to test it, which is the practical combination for most businesses because it needs no experiment tooling, only two exports and patience.

The reason this matters more in 2026 than it did in 2019 is that the assumption underneath most paid budgets, that paid and organic hold a stable ratio, no longer holds. A Similarweb analysis by Aleyda Solis, reported by Search Engine Land on February 18, 2026, compared the top U.S. queries in four verticals between January 2025 and January 2026. Classic organic lost 11 to 23 percentage points of click share in every vertical, text ads gained 7 to 13 points in every case, and AI Overview presence rose by roughly 10 to 30 points depending on the category. When the organic side of the ledger is moving that fast, a paid budget set on last year’s overlap is measuring against a page that no longer exists.

Does paid search cannibalize organic?

Sometimes, and the honest answer depends almost entirely on where your organic result already ranks. That is not a hedge; it is what the two most-cited studies on the subject found when read together rather than against each other.

Google’s own research is the case for leaving ads on. A 2011 study by David Chan, Yuan Yuan, Jim Koehler and Deepak Kumar, published in the Journal of Advertising Research, analyzed several hundred Search Ads Pause studies and concluded that on average 89% of ad clicks were incremental: organic did not pick them up when the ads stopped. A follow-up published on Google’s research blog on March 27, 2012 explained why the number is so high. Across 390 pause studies, 81% of ad impressions and 66% of ad clicks occurred with no associated organic result for that advertiser on the first page at all. There was nothing for organic to substitute.

The same follow-up is also the best evidence for the cannibalization argument, because it broke incrementality out by organic rank. When the advertiser already held the top organic position, only 50% of ad clicks were incremental. Between positions 2 and 4, 82% were. At position 5 or lower, 96% were. Read that as a rule: the better you rank organically for a query, the more of the paid click you were going to get anyway.

Branded queries are the extreme case of ranking first, which is why the eBay experiment landed the way it did. Tom Blake, Chris Nosko and Steven Tadelis ran a series of large-scale field experiments at eBay and published the results in Econometrica in 2015. For brand keywords, they concluded that natural search is close to a perfect substitute for paid search, making brand-keyword search marketing ineffective for short-term sales. Nearly all of the forgone paid clicks and attributed sales showed up in organic instead. A 2018 VoxEU column summarizing the work put the substitution rate at 99.5%.

Both findings are true. Google measured the average advertiser, most of whom lack a page-one organic result for the queries they buy. eBay measured a household name on its own name. Your account sits somewhere on that spectrum, query by query, and no benchmark will tell you where. The match will.

Should you bid on your own brand name?

Bid on your brand when a competitor is bidding on it, when your organic result for the brand query is anything other than a clean first position with sitelinks, or when a test shows the clicks do not come back through organic. Skip it when none of those hold. The eBay result gets quoted as if it settled the question, and a second experiment shows why it did not.

Lorenzo Coviello, Uri Gneezy and Lorenz Goette repeated the eBay design at Edmunds.com, an automotive information site that ranked in the top 1% of U.S. websites at the time but competed against rivals of similar size. They randomized 105 of the 210 U.S. designated market areas into a treatment group, shut off branded search ads there from August to November 2015, and compared traffic against the 105 control markets. Organic traffic rose in the treatment markets, but nowhere near enough to cover the loss. Less than half of the traffic that had been arriving through branded ads still reached the site through organic links. In markets where branded ads had been a large share of baseline traffic, Edmunds lost about 72% of it. The authors’ explanation for the missing half: competitors who were bidding on the keyword “Edmunds” and now had the top of the page to themselves.

That is the whole brand-bidding decision in one contrast. eBay had no meaningful competitor on its name and substitution was 99.5%. Edmunds had several and substitution was under 50%. Before you touch the campaign, search your own brand name in an incognito window and count the competitor ads above your organic result. If the answer is zero and you own the top organic slot, you are closer to eBay. If there are two or three, you are closer to Edmunds, and the ad is doing a job organic cannot.

One more implied question: does the ad make people trust the brand more, or give you more of the page? Possibly, and neither effect shows up in a click count. The 2012 Google study says as much, noting that it measured clicks rather than conversions and recommending randomized experiments for anyone who wants to know the conversion lift. Treat the halo as a hypothesis the test can confirm or reject, not as a reason to skip the test.

How do you match Google Ads search terms to Search Console queries?

Export both reports for the same date range, join them on the exact query string, and sort every overlapping query into one of three buckets: cannibalized, dependent, or incremental. This is the analysis that turned “we’ve always run brand” into an 89.1% number in Crowe’s case study, and it takes an afternoon rather than a quarter.

1. Pull the paid side. In Google Ads, export the search terms report (not the keyword report) for the branded campaign over a clean 90-day window, with impressions, clicks, cost and conversions per term. Search terms are what people typed; keywords are what you bid on. The match only works on the former.
2. Pull the organic side. In Search Console, export the Performance report filtered to the same 90 days with the query dimension, plus clicks, impressions, average position, and the landing page. Search Console caps exports, so for a large account use the API or a connector to get past the 1,000-row limit.
3. Normalize and join. Lowercase both, strip punctuation, and join on the query. Expect a partial match: Search Console anonymizes low-volume queries, and Google Ads groups some close variants. Anything on the paid side with no organic twin gets a fourth label, no organic coverage, and is presumptively incremental.
4. Score each overlapping query. Record organic average position, organic click share of the combined paid-plus-organic clicks, and paid cost. Crowe’s account was already winning 71% of clicks on overlapping queries through organic even while the ads ran.
5. Assign the bucket. Organic position 1 to 2 and organic click share above roughly two-thirds: cannibalized. Organic present but position 3 or worse, or a query where organic clicks fell in past periods when paid was reduced: dependent. Organic absent from page one: incremental. The thresholds are yours to set; what matters is that you set them before you look at the totals, so the answer cannot drift toward the one you wanted.
6. Price the defense. Sum paid cost by bucket. The cannibalized total, annualized, is the number to bring to the CFO. In the case study, the annualized cannibalized spend was roughly $128,000 against roughly $15,000 of incremental spend in one campaign.

The dependent bucket is where most teams get it wrong, because it is the one that requires history. A query is dependent if organic performance has previously moved with paid spend, not simply because both channels show up for it. If you have no period of reduced spend to compare against, label the query cannibalized-pending and let the pause test decide. Guessing “dependent” for every overlapping query is how brand defense budgets survive audits.

How long should a paid search holdout test run?

Long enough for revenue to settle, and revenue settles later than clicks, which in the one recent public test with full data meant 13 weeks before the recapture figure stopped moving. Clicks come back in weeks. Revenue lags them, because paid and organic visitors do not buy at the same rate, and because direct traffic takes time to absorb people who used to arrive through an ad.

The case study’s timeline is the clearest published example of that lag. The company paused branded search across the U.S., U.K., Australia and Canada, then most non-brand search by the end of the month, coming off an average of $113,000 a month across branded, non-brand, Shopping and Performance Max. Six weeks in, organic and direct had recaptured 30% of the paid-attributed revenue. By weeks 7 through 12 it was 39%. From week 13 onward it reached 65%, driven by an organic lift of $15,800 to $28,100 a month across the phases and a direct-revenue lift that climbed from about $14,000 a month to roughly $54,000. A team reading the result at week 4 would have concluded the pause failed. A team reading it at week 13 saw a different business.

Two design rules follow from that curve. First, set the read date before you start and commit to it in writing, because every week of a pause produces a number that someone will want to act on. Twelve weeks is a defensible minimum for a business with a purchase cycle under a month; longer cycles need longer reads. Second, hold something back. If you can split by geography, run the eBay and Edmunds design and keep half your markets live, which lets you read the difference between groups instead of the difference between periods and removes seasonality from the argument. A full pause is acceptable when you cannot split, but then compare against the same weeks last year, not the weeks immediately before.

Watch conversion rate separately from revenue. In the case study, paid traffic converted at about 2.9% and organic at about 2.3%, so clicks recovered faster than money did. If your gap is wider than that, the recapture percentage will flatter the test; if it is narrower, the test will look worse than it is at the halfway mark. Either way, the number that decides the question is net contribution after media cost. Top-line revenue in the case study fell roughly $30,000 to $40,000 a month from the pre-pause baseline, and net P&L was estimated $5,000 to $20,000 a month higher once the spend that was no longer going to Google was counted. Selling slightly less and making more money is a result a channel dashboard cannot display, and it is the one that matters.

When should you leave brand search on?

Leave it on when the query match shows real incremental spend, when competitors hold the top of your brand SERP, when your organic brand result is weak or shared, or when the holdout shows the clicks leaving rather than moving. Agencies that sell this analysis rarely publish the conditions under which the answer is no, so here they are.

• Competitors are bidding on your name. This is the Edmunds condition, and it is the most common reason a brand test fails to replicate eBay. If a rival’s ad sits above your organic listing, the paid click is not cannibalized; it is contested.
• Your organic brand result is not clean. A retailer listing, a review site, or an old subdomain outranking your homepage for a brand-plus-product query means organic is not positioned to absorb the click. Fix the organic problem first, then retest.
• The match found meaningful incremental spend. In the case study it was 10.9% of one campaign. If your incremental bucket is 40%, the pause is a much bigger bet, and the more useful move is to pause only the cannibalized terms rather than the campaign.
• Non-brand and Shopping are in the same test. The published pause eventually included most non-brand search too. Non-brand queries sit lower on the incrementality chart above, closer to the 82% and 96% rows, and should be tested on their own terms rather than swept into a brand decision.
• You need the landing page control. Brand ads let you route a query to a promotion or a specific page; organic sends it wherever Google decides. If that routing is worth money to you, price it and keep it.

The pattern across every one of those conditions is the same: brand search earns its place when organic cannot do the job alone, and the query match is how you find out whether it can. This sits inside the broader paid search question of what a budget should be buying now that Google is expanding paid real estate at organic’s expense, a shift we covered when AI Max replaced Dynamic Search Ads and again in what agentic commerce changes about a paid budget. The incrementality test is the piece those discussions assume you have already done.