Running a CTV holdout test: design, sizing and reading the result

Attribution shows what exposed households did. A holdout test asks what they would have done anyway. By keeping a random or matched share of the eligible audience from seeing the ads, you create a comparison group. The difference in outcomes between the exposed group and the holdout group is the lift the ads produced.

• Pick one primary metric, such as new customers or revenue per household, before launch.
• Decide the holdout share. A larger holdout gives a clearer read but costs reach.
• Randomise at the household level, or use matched regions for a geo test.
• Fix the test length to cover at least one purchase cycle.
• Agree in writing how the platform will report exposed and holdout results.

A household holdout randomises individual households and needs the platform to keep the holdout clean across all its inventory. A geo test switches ads on in some regions and off in comparable others; it needs no household tracking and suits TV well, but needs enough regions and enough budget for regional noise to wash out. Either is better than no comparison group.

Write the plan down in one page and share it with the platform and with whoever will read the result internally. When the numbers arrive, nobody should be able to change the metric, the window or the comparison after the fact.

Compare the metric in both groups and look at the size of the difference and its uncertainty, not only whether it is positive. A small lift on a large audience can still pay; a large lift with a wide range of uncertainty may need a longer or bigger test. Read the platform's report next to any third-party read and your own sales data.

There is no single right share. The holdout must be large enough that a realistic lift would show up clearly against normal variation in your metric, and small enough that you are not giving up too much reach. Brands with many daily conversions can use a smaller holdout share; brands with few conversions need a larger one or a longer test. Ask the platform or your measurement partner to estimate the smallest lift the test could detect at your planned budget and length, and agree that number before launch. If the smallest detectable lift is larger than any result that would be realistic, change the design before spending the budget.

• Changing targeting or budget mid-test, which breaks the comparison.
• Holdout households seeing your ads through another platform or channel.
• Stopping early because the first week looks good or bad.
• Judging on a metric that was not chosen before launch.