A message can look strong in a 300 x 250 banner, disappear in a 15-second video and feel forced in a living-room ad. That does not always mean one execution is better. Display, online video and connected TV ask people to process the same idea in different contexts.
A useful Amazon DSP creative test separates the message from the format. It keeps the audience, supply, frequency and outcome as stable as practical, then gives each format enough room to do its job. The goal is not to crown one universal asset. It is to find which message earns attention and action in each environment.
Quick answer: Start with one business question and 2 or 3 message hypotheses. Test each message within display, online video and CTV using comparable audiences, inventory, budgets and frequency rules. Judge winners on format-appropriate metrics, then validate the strongest message against a shared business outcome. Do not compare banner CTR with CTV completion rate.
Naming note: Amazon now describes the combined ads console and Amazon DSP buying surface as Amazon Ads Agent. This article uses “Amazon DSP” because it remains the established programmatic product term and the target search language. Confirm the current workflow, study access and advanced controls for the advertiser's market and seat.
What should an Amazon DSP creative test answer?
A creative test needs a decision, not just a leaderboard. Write the decision before the campaign structure.
Good test questions include:
Does a product-benefit message create more qualified response than a lifestyle message?
Does proof from customers outperform a brand claim for a new audience?
Does a problem-first opening improve attention without reducing brand recall?
Does a direct offer work in display while a story-led message works better in CTV?
Can one message travel across formats, or should each format receive a different role?
“Which ad wins?” is too broad. A result can change because the audience, publisher, device, duration, bid, frequency or landing page changed. A usable test isolates one primary variable and documents the rest.
Creative Testing surveys and live campaign experiments are different
Amazon's named Creative Testing capability uses surveys to understand reactions, preferences and perceptions before a brand campaign. Amazon says advertisers can compare image or video options, copy, messaging tone and visual style. The feature began in the United States and was later expanded in closed beta to Canada, Germany and the United Kingdom, with access models that vary by market.
A live campaign experiment answers a different question: how did the creative perform when exposed to real media conditions? Survey feedback can identify appeal or clarity. In-market delivery can show reach, response and business outcomes. Use both when available, but do not present one as a substitute for the other.
Method | Best question | Typical evidence | Main limitation |
|---|---|---|---|
Creative Testing survey | Which concept, image, copy or tone resonates before launch? | Audience reactions, preferences and perceptions | Stated response is not the same as in-market behavior |
Live media test | Which message drives stronger attention, response or outcomes under comparable delivery? | Reach, viewability, completion, visits, conversions and cost | Media conditions can contaminate the comparison |
Amazon Brand Lift | Did exposed audiences show a change in awareness or intent? | Lift in eligible brand measures | Requires study eligibility, scale and correct setup |
Amazon Marketing Cloud analysis | How did exposures, paths, frequency and outcomes differ? | Privacy-safe aggregate analysis across eligible signals | Custom analysis is not automatically causal |
The creative testing framework at a glance
Step | What the team decides | Output |
|---|---|---|
1. Define the decision | The business question, audience and action the result will change | One-sentence decision statement |
2. Write message hypotheses | 2 or 3 distinct reasons the audience may care | Message matrix with proof and action |
3. Adapt by format | What stays constant and what must change for display, online video and CTV | Format-specific creative brief |
4. Build clean cells | Audience, supply, budget, bid, frequency, flight and destination controls | Trafficking map and naming convention |
5. Set decision rules | Minimum delivery, primary KPI, guardrails and tie-breakers | Written readout rules before launch |
6. Run quality checks | Approval, tracking, rendering, audio, safe zones and landing-page match | Signed launch checklist |
7. Read within format | Which message works best in each environment | Display, online video and CTV winners |
8. Validate across formats | Whether the strongest message improves a shared business outcome | Scale, revise or stop decision |
Step 1: write a test brief that can produce a decision
Use a short brief. If the test requires a deck to explain, the design is probably too complicated.
Business decision: What will change if message A beats message B?
Audience: Which group should respond, and why?
Format role: Is the format expected to create awareness, explain, remind or convert?
Primary outcome: Which metric decides the result?
Guardrails: Which metrics must remain acceptable?
Minimum evidence: What delivery or outcome volume must each cell reach?
Action: Scale, adapt, run a follow-up test or stop.
Example: “For new-to-brand prospects in the United Kingdom, determine whether message A, practical time saved, or message B, premium product quality, produces more qualified product-page visits in display and online video while maintaining acceptable viewability and frequency. Use CTV to compare brand attention and downstream exposed-audience behavior.”
Step 2: build message hypotheses before making assets
A message is the reason to care. A format is how that reason is delivered. Separate them in the brief.
Message component | Message A example | Message B example |
|---|---|---|
Audience tension | The current task takes too long | Most alternatives feel generic |
Promise | Finish the task faster | Choose a product designed for your standard |
Proof | Demonstration of the shorter process | Material, review or performance proof |
Brand role | The practical helper | The trusted specialist |
Next action | See how it works | Explore the range |
Do not test 2 messages that differ only by a headline synonym. The concepts should be far enough apart to create a useful choice, but close enough to support the same campaign objective.
Step 3: adapt the message to display, online video and CTV
Fair testing does not mean forcing the same asset everywhere. Keep the strategic idea constant while allowing format-native execution.
Display: test comprehension and action
Display has little time and limited space. Test one promise at a time. Keep the brand visible, the claim readable and the next action clear. Use a consistent size mix and placement strategy across message cells.
Useful display variables include:
Benefit-led headline versus proof-led headline.
Product image versus human-use context.
Direct offer versus softer discovery action.
Static visual versus restrained motion, when inventory and specs allow.
Amazon publishes separate desktop and mobile display specifications. Build all required sizes from the same message system. If one cell receives more sizes or premium placements, the test is no longer only about the message.
Online video: test the opening and proof sequence
Online video can run in-stream or out-stream across desktop, mobile and tablet. The opening needs to work before the viewer understands the full story. Put the message in the first few seconds, then show proof.
Amazon's online video specifications support several lengths and recommend common durations such as 6, 15, 20, 30 and 60 seconds. Keep duration constant when comparing messages. A 6-second cut and a 30-second cut do not create a clean message test.
Useful online video variables include:
Problem-first opening versus outcome-first opening.
Demonstration versus testimonial proof.
Brand in the first scene versus brand after the hook.
Direct response ending versus brand-led ending.
CTV: test memory, emotion and the living-room action
CTV is a full-screen, usually non-skippable environment. The viewer may be farther from the screen and may not have a clickable interface. Use large text, clear audio and one idea. Avoid treating a television ad as a stretched banner.
Amazon's Streaming TV and Prime Video specifications state that non-interactive ads run in non-clickable environments and should not show button-shaped calls to action that imply clicking. Supply sources and accepted durations differ by market, so confirm the exact inventory before producing every cut.
Useful CTV variables include:
Emotional story versus product demonstration.
One brand promise versus one category problem.
Spoken brand cue early versus a later reveal.
Memorable verbal action versus a simple URL or eligible interactive route.
The Amazon Streaming TV inventory guide explains why Prime Video, Twitch, Fire TV and third-party CTV should remain visible in setup and reporting. Do not let a different publisher mix decide the creative winner.
Step 4: build test cells that isolate the message
Use separate line items or another structure that preserves spend, delivery and outcomes by message. The exact hierarchy depends on the current interface and account controls.
Variable | Keep constant for the first test | Why it matters |
|---|---|---|
Audience | Same inclusion, exclusion and lookback rules | Different audience quality can look like a creative effect |
Geography and language | Same eligible markets and language treatment | Market mix changes cost and response |
Supply | Same inventory package, device and placement controls | Publisher and screen context affect attention |
Bid and optimization | Same bid logic and goal | Algorithms may favor one cell before evidence is balanced |
Budget | Equal opportunity to deliver, not only equal nominal budgets | A starved cell cannot compete fairly |
Frequency | Same cap and measurement window | More exposure can improve recall or create fatigue |
Flight | Run cells over the same dates and dayparts | Weekdays, events and seasonality change behavior |
Destination | Same page or equivalent journey | Landing-page differences contaminate conversion results |
Use a naming pattern such as FORMAT_MESSAGE_AUDIENCE_MARKET_VERSION. Save the final file, preview, click destination, approval status and trafficking ID in one test log. The Amazon DSP campaign audit checklist provides a wider QA structure for settings outside the creative test.
Step 5: choose fair metrics for each format
A cross-format test needs 2 layers of measurement. First, compare messages within each format. Then compare the winning messages against a common business outcome.
Format | Primary diagnostic metrics | Shared outcome candidates | Avoid using alone |
|---|---|---|---|
Display | Viewability, qualified click-through rate, landing-page quality, conversion rate | Qualified visits, product views, leads, purchases, cost per outcome | Raw CTR without placement and quality context |
Online video | Video starts, completion rate, quartiles, viewability, qualified response | Qualified visits, conversions, branded search, brand lift | Completion rate without duration and placement context |
CTV | Reach, frequency, completion, incremental reach, eligible brand lift | Exposed-audience visits, searches, conversions, retail or offline outcomes | Clicks, or completion rate treated as proof of persuasion |
CTR and completion rate answer different questions. Do not rank a banner and a CTV ad on one raw metric. If the business needs one cross-format view, use a shared downstream outcome such as qualified visits, new-to-brand purchases, applications or incremental reach. State the attribution window and eligibility rules.
Step 6: write the decision rules before launch
Pre-written rules reduce the temptation to choose the metric that makes a favorite concept look good.
A practical rule set can include:
Primary KPI: the metric that determines the result within the format.
Guardrails: delivery, frequency, viewability, completion, brand safety and cost conditions that must remain acceptable.
Minimum evidence: enough impressions, completed views, reach or outcomes for a stable read.
Material difference: a threshold large enough to justify production or media changes.
Tie-breaker: the secondary metric used only when the primary result is close.
Stop rule: conditions for creative rejection, tracking failure, overspend or harmful frequency.
There is no universal sample size for every Amazon DSP creative test. Required volume depends on the base rate, expected effect, allocation, variance and confidence standard. For conversion tests, calculate the required sample from the expected conversion rate and the smallest improvement worth acting on. For brand studies, follow current eligibility and study-design requirements.
Step 7: QA the assets and data path
Creative approval is not the same as experiment readiness. Check:
Every message has the same required size, duration and market coverage.
Audio, captions, claims, disclosures and safe zones are correct.
The final asset renders on the real device or environment.
Click URLs, tags, pixels and conversion events fire correctly.
Campaign, line item and creative names match the test log.
Reports can separate message, format, audience, supply and outcome.
Attribution windows, time zones and deduplication rules are documented.
Automated optimization will not collapse delivery into one cell before the planned read.
Amazon has introduced creative galleries, pre-flight recommendations and policy guardrails over time. Use current account tools to reduce trafficking errors, but keep a human QA pass for the message, landing experience and experiment design.
How to read the results without mixing signals
Read delivery first
Confirm that each cell had a fair chance to serve. Compare spend, impressions, reach, frequency, CPM, inventory mix and approval timing. A message that received cheaper placements or more frequency may appear stronger for reasons unrelated to the idea.
Read the primary metric within each format
Name a display winner, an online video winner and a CTV winner only when the evidence clears the pre-set rule. A different winner by format is a useful result. It may show that the audience needs a direct benefit in display and a fuller proof story in video.
Check business outcomes
Connect format results to the outcome the company values. The Amazon DSP measurement and Amazon Marketing Cloud guide explains how to plan events, paths and custom analysis. Platform-attributed conversions are evidence, but they are not always proof of incrementality.
Look for interactions
A message may work only for one audience or supply environment. Report that interaction instead of averaging it away. If message A wins with prospecting audiences but message B wins with retargeting, the next campaign may need both.
A practical 3-wave test plan
Wave 1 - concept screen: Use Creative Testing surveys when eligible, or a controlled low-cost media screen, to remove unclear concepts. Keep 2 strong messages.
Wave 2 - within-format test: Compare the 2 messages separately in display, online video and CTV. Preserve the planned controls and minimum evidence.
Wave 3 - validation: Put the winning message for each format into a coordinated plan. Measure shared outcomes, reach overlap, frequency and contribution.
This sequence protects the media budget from weak concepts without pretending that survey preference guarantees campaign performance. It also avoids building a large asset matrix before the core message is clear.
Common Amazon DSP creative testing mistakes
Changing message, format and audience together. The result cannot identify the cause.
Testing too many cells. Thin budgets create slow delivery and noisy conclusions.
Using one raw KPI across formats. A CTV ad should not lose because it cannot generate banner clicks.
Letting algorithms choose too early. Delivery can concentrate before each cell reaches the planned evidence threshold.
Ignoring supply mix. Premium and low-quality placements can make the same creative look different.
Comparing different durations. Video length affects completion and available inventory.
Calling a survey the final answer. Preference should inform the campaign, then be validated in market.
Scaling a tiny difference. The gain should be large enough to justify production and operational cost.
Skipping the landing page. A strong ad can be blamed for a weak or mismatched destination.
Ending at platform reporting. Reconcile media results with analytics, commerce, CRM or offline outcomes.
How Ad Geeks runs creative tests
Ad Geeks can turn a creative question into a controlled Amazon DSP activation. The work can include message design, asset matrix planning, format adaptation, trafficking, creative QA, budget allocation, pacing, reporting and outcome analysis.
The operating path is practical:
Audit the objective, audience, existing assets and measurement.
Write a small set of testable message hypotheses.
Adapt the ideas for display, online video and CTV without changing the strategic promise.
Build clean campaign cells and verify delivery controls.
Report winners within format and connect them to shared outcomes.
Turn the result into the next production and media decision.
We can design the creative matrix, manage Amazon DSP activation and report the result across display, video and CTV.
Frequently asked questions
What is Amazon DSP creative testing?
It can refer to Amazon's Creative Testing survey capability or to controlled live media experiments in Amazon DSP. Surveys measure reactions and preferences before or around a campaign. Live tests compare delivery, attention, response and business outcomes under real media conditions.
Can I compare display, online video and CTV directly?
Start with 2 or 3 distinct message hypotheses. More cells require more budget, time and outcomes. It is usually better to learn clearly from 2 messages than to spread delivery across 8 small variations.
How many creative variants should I test?
Lorem Ipsum is simply dummy text of the printing and typesetting industry.
Should every format use the exact same creative?
No. Keep the message, proof and brand role consistent, but adapt the execution. Display needs fast comprehension, online video needs a strong opening and sequence, and CTV needs a clear living-room story with an appropriate action.
What is the best KPI for a creative test?
Choose the KPI from the format's job and the business decision. Display may use qualified response or conversion. Online video may use attention plus downstream action. CTV may use reach, brand lift and exposed-audience outcomes. Set guardrails for delivery and cost.
How long should an Amazon DSP creative test run?
Run it until each cell reaches the pre-defined evidence threshold across a representative part of the flight. Do not choose a universal number of days. Audience size, inventory, budget, conversion volume and weekly patterns determine the required duration.
Can Amazon Marketing Cloud prove which creative caused a sale?
Amazon Marketing Cloud can support privacy-safe aggregate analysis of eligible exposure paths and outcomes. That does not make every comparison causal. A defensible causal claim needs a suitable control, holdout or experiment design.
Is Amazon Creative Testing available in every country and account?
No. Amazon's announcements show different availability and access models by country and service type. Confirm current eligibility, cost, study access and setup with the account team or operating partner before including it in the plan.
