DMA (Designated Market Area) tests are one of the most powerful tools for measuring marketing incrementality. But they are also easy to get wrong. Here are some hard-won lessons from running these tests myself.
Why DMA Tests Matter
Unlike user-level attribution, DMA tests can capture:
- Brand effects that take weeks to materialize
- Cross-device conversions
- Word-of-mouth and social spillover
- The full funnel from awareness to purchase
This is why I keep coming back to them, despite the complexity.
Common Pitfalls
1. Not Enough DMAs
Statistical power in geo-experiments comes from the number of geographic units, not total users. With 20 DMAs, you need a massive effect to detect anything significant.
Rule of thumb: Aim for 50+ DMAs minimum, ideally 100+. I know this limits the channels you can test, but underpowered tests are worse than no test at all.
2. Spillover Between Markets
Consumers travel. Digital ads cross borders. If your “holdout” market is contaminated by treatment, your estimates are biased toward zero.
Mitigation: Use buffer zones, exclude border areas, or model spillover explicitly. I have learned to be paranoid about this.
3. Ignoring Seasonality
Marketing effects vary by season. A test run in November tells you little about February.
Best practice: Run long enough to capture at least one full cycle, or use time-series methods that account for seasonality.
A Better Approach: Synthetic Control
Instead of simple treatment vs. control comparisons, synthetic control methods construct a weighted combination of control markets that best matches the treated market pre-intervention.
This handles:
- Heterogeneity across markets
- Trends that differ by region
- Noisy outcome data
I have had much better results with synthetic control than naive DiD for geo tests.
Key Takeaways
- More markets > more users per market
- Design for spillover from the start
- Pre-registration prevents p-hacking
- Synthetic control often beats simple DiD
- Always run power calculations before launch
If you skip the power analysis, you are setting yourself up for an inconclusive result.