How do growth teams test ideas before an A/B test?
TL;DR
- Most experiments don't move the metric they were built for. Screening before testing raises the hit rate of what you ship to live traffic.
- A pre-screen reads clarity, appeal and the likely next step for each variant, plus the reason.
- Use it where traffic is thin, cycles are long, or a bad variant is costly: pricing, paywalls, onboarding and offers.
- Neon Blue returns a ranked list of variants by user segment, labeled as simulated. Run your first study free
Neon Blue builds simulations of human behavior. Growth teams pre-screen ideas by showing each variant to a sample that matches their users and reading which ones people understand, trust and would act on. The goal isn't to replace the A/B test. It is to stop spending live traffic and weeks of test time on variants that were never going to win. Neon Blue gives growth teams a simulated audience of their users so they can screen ten ideas and send the best two to a live test.
Why should growth teams screen ideas before testing them?
In Trustworthy Online Controlled Experiments (2020), Ron Kohavi, Diane Tang and Ya Xu report that at one large software company, only about a third of well-designed experiments improved the metric they were meant to improve. Roughly a third had no effect and a third made things worse.
If two in three tests won't win, the real constraint is how many tests you can run. Every slot spent on a weak idea delays one that might have worked. A pre-screen doesn't tell you which idea wins in market; it tells you which ones are clearly confusing, untrusted or unwanted, so they never take a slot.
Which growth decisions benefit most from a pre-screen?
| Decision | Why live testing alone is hard | What the pre-screen reads |
|---|---|---|
| Price or plan change | Risky to show worse prices to real users | Acceptable range, plan choice, reasons |
| Paywall layout | Long cycle to see renewal effects | Plan picked, trial vs pay now, doubts |
| Onboarding steps | Small changes, noisy metrics | Where people would stop and why |
| Win-back or promo offers | Margin cost of over-discounting | Which offer pulls lapsed users back |
| Referral or invite copy | Low base rates need huge samples | Which version people would share |
For price points, see how to measure willingness to pay.
What questions does a growth pre-screen ask?
- Looking at this screen, what do you think you get if you continue?
- Which option would you pick? [Plans or offers shown, plus "none, I'd leave"]
- What, if anything, makes you hesitate?
- If you saw this after a month of using the product, what would you do? [Renew, switch plan, cancel]
- Which of these versions would you be most likely to act on? [Forced choice across finalists]
Keep "I'd leave" as an option. Without it, every variant looks like it converts.
What does a growth pre-screen look like on a simulated audience?
Hypothetical: a language-learning app has six paywall variants that differ in trial length, the default plan and whether the annual price is shown monthly. It has traffic for two live tests in the quarter.
- Audience: US adults who installed a learning app in the past 6 months, split by those who reached day 7 and those who dropped earlier, and by free users versus lapsed subscribers.
- Design: one variant per respondent, the five questions above, then a forced choice between the top three.
- Readout: plan chosen, share who would leave, main hesitation, and a ranked list of variants for each segment.
Example output layout (illustrative, not a real result)
| Variant | Would start trial or pay | Would leave | Main hesitation (coded) | Rank for day-7 users |
|---|---|---|---|---|
| 7-day trial, monthly default | __% | __% | Coded theme | __ |
| 14-day trial, annual default | __% | __% | Coded theme | __ |
| No trial, annual shown monthly | __% | __% | Coded theme | __ |
The two variants that lead for the segment you care about most go to the live test. If early drop-offs and day-7 users disagree, that tells you to segment the paywall, not to pick an average.
How do growth teams combine simulation and live data?
- Screen, then test. Simulated reads narrow the list; live tests pick the winner.
- Keep the sources separate. Report pre-screen scores and live results side by side, never blended.
- Close the loop. After each live test, note whether the pre-screen favorite also won. Over time that tells you where the screen is most useful.
Teams working on landing pages rather than in-app screens can apply the same approach with landing page testing. Subscription businesses will find category-specific examples in how subscription apps research pricing and paywalls.
Questions about pre-testing for growth teams
Does a pre-screen replace A/B testing?
No. It decides what goes into the A/B test. Only live traffic measures real conversion and retention.
How many variants should a growth team pre-screen?
As many as you have serious ideas for, often five to ten. The point is to reach a short list of two or three for live testing.
What if the pre-screen and the A/B test disagree?
Trust the live result for that decision, then look at the reasons from the pre-screen. Disagreements often point to a segment or context the screen didn't include.
How does Neon Blue help growth teams?
Neon Blue builds a simulated audience of your users, screens your variants and returns a ranked list with reasons by segment, labeled as simulated. Start a free first study.