Most growth experiments should fail. (If not, test bolder stuff). But, when an experiment fails, how do you know if you’re moving forward? Or wasting precious time? You can feel it.
Do you feel smarter? Does it inform your next experiment?
- Can you rule out an option?
- Did you find some qualitative insight?
- Did some other metric move? In an unexpected way?
- Did you learn something about your tech or process?
Ultimately you need to refine your understanding of the prospects' conception of the problem space, their expectations, assumptions, doubts and behaviours.
You absolutely should get smarter after each experiment. If not, you’re going sideways.
How to Fix This?
If your experiments are not making you smarter, fix that with two simple process changes:
Better up-front experiment thinking: Get very clear on your (specific written) hypothesis and success metric. Ask people to write predictions. Isolate one variable (e.g. message or channel) and/or test bold things that challenge your most important beliefs & assumptions about your customers.
More post-experiment thinking: People tend to overthink what might go wrong and underthink what actually did go wrong. For each failed experiment, force yourselves to spend a full 20 minutes looking through the results and talking through why the results differed from your predictions. List out as many ideas as possible.
It’s hard to know if you’re making progress when the numbers aren't going up. But it’s easy to know if you’re learning. And if you're learning, you're making progress.