Design acquisition tests and interpret retention

Trace exposure to value

Define exposure, entry, activation, core completion and reuse with numerators, denominators and windows. Activation means first value, not necessarily sign-up. Compare entry cohorts; a growing weekly total can reflect new traffic rather than retention.

Give an experiment one primary question

State a mechanism for a channel or message and predefine the primary outcome and cost. Randomization reduces some confounding; without it, record audience, time and channel differences and avoid causal lift claims. With small traffic report counts and uncertainty.

Worked example · fictional

From a problem to inspectable work

Fictional: A has 100 visits and 20 completions; B has 40 and 12, at a higher cost.

1. Completion rates are 20% and 30%, but costs, audiences and timing differ.
2. Compare cost per completion and reuse within the same window, not just the 30% rate.

Report counts, rates, cost and windows, then plan a comparable test; do not claim B caused a ten-point lift.

Failure case and repair

Treat social likes as product activation.

Core task value remains unobserved.

Define and observe a first-value event.