Experiments

What is interference?

One unit's treatment changes another unit's outcome. The comparison is then not a clean treated-versus-control contrast.

What is a sample ratio mismatch?

The split you observe is far from the split you assigned. Something after assignment, often logging, dropped one side.

What is statistical power?

The chance your test detects a true effect of the size you cared about. An underpowered test says 'no difference' when it mostly could not see one.

What is a minimum detectable effect?

The smallest lift the test was sized to find. A smaller estimate, with a wide interval, is not evidence the effect is zero.

What is peeking?

Checking a p-value repeatedly and stopping when it looks good. Each look raises the chance of shipping noise.

What is a fixed horizon?

The sample size or the end date chosen before the test. You read the result then, not the first morning it crosses a line.

What does a p-value say?

How often a result at least this extreme would show up if the treatment did nothing. It is not the chance the treatment works.

Why report an interval, not only a p-value?

The interval shows the range of lifts still compatible with the data, including lifts too small to matter.

What is a practical significance bar?

The lift that would be worth the cost. A statistically clear lift below that bar is a real effect you may still not ship.

What is a novelty effect?

A change in behavior because the thing is new, which fades. A short test can measure the fade window and call it the lasting effect.

What is a primacy effect?

Users need time to adjust, so the early window understates the lasting effect. Ending the test in that window cuts it off.

What is an A/A test?

Both sides get the same experience. It checks that assignment, logging, and the analysis do not invent a difference.

What does CUPED adjust for?

Variation you can see in a pre-period metric, so the treated-versus-control comparison is less noisy.

When is a covariate safe to adjust for?

When it was fixed before assignment. Adjusting for something the treatment changed will hide or invent part of the effect.

Why is a before-after comparison weak?

The world also changed: a sale, a season, a bug. The difference mixes the treatment with everything else that moved.

What is a holdout?

A group kept on the old experience after launch, so you can still see the effect once everyone else has moved.

What is a ramp?

Raising the treated share in steps. It limits damage. It is not a substitute for a read at a planned sample size.

What is a guardrail metric?

A metric that must not get worse, even if the goal metric improves. Shipping on the goal alone can break the guardrail.

Why name one primary metric?

So the decision is about one comparison you sized for. A pile of metrics will produce a winner by chance.

What is the multiple-comparisons problem?

Each extra test has its own chance of a false win. Twenty quiet metrics will show about one false 'significant' result.

When is a segment result trustworthy?

When the segment was named before the test, and it is large enough. A slice found by hunting will look sharper than it is.

What is a heterogeneous effect?

The treatment helps some units and hurts others. The average can look flat while both are real.

What is differential attrition?

One side loses more units from the data than the other. The people who remain are no longer the people you assigned.

What is intent to treat?

Analyze every unit by the group they were assigned to, even if some did not actually see the treatment.

What is an exposure?

The unit actually received the treatment. Conditioning on exposure can bias the result if the reason they saw it is related to the outcome.

What is a trigger analysis?

Restricting to units who reached the point where the treatment could matter. The filter has to be one the treatment did not change.

Why check logging before the readout?

A missing event on one side looks like a change in behavior. The sample ratio is the first place that shows up.

How do you handle a heavy tail?

Decide the rule before you look: a cap, a trimmed mean, or a metric that is less sensitive. Dropping rows after you see them moves the answer.

What is tricky about a ratio metric?

The unit of analysis may not be the denominator. A mean of ratios and a ratio of means answer different questions.

What is a cluster-randomized test?

You assign a group, such as a store or a market, and the people inside it are not independent samples.

What is a switchback?

The same units alternate between treatment and control over time buckets. Carryover between buckets biases it.

What is carryover?

Yesterday's treatment still affects today's outcome, so today's control bucket is not really untreated.

Why is a marketplace hard to randomize by user?

A treated buyer changes the experience of an untreated seller, and the other way around. The units interfere.

What is a geo experiment?

Whole regions are assigned, because the product or the interference cannot be split inside a region. You have few units and a noisy estimate.

What reduces variance before you add users?

A metric closer to the decision, a covariate from before the test, and a unit that is less noisy than a single session.

What sets the test's length, besides sample size?

The cycle of the behavior, such as a week of seasonality, and the time a novelty effect needs to fade.

Why cover full weeks?

A test that starts on Monday and ends on Wednesday compares different kinds of days, unless both sides see the same mix.

What do you do on a sample ratio mismatch?

Stop the readout. The groups are not the groups you assigned, and a clever model will not repair a missing side.

A confident null is not 'no effect.' What is it?

The effect is unlikely to be as large as the one you sized for. A small lift can still be hiding inside the interval.

What do you need in order to ship?

The primary metric cleared the bar you set before the test, the guardrails held, and the assignment check passed. You need all three on the date you fixed in advance.

Why hold out after a win?

The short test cannot see an effect that compounds or fades over months. A small holdout can.

What is a learning agenda?

The decisions you still cannot make, and the test or the analysis that would decide each one. Every line names the decision, the evidence, and the date you will have it.

What does a Bayesian readout add?

A distribution for the lift, and a direct probability the lift clears the bar you care about, given the prior you stated.

What must you state about a prior?

It was chosen before the data, and how strongly it pulls. A prior picked after seeing the result is another way to peek.

What is a sequential test?

A design that allows looking early, with boundaries set so the false-win rate stays at the level you chose.

How does a bandit differ from a test?

It shifts traffic toward the current winner to reduce regret. It is a poor tool when you need a clean estimate of the lift.

What is dilution?

Units assigned to treatment who never saw it. They pull the measured effect toward zero if you analyze by assignment.

What is survivorship in a funnel metric?

You only see users who reached a later step. If the treatment changes who reaches it, the later metric compares different people.

What do you write down before the test starts?

The unit, the primary metric, the size, the segments, and the decision rule. Changing them after you see the data is a second test you did not run.