Statistics and A/B testing

What are the mean and variance of a Bernoulli variable?

Mean p and variance p(1 - p), where p is the chance of success. The variance is largest at p = 0.5.

What are the mean and variance of a binomial distribution?

For n independent trials with success chance p: mean np and variance np(1 - p).

What is a Poisson distribution?

The count of independent events in a fixed interval at a constant average rate. Its mean and variance are equal.

What is an exponential distribution?

The waiting time between events in a Poisson process. It is memoryless: time already waited does not change the remaining wait.

What is a heavy-tailed distribution?

One where extreme values are much more common than under a normal distribution, such as revenue or latency. Means are unstable, and percentiles describe it better.

What is the law of large numbers?

As a sample grows, its average gets closer to the true average.

What is the central limit theorem?

The average of many independent draws with finite variance is approximately normal, whatever the original distribution's shape.

What is the difference between standard deviation and standard error?

Standard deviation is the spread of individual values. Standard error is the spread of an estimate, such as a mean, across repeated samples.

What is the standard error of a mean?

σ / √n. Cutting it in half takes four times as many samples.

What is the standard error of a proportion?

√(p(1 - p) / n).

What is a 95 percent confidence interval?

A range built so that, over many repeated studies, about 95 percent of such ranges contain the true value. Roughly the estimate ± 1.96 standard errors.

What is the bootstrap?

Estimating uncertainty by resampling the data with replacement many times and recomputing the statistic each time. Resample the unit that was randomized.

What is a null hypothesis?

The default claim being tested, usually that there is no effect or no difference.

What is a p-value?

The probability of a result at least as extreme as the one observed, assuming the null hypothesis is true. It is not the probability that the null is true.

What is the significance level, alpha?

The false positive rate you accept, often 0.05. You reject the null when p is below alpha.

What is a type I error?

A false positive: concluding there is an effect when there is none.

What is a type II error?

A false negative: missing an effect that is really there.

What is statistical power?

The chance a test detects an effect of a given size when it is real. It equals 1 minus the type II error rate; 80 percent is a common target.

What raises a test's power?

More samples, a larger true effect, lower variance in the metric, or a looser alpha.

What is the difference between one-sided and two-sided tests?

A two-sided test looks for a difference in either direction. A one-sided test looks in one direction only and must be chosen before seeing the data.

When is a t-test used?

To compare means when the variance is estimated from the sample. The t distribution has heavier tails than the normal, which matters for small samples.

What is a paired test?

A test on differences within matched pairs, such as the same users under two conditions. It removes variation between units.

What does a chi-square test of independence check?

Whether two categorical variables are related, by comparing observed counts with those expected if they were independent.

What is the Mann-Whitney U test?

A rank-based test of whether values in one group tend to be larger than in the other. It does not assume normal data.

What is the difference between statistical and practical significance?

Statistical significance says an effect is probably not zero. Practical significance asks whether it is large enough to matter. Huge samples make tiny effects significant.

What is the multiple testing problem?

Running many tests raises the chance of at least one false positive. With 20 independent tests at 0.05, that chance is about 64 percent.

What is the Bonferroni correction?

Dividing alpha by the number of tests. It is simple and strict, controlling the chance of any false positive.

What is false discovery rate control?

Limiting the expected share of false positives among results called significant, as Benjamini-Hochberg does. It keeps more power than Bonferroni.

What is Bayes' theorem?

P(A | B) = P(B | A) × P(A) / P(B). It updates a prior belief about A using evidence B.

What is the base-rate fallacy?

Ignoring how rare a condition is. With a rare condition, most positive test results can be false even when the test is fairly accurate.

What does Pearson correlation measure?

The strength of a linear relationship between two variables, from -1 to 1. Zero means no linear link, not no relationship.

What does Spearman correlation measure?

Whether one variable consistently rises or falls with another, using ranks. It catches monotonic, non-linear relationships.

What is a confounder?

A variable that affects both the treatment and the outcome, creating a link between them that is not causal.

What is Simpson's paradox?

A trend that appears in every subgroup reverses or disappears when the groups are combined, because the groups have different sizes or mixes.

What is regression to the mean?

Extreme measurements tend to be followed by more ordinary ones, so selecting on an extreme value guarantees some bounce back.

What is an A/B test?

A randomized experiment that assigns units to a control and a treatment and compares a chosen metric between them.

What is the unit of randomization?

The thing assigned to an arm, such as a user, session, or region. Analysis should be done at that level.

What is the minimum detectable effect?

The smallest true effect a test is designed to detect with the chosen power and alpha. It sets the sample size.

What is a rule of thumb for sample size per arm for a proportion?

About 16 × p(1 - p) / δ² for 80 percent power at two-sided alpha 0.05, where δ is the absolute lift. Halving δ needs four times the users.

What is peeking, and why is it a problem?

Checking results repeatedly and stopping as soon as p drops below 0.05. Each look is a new chance for noise, so false positives rise well above 5 percent.

What is sequential testing?

A test design with boundaries planned for repeated looks, so you can stop early while keeping the stated false positive rate.

What is a sample ratio mismatch?

When the share of units in each arm differs from the planned split by more than chance allows. It signals a bug that can bias every metric.

What is an A/A test?

An experiment where both arms get the same experience. It checks that assignment and analysis give the expected false positive rate.

What is a guardrail metric?

A metric that should not get worse, such as latency, errors, or cancellations, watched alongside the primary metric.

What are novelty and learning effects?

Behavior that changes over time because something is new: a short-lived boost from curiosity, or a short-lived dip while users adjust.

What is CUPED?

A variance-reduction method that adjusts each unit's metric using its pre-experiment value of a related metric. It narrows intervals without bias.

What is the delta method used for in experiments?

Estimating the variance of a ratio metric, such as clicks per page view, when the unit of randomization is the user.

What is interference between experiment units?

When one unit's treatment affects another's outcome, as in marketplaces or social products. It biases simple A/B estimates.

What is a multi-armed bandit?

An adaptive method that shifts traffic toward better arms during the test. It earns more along the way but gives less clean effect estimates.