Statistics and A/B testing

Question 1

Response times are roughly normal with a mean of 200 ms and a standard deviation of 20 ms. About what share of requests fall between 160 and 240 ms?

Accepted Answer

About 95 percent

Question 2

Real response times usually have a long right tail. Why is reporting only the mean and standard deviation misleading for them?

Accepted Answer

A few very slow requests pull the mean up and say little about what most users see; percentiles like p50 and p99 describe it better.

Question 3

A coin with a 0.3 chance of heads is flipped 100 times. What are the mean and standard deviation of the number of heads?

Accepted Answer

Mean 30, standard deviation about 4.6

Question 4

For a yes-or-no outcome with success rate p, at what value of p is the variance largest?

Accepted Answer

p = 0.5

Question 5

Support tickets arrive at an average of 4 per hour, independently. Which distribution fits the count per hour, and what is its variance?

Accepted Answer

Poisson, variance 4

Question 6

Daily ticket counts have a mean of 50 and a variance of 400. What does that tell you about the Poisson assumption?

Accepted Answer

The data is overdispersed, likely because the rate varies by day, so a Poisson model will understate uncertainty.

Question 7

Time between independent arrivals at a steady rate follows an exponential distribution. What is its notable property?

Accepted Answer

It is memoryless: having waited 10 minutes does not change the chance of an arrival in the next minute.

Question 8

Revenue per user is mostly zero, with a few very large purchases. What does that imply for an A/B test on revenue?

Accepted Answer

The variance is huge, so the test needs many users, and capping extreme values or using a variance-reduction method can help.

Question 9

What does the central limit theorem say?

Accepted Answer

The average of many independent draws with finite variance is close to normally distributed, whatever the shape of the original distribution.

Question 10

When can the central limit theorem be slow to kick in?

Accepted Answer

When the data is very skewed or heavy-tailed, so a few extreme values dominate the mean.

Question 11

What does the law of large numbers say?

Accepted Answer

As the sample grows, the sample average gets closer to the true average.

Question 12

A metric has a standard deviation of 20 per user. With 400 users, what is the standard error of the mean?

Accepted Answer

1

Question 13

How many times more users do you need to cut a standard error in half?

Accepted Answer

4 times

Question 14

A conversion rate is 10 percent on 10,000 users. What is the standard error of that rate?

Accepted Answer

0.003

Question 15

Using that standard error of 0.003, what is an approximate 95 percent confidence interval for the 10 percent rate?

Accepted Answer

About 9.4 to 10.6 percent

Question 16

Which reading of a 95 percent confidence interval is correct?

Accepted Answer

If you repeated the study many times, about 95 percent of the intervals built this way would contain the true value.

Question 17

What is the difference between a standard deviation and a standard error?

Accepted Answer

Standard deviation describes the spread of individual values; standard error describes how much an estimate, such as a mean, would vary across samples.

Question 18

You want a confidence interval for the median session length, and no simple formula comes to mind. What is a general approach?

Accepted Answer

Bootstrap: resample users with replacement many times, compute the median each time, and take the 2.5th and 97.5th percentiles.

Question 19

Each user has many sessions, and a team bootstraps over sessions to get an interval for a per-user test. Why is the interval too narrow?

Accepted Answer

Sessions from the same user are correlated, so treating them as independent overstates how much data you have.

Question 20

When should a t-test be used instead of a z-test for comparing means?

Accepted Answer

When the variance is estimated from a small sample; the t distribution's heavier tails account for that extra uncertainty.

Question 21

What is the null hypothesis in a typical A/B test?

Accepted Answer

There is no difference between the arms in the metric being tested.

Question 22

A test reports p = 0.03. What does that mean?

Accepted Answer

If there were truly no difference, a result at least this extreme would happen about 3 percent of the time.

Question 23

A test gives p = 0.40. What can you conclude?

Accepted Answer

The data is not strong evidence of a difference, which is not the same as showing there is none.

Question 24

What is a type I error?

Accepted Answer

Concluding there is an effect when there is none, a false positive.

Question 25

A test has 80 percent power to detect a 2 percent lift. What does that mean?

Accepted Answer

If the true lift is 2 percent, the test will reach significance about 80 percent of the time, and miss it about 20 percent of the time.

Question 26

Which change raises a test's power, all else equal?

Accepted Answer

Adding more users.

Question 27

When is a one-sided test appropriate?

Accepted Answer

When you decided before the test that only one direction matters, and you would act the same on no change as on harm.

Question 28

A two-sided z-test gives z = 1.96. What is the p-value?

Accepted Answer

About 0.05

Question 29

Which test fits comparing conversion rates between two arms with thousands of users each?

Accepted Answer

A two-proportion z-test, or the equivalent chi-square test on the 2×2 table.

Question 30

The same 50 users rate an old and a new ranking. Which test fits?

Accepted Answer

A paired test on each user's difference, since the same people are in both conditions.

Question 31

When would you use a Mann-Whitney U test instead of a t-test?

Accepted Answer

When the data is skewed or ordinal and the samples are small, so the t-test's assumptions are doubtful.

Question 32

Which test checks whether two categorical variables, such as device type and plan choice, are related?

Accepted Answer

A chi-square test of independence on the table of counts.

Question 33

A change lifts average time on page by 0.2 seconds with p = 0.0001 on 50 million users. What should the team ask next?

Accepted Answer

Whether 0.2 seconds matters to the business at all. With huge samples, tiny effects become significant.

Question 34

A team runs 20 independent tests at alpha 0.05, and none of the changes do anything. What is the chance at least one comes out significant?

Accepted Answer

About 64 percent

Question 35

With 20 tests, what per-test threshold does a Bonferroni correction give for an overall alpha of 0.05?

Accepted Answer

0.0025

Question 36

How does controlling the false discovery rate (for example, Benjamini-Hochberg) differ from Bonferroni?

Accepted Answer

It limits the expected share of false positives among the results you call significant, which keeps more power when there are many tests.

Question 37

An experiment shows no overall effect, so an analyst splits results by 30 segments and finds one significant at p = 0.04. How should that be treated?

Accepted Answer

As a hypothesis to test in a new experiment, since one hit among 30 looks is about what chance alone would give.

Question 38

What is p-hacking?

Accepted Answer

Trying many analyses, metrics, filters, or stopping points and reporting the ones that give p below 0.05.

Question 39

Why should the primary metric of an experiment be chosen before the test starts?

Accepted Answer

So the decision does not depend on picking whichever of many metrics happened to move.

Question 40

A disease affects 1 percent of people. A test catches 90 percent of cases and gives a false positive for 5 percent of healthy people. Someone tests positive. About how likely is it that they have the disease?

Accepted Answer

About 15 percent

Question 41

That screening example is a lesson for fraud and abuse models too. What is it?

Accepted Answer

With a rare positive class, even a fairly accurate model can produce mostly false alarms, so precision depends heavily on the base rate.

Question 42

How does Bayes' theorem relate the probability of A given B to the probability of B given A?

Accepted Answer

P(A | B) = P(B | A) × P(A) / P(B)

Question 43

How does a Bayesian analysis of an A/B test usually report its result?

Accepted Answer

As a posterior distribution for the effect, which supports statements like 'there is an 85 percent chance B is better'.

Question 44

What is the first thing to decide when designing an A/B test?

Accepted Answer

The question, the primary metric, the unit of randomization, and the smallest effect worth detecting.

Question 45

What is the minimum detectable effect (MDE) in test planning?

Accepted Answer

The smallest true effect the test is designed to detect with the chosen power and alpha.

Question 46

A rough sample size per arm for comparing proportions, at 80 percent power and two-sided alpha of 0.05, is 16 × p(1 − p) / δ². For a 10 percent baseline and an absolute lift of 1 point, how many users per arm is that?

Accepted Answer

About 14,400

Question 47

Using the same rule, what happens to the needed sample size if you want to detect a lift half as large?

Accepted Answer

It grows four times.

Question 48

A test needs 200,000 users per arm, and the page gets 20,000 eligible users a day split across two arms. About how long must it run?

Accepted Answer

About 20 days, and rounding up to whole weeks helps cover weekday patterns

Question 49

Why is the unit of randomization important?

Accepted Answer

It decides what counts as independent; analyzing at a finer level than you randomized makes the results look more certain than they are.

Question 50

Why randomize by user rather than by page view for a ranking change?

Accepted Answer

Users would see the ranking change from one page to the next, which is a confusing experience and mixes the two treatments for each person.

Question 51

Why is it a problem to check results every day and stop the test the first time p drops below 0.05?

Accepted Answer

Each look is another chance for noise to cross the line, so the real false positive rate becomes much higher than 5 percent.

Question 52

The team really does need to check results daily and stop early on a clear win or harm. What is the right approach?

Accepted Answer

Use a sequential testing method with boundaries designed for repeated looks.

Question 53

An experiment meant to split users 50/50 has 50,800 in treatment and 49,200 in control. What should you do?

Accepted Answer

Treat it as a sample ratio mismatch and find the cause before trusting any metric, since a split this uneven is very unlikely by chance.

Question 54

What commonly causes a sample ratio mismatch?

Accepted Answer

A bug that makes one arm crash or log less, redirects that drop users, bot filtering that treats arms differently, or assignment that depends on something affected by the treatment.

Question 55

What is an A/A test, and why run one?

Accepted Answer

Two identical arms. It checks that the assignment, logging, and analysis give about 5 percent false positives and no sample ratio mismatch.

Question 56

A new homepage shows a big lift in clicks in the first three days, which fades by week three. What is a likely explanation?

Accepted Answer

A novelty effect: returning users click on what is new at first, then settle back.

Question 57

The opposite pattern appears: a redesign hurts at first and slowly recovers. What is that called?

Accepted Answer

A change-aversion or learning effect, as users adjust to the new layout.

Question 58

What is a guardrail metric?

Accepted Answer

A metric that should not get worse, such as latency, errors, or unsubscribes, checked alongside the primary metric.

Question 59

A new recommendation model raises clicks by 4 percent but lowers 30-day retention slightly. How should a team think about it?

Accepted Answer

Clicks may be a poor proxy here; the decision should weigh the metric closer to long-term value, and the retention change needs a proper look.

Question 60

Click-through rate is total clicks divided by total page views, while users are what was randomized. Why does its variance need special treatment?

Accepted Answer

It is a ratio of two per-user sums, so a naive formula that treats each page view as independent understates the variance; the delta method or a bootstrap over users handles it.

Question 61

What does CUPED do in an A/B test?

Accepted Answer

Adjusts each user's metric using their pre-experiment value of a related metric, which removes predictable variation and shrinks the confidence interval.

Question 62

Why must the covariate used in CUPED be measured before the experiment starts?

Accepted Answer

If the treatment could affect it, adjusting for it would remove part of the real effect and bias the estimate.

Question 63

In a two-sided marketplace, a pricing test gives treatment riders lower prices. Why might the result be misleading?

Accepted Answer

Treatment riders compete with control riders for the same drivers, so the arms interfere and the measured effect does not match a full rollout.

Question 64

A messaging feature is tested by randomizing individual users. Why might that underestimate the effect?

Accepted Answer

Treated users' friends are often in control and cannot use the feature, which mutes the effect; randomizing by friend cluster helps.