Experiments

1.A grocery chain flips a new endcap by store. The readout's standard error treats each receipt as its own draw. There are 30 stores and about 400,000 receipts. What did that standard error forget?

2.Tutoring software is randomized by classroom. A dashboard t-test ignores classroom and reports a tiny p-value on 8,000 students in 25 rooms. Which count belongs in the uncertainty?

3.A bank randomizes a new overdraft alert by branch. Risk wants a customer-level p-value because the alert is addressed to customers. Why is that the wrong precision?

4.Creators are randomized to a new payout rule. The analyst averages daily revenue and treats each creator-day as a fresh sample. Most creators appear on many days in the same arm. What is inflated?

5.The memo says shoppers were randomized. The config hashes the new shelf copy by store, and every shopper in a store sees the same copy. A shopper-level model is already fitted. What should you correct first?

6.Search ranking is randomized by query, since one person issues many queries. The scorecard keeps one row per searcher and one p-value on searchers. Most people contributed dozens of assigned queries. Which unit did the scorecard fold away?

7.A food-delivery test gives treated diners a free delivery credit. Treated diners order more, couriers get busier, and untreated diners see longer arrival times. The diner contrast shows a large gain. What else is inside that gap?

8.A social app boosts treated users' posts into friends' feeds. Many of those friends were assigned to control, and control users comment more than they did before the test. How does that leak push on the measured lift?

9.Two sellers share a small catalog. The test discounts seller A, shoppers buy A, and seller B's items stay in stock more often. B is in control. What does the seller contrast include besides the discount?

10.A rideshare bonus is randomized across drivers in one city. Treated drivers stay online longer, open trips are taken faster, and untreated drivers earn less. Why is "the bonus raised driver earnings" too large a claim?

11.A collaboration suite assigns a new comment tool by user inside the same document. Control coworkers still read threads the treated user reorganized. Which assumption is failing?

12.An auction test could randomize bidders. Product instead assigns the new bid logic by region, so bidders in one auction sit on the same arm. What are they trying to keep out of the contrast?

13.The team refreshes a dashboard every morning and will ship the first day the conversion p-value drops below 0.05. That happens on day 4 of a planned 21-day run. Why is the day-4 read too eager?

14.An analyst says the daily checks do not count because nobody flipped a stop flag in the platform. Product ships the afternoon the chart turns green. What actually sets the error rate?

15.On the planned end date the primary p-value is 0.08. The team adds four days and stops when it hits 0.04. How should you label that extension?

16.The team avoids opening the primary metric. A bot posts in chat when any of ten secondary metrics crosses a naive 0.05, and that ping is enough to launch. What is still true?

17.A written rule says to stop on day 14 if the revenue interval lies entirely below the cost of keeping the feature. On day 14 it does, and they stop. How is this different from waiting for a flattering sign?

18.The platform allows a daily look and spends alpha so early days use a stricter bar. On day 6 the estimate clears that day's bar and the team stops. When is that stop a fair one?

19.Forty secondary metrics sit on the scorecard. Shares per session cross a naive 0.05 line, and primary conversion does not. The team wants to ship on shares. What is the problem?

20.The overall result is flat. Cuts by device, country, tenure, and hour continue until tablet users in one country look positive. The team wants to ship only there. What should happen before that ship?

21.The plan named seven-day retention. Retention is flat and items added is up, so the write-up calls items added the primary success metric. What moved?

22.Each squad ships when its own test beats a naive 0.05. Dozens of squads do this every month, and the winners are the tests everyone hears about. What should a reviewer expect from that stack?

23.Four button colors run against one shared control. The best of the four beats control on a single-comparison line, and a teammate treats that as one planned comparison. What is being ignored?

24.The plan's only subgroup read is new versus tenured accounts, with a stricter bar written down for those two looks. New accounts lift and tenured accounts are flat. How is this unlike hunting through slices?

25.The test was sized to detect a 2 percent relative conversion lift. The estimate is 0.3 percent, and the interval still runs from about -1.5 percent to 2.1 percent. A note says the team proved there is no effect. What is wrong with that note?

26.Sample size was computed for a yes-or-no conversion. The decision metric is revenue per user, and a few large orders dominate it. Why is the conversion math not enough?

27.A layout is randomized across 18 stores. Hundreds of thousands of shoppers are in the file, and the test is called well powered. What was overstated?

28.The team needs to detect a drop in a high-stakes action that happens about once in 800 sessions. On a normal two-week sample the interval is wide enough to hide a serious drop. What is the honest planning line?

29.The plan needs about 200 stores. They launch in 25 stores, then add an unclustered shopper-level analysis to fill the sample. Why does that add-on fail?

30.A billing test was meant to run until the interval could resolve a 1 percent revenue change. It was cut off at about a third of that sample because a campaign ended. The estimate is slightly positive and not significant. What did the short window give up?

31.The reported metric is clicks per session. Treatment creates many extra sessions and only a few extra clicks, so the ratio falls. A note says engagement fell. What should you unpack before accepting that?

32.Support hours per active account fall. A new digest marks many barely-active accounts as active, and total hours rise only slightly. A note says the support load improved. What should you separate first?

33.Revenue per click rises. Clicks fall by more than revenue falls, so total revenue is down. A launch note says the page got better at making money from attention. Which comparison is easy to skip?

34.Total minutes in a notes app rise. Notes opened rise by a similar percent, so minutes per note looks flat. A review says the app did not change usage. What did the flat ratio miss?

35.Conversion is purchases divided by sessions. Treatment adds a bug that splits one visit into several sessions without adding purchases. The ratio falls. What might you be looking at?

36.The refund rate, refunds divided by orders, is almost unchanged. A faster checkout raises both orders and refunds. Finance wants to ship because the rate is flat. What is incomplete about that comfort?

37.A celebration animation lifts next-day return for nine days, and the gap is gone by week five. A nine-day readout would have shipped it. What is the risk in that short window?

38.Long-time users of a desktop editor are slower for two weeks after a layout change, then match their old speed. A six-day test on that existing base says the layout lost. What is that window mostly seeing?

39.A "new" badge sits on the treatment icon. Clicks jump. A later test that keeps the icon and drops only the badge brings clicks back down. The ship note treated the jump as the steady rate of the icon. What was misread?

40.Week-one return jumps when a coaching prompt is new. Week-four return for that same cohort matches control. A launch review quotes week one as the ongoing gain. What is in the week-one number that week four leaves behind?

41.A dark-mode default delights people on day one and is neutral by day twenty among those who started on day one. New installs in week three still show the day-one bump. Someone says the effect came back. What is a better reading?

42.The team wants to ship a sound effect because a three-day test showed more sessions. A held-out 5 percent kept the old silence for six weeks, and that gap closed after week two. Which series is the better steady-state read?

43.A client retry logs the purchase event twice on the treatment page and once on control. The client revenue chart rises in treatment. Settled charges in billing do not. Which series says whether more money was taken?

44.The assignment service stored nothing during the test. A week later an engineer recomputes buckets from the current hash, after a salt change on day three. What is that recomputed label worth?

45.The client renders the new page, but a stale server cache writes the log line as control. The readout trusts the server log. What will the contrast describe?

46.Two different tests both write rows under the name checkout_v2. An analyst filters that name and pools the rows. What can that pool become?

47.Purchase events include the bucket, and people who do not purchase have no row. The analyst reports each arm's share of those purchase rows. The split of purchases is even. What was computed?

48.A batch job attaches yesterday's purchases to today's bucket. Buckets can change overnight when a flag defaults on. What did that join get wrong?

49.The design is an even split. The table comes back 47/53, a simple check says that split is wildly unlikely, and conversion looks better in the smaller arm. What is the first move?

50.The treatment build crashes on a common phone. Those sessions never log a conversion event, the arm counts skew, and conversion among sessions that remain looks fine. If you ignore the crashes, what are you estimating?

51.Control and treatment live on different URLs. The treatment redirect fails for a slice of locked-down browsers, and those people never arrive. Why is conversion among arrivals a misleading scorecard?

52.The test runs at 10 percent treatment for two days, then at 50 percent. An analyst pools every day and expects an even-split check. The check fails. What was mis-specified?

53.The assigned split is even and healthy. Among users who reached checkout, treatment is 61 percent, and the new design adds a shortcut into checkout. A teammate calls the whole test broken because of that checkout split. What is easy to miss?

54.Arm sizes do not match the design, and nobody has found the bug. A teammate says to read the lift anyway because the missing users are probably bots. Why is that unsafe?

55.A recommendation model ships to everyone over a week because error rates stayed calm. Revenue that week is up, and leadership asks for the experiment's lift. What is missing?

56.After a spell-correct launch, 5 percent of users stay on the old corrector for a month while everyone else gets the new one. Why is that more informative than the ramp week?

57.Engineering turns on a new scoring pipeline first for internal staff and friendly design partners, then cites those accounts' engagement as the treatment effect. What kind of selection is that?

58.A feed change may alter what creators publish, which then changes recommendations months later. A one-week user-level test is flat. Why can a long holdout still be worth running?

59.A 2 percent holdout watches a rare cancellation. After a month the interval still covers both a serious harm and a modest gain. The gap is not significant. Which conclusion fits that interval?

60.Product wants the editor holdout turned off because a few held-out customers complained they lack the new editor. What do you give up if you turn it off to stop the complaints?

61.A marketplace alternates a courier bonus by hour. Couriers stay online across the boundary and keep the previous hour's pace for a while. What does that carryover do to an hour-level contrast?

62.A city flips a delivery fee every day at midnight. The analyst compares Mondays with the fee to Mondays without it, pooling many weeks. Why is the weekday match not the whole design?

63.A switchback is on the table for a driver bonus because drivers in one city share the trip pool. A coworker still wants to randomize drivers inside the hour. What does the switchback buy that the within-hour split does not?

64.A fee is on for two hours, then off for two hours, all week. A national holiday lands entirely inside on-hours. Those hours look wonderful. What is a credible rival explanation?

65.You have 14 switched days and a huge number of orders. The p-value treats every order as independent and is tiny. Which sample size should discipline that confidence?

66.A team runs a driver-bonus switchback and a diner-coupon test in the same city on the same days. Each readout ignores the other test. Why might both contrasts be hard to read?

67.Users are bucketed at login, but a new search box appears only on the results page. About half of each bucket never searches, and both arms reach that page at the same rate. Which group answers what the box did for people who saw it?

68.The readout keeps users who opened the help panel. The new panel is brighter, so more people open it, including people who were only curious. What goes wrong in that comparison?

69.A new toggle sits three menus down. The effect on everyone who opened the app is about zero. Design wants to quote only people who opened the third menu. Launch will leave the toggle buried. Which number matches that launch?

70.Both arms can open a tutorial. Treatment auto-opens it. Completion is compared among people who opened it, and treatment's openers include many who would not have opened it themselves. What is that comparison?

71.The average effect on everyone assigned is a small positive. Among people who opened the panel it is a large positive, and the open rate matches across arms. How can both be true without a logging bug?

72.A blocker stops the treatment script from logging impressions, but the new styling still changes the page. The readout keeps only users with an impression log. Who is left out of treatment?

73.People differ a lot in last month's purchases. This month's purchases are the outcome. A coworker adjusts using last month, which ended before anyone was assigned. What is that adjustment for?

74.A purchase test is adjusted using this week's session count. The home-page change under test is what moves sessions. Why is that adjustment dangerous?

75.The covariate is purchases in the week before the analysis window. Assignment was a week earlier, and the new checkout was already live for treatment during that week. What is wrong with calling those purchases pre-period?

76.An analyst residualizes the outcome on a covariate measured after assignment, saying the feature should not touch it. Nobody checks whether that covariate's mean matches across arms. What is the gap?

77.The pre-period is the 28 days before each user's first exposure. Treatment users hit exposure earlier because the new prompt is aggressive, so their window sits earlier on the calendar than control's. What can that shift mix in?

78.A CUPED-style adjustment cuts the variance a lot. The point estimate also moves from a small negative to a clear positive, and the covariate is this week's use of the new module. A teammate says the move is just precision. What else could it be?

79.The primary metric is flat. Support contacts, listed as a guardrail, fall and cross a naive significance line. The team ships because a guardrail won. What is that guardrail for?

80.The test is sized for conversion. Crashes are rare, the crash gap is not significant, and the review says the build is safe. Why is that a weak safety claim?

81.Search relevance rises and time-to-result rises with it. Relevance was primary, and they ship the same afternoon. A written rule says not to ship if time-to-result worsens past a set amount, and it did. What was skipped?

82.A promotional email lifts purchases and also lifts unsubscribes. Growth says unsubscribes belong to another team. Why might you still block the ship?

83.A faster checkout raises completed orders and raises refunds by a similar amount. The team wants to ship on orders. Refunds were named as a guardrail with a maximum increase that would block a ship, and the increase is past it. What is the refund series saying?

84.The guardrail board uses the same naive 0.05 as the primary, on twelve guardrails, and blocks a ship if any one of them loses. Sound features keep getting blocked. What is clumsy about that rule?

85.A new search blender ships to everyone the morning of a sitewide sale. Conversion jumps, and a before-after note credits the blender. What else can explain the jump?

86.New map pins launch on Monday. A long-planned email sale starts the same Monday, only in the regions that received the pins. Other regions were held for a later sale. Regional conversion rises where the pins launched. What is tangled?

87.A 10 percent holdout did not get a new pricing page. A sitewide sale hits the holdout and everyone else alike. Treated conversion rises more than holdout conversion during the sale. What can you say that a before-after cannot?

88.Last year's sale week is used as the control for this year's sale week, when a new checkout was on for everyone. The checkout is credited with beating last year. What is the weak point?

89.An onboarding checklist looked flat in a quiet week. It is turned on for everyone on the first morning of a sale, and conversion is much higher than in the quiet week. Leadership says the quiet-week test was wrong. What is a more careful split?

90.A flash sale is announced in a push that deep-links into a new checkout. Both arms of an ongoing checkout test receive the push, but the deep link resolves only for treatment. Treatment conversion spikes for three hours. What might that spike be?

91.Average revenue is flat. New accounts spend more and long-tenure accounts spend less, by a similar total. A full launch matches today's tenure mix. What does the average hide?

92.The test enrolls monthly-active people who opted into betas. The ship will default on for dormant accounts too. Beta users like the denser inbox. What is the extrapolation risk?

93.One country is about 1 percent of traffic and shows a huge lift on a few thousand users. The team wants a country-only ship. What is the caution?

94.The test was sized to see an overall lift. The review then treats the mobile-minus-desktop difference in lifts as a precise estimate. Why is that difference shaky?

95.The feature helps in both countries. Mid-test, traffic swings toward the country with the smaller effect, and the pooled lift shrinks. A teammate says the feature stopped working. What else changed?

96.People assigned in week one show a lift. People assigned in week three do not. In week two a headline changed for everyone, outside the test. Why might the cohorts disagree even if the feature's own effect is steady?

97.Treatment crashes before the conversion event can be logged. The analyst drops sessions with a missing conversion and compares the rest. What happens to the crashes?

98.A satisfaction survey is offered to both arms, and treatment users close it more often. The write-up averages scores among people who submitted. What is missing from that average?

99.Refunds are stored only when a warehouse scan succeeds. The new checkout produces more orders from a region whose scanner was down all week, so those orders have no refund rows. Treatment's refund rate looks lower. What should you check?

100.The metric is 14-day retention. Anyone without 14 days since assignment is dropped. Treatment was ramped up late, so treatment has fewer mature users, and the mature-user contrast favors treatment. What is distorted?

101.A payment provider times out more often for treatment, and timed-out checkouts are excluded from revenue instead of counted as zero. Revenue per included checkout rises. What is a more honest revenue read?

102.Control users who never return still have an assignment row. Treatment logging writes a row only after the first return, so non-returners vanish. Retention is computed on rows that exist. Which way does that push treatment retention?

103.The team will ship only if a new checkout beats the old one, and they report a one-sided p-value aimed at improvement. The point estimate is slightly negative. A reviewer asks how harm would show up. What is the gap?

104.Refunds, a guardrail, are tested one-sided for an increase, matching a rule to block if refunds get worse. Primary conversion is tested two-sided. Why can those two choices both be right?

105.Before the test, the team says they will ship on a positive conversion lift and will not ship on a null. They compute a two-sided p-value, see 0.04, and ship without checking the sign. The estimate is negative. What did they ignore?

106.An engineer halves the p-value from a two-sided test and calls it one-sided so the result clears 0.05. The direction was chosen after seeing which way the estimate went. What is wrong with that halving?

107.Leadership will kill the feature if conversion drops, and will also investigate if it rises, because a surprise rise has meant a logging bug before. Which testing plan matches both actions?

108.The written rule is to ship only if a 95 percent interval lies entirely above zero. The interval runs from -0.2 percent to 1.4 percent. A teammate halves the two-sided p-value to 0.04 and wants to ship. What does the written rule require?