Reliability

1.The concert site's SLO is homepage status 200. During the on-sale, the homepage stays green and every attempt to buy a seat returns an error. What is the SLO measuring?

2.By the 10th, a fare experiment has spent the month's error budget. Another pricing experiment is still on tomorrow's calendar. What is the budget for?

3.Checkout pods sit at 40 percent CPU. Purchases fail because every database connection is checked out and waiting. The CPU graph looks calm. What is saturated?

4.The card processor blips for ten seconds. Each checkout retries three times with no delay. The processor, just coming back, receives a triple load and falls over again. What should the retries have done?

5.The gateway stops waiting at two seconds. The worker's call to the ticket printer times out at thirty seconds. The buyer is gone, and the worker stays busy until the printer answers. What is holding the worker?

6.The ticket app runs in three zones of one region. A region-wide network cut takes all three down. The design note calls this multi-region. What does this layout survive?

7.Nightly database snapshots have reported success for a year. Nobody has restored one. A bad delete removes the day's ticket sales. What do you actually have?

8.A page fires because a host is at 90 percent CPU. Checkout is stuck on a lock and the app's own CPU is idle. The busy CPU is a log shipper on that host. What should have paged?

9.The only person allowed to set severity is also the only person in a shell changing production. Questions from support sit unanswered while that person types. What should be split?

10.The monthly error budget can absorb a forty-minute outage. Checkout is entirely down for forty minutes on Tuesday. No page fires, because on Friday the month still looks inside the line. What alert is missing?

11.The error-budget policy says the team will discuss it. The budget hits zero on Wednesday, and a launch ships because the discussion is scheduled for next week. What did the policy fail to change?

12.Checkout latency climbs. CPU stays low. Eight worker threads are busy, and two hundred requests wait in an unbounded queue in front of them. What is saturated?

13.After a five-second blip, every client retries at the same instant. The backoff is a fixed one second with no jitter. The dependency falls over a second time as the wave arrives. What would spread that wave?

14.The client stops waiting on a charge at two seconds and sends the charge again. The first charge succeeds at three seconds. The buyer pays twice. What did the timeout prove?

15.App pods run in three zones. The only database is in zone A. Zone A loses power. Pods in B and C stay up and cannot sell a ticket. How multi-zone is the sale?

16.You tell finance checkout can be back in thirty minutes. The only recovery you have is a restore, and last quarter's drill took six hours at the current data size. What did you promise?

17.A disk-usage page fires for a build agent. At the same time checkout fails because a certificate on the pay host expired. The on-call spends the first half hour on the agent's disk. Which page matches the buyers?

18.The communications lead posts that everything is clear. The person making the fix is still running a purchase and has not seen it succeed. What is that post ahead of?

19.The process is up. The load balancer marks the target healthy. Every purchase returns an error body. The uptime number stays at 100 percent. What is that number counting?

20.The latency SLO is an average. A pocket of checkout calls takes twenty seconds, and the average stays under 300 milliseconds because most calls are health checks. What does the average hide?

21.When the card processor fails, checkout returns HTTP 200 and the words please try again. The error budget counts only HTTP 500. The budget does not move while buyers fail. What is missing from the budget?

22.A memory leak ends in a restart every few hours. Each restart is quick, the availability SLO stays green, and buyers see a short break every cycle. What is the SLO failing to show?

23.A hot tax key expires at one moment. Every checkout recomputes tax against a slow service together, and the service tips over. What would keep one expiry from becoming a herd?

24.The buyer cancels at one second. The worker keeps the database connection until a thirty-second timeout. Eight cancels fill the pool, and new checkouts cannot start. What did the cancel fail to do?

25.Two regions both accept writes for the last seat. There is no conflict rule. Both commits succeed, and one ticket disappears on the next sync. Uptime in each region is 100 percent. What did uptime miss?

26.The ticket database has nightly snapshots. Receipt PDFs live in a bucket with no copy. You restore the database and every receipt link returns not found. What did the backup leave out?

27.The backup job exits zero and uploads a file. A staging restore is empty. The dump failed, and the script still compressed the empty stream because the pipe hid the dump's exit status. What did a green job prove?

28.CPU at 95 percent pages at 3 a.m. Checkout is inside its SLO. The CPU is a tax export that is supposed to use a core. The on-call wakes up. What made this a page?

29.The incident commander is the person typing the rollback. Status questions wait. A second person starts a different rollback at the same time. What is uncoordinated?

30.Nobody writes down changes during the incident. The next day's review blames the noon deploy. The harmful change was a flag flip at 2 p.m. that was never noted. What would have named the flag?

31.The SLO ignores any hour labeled maintenance. Deploys happen all afternoon, and someone labels each hour as it starts. The budget never moves on those deploys. What did the label become?

32.Checkout meets its latency SLO by rejecting extra buyers with a fast 503. There is no availability SLO on the dashboard. Leadership hears that latency is fine. What did the fast 503 spend?

33.Workers stay busy retrying one poison message that always throws. Fresh ticket events wait behind it. CPU is high and the queue of new work grows. What should happen to that message?

34.A dependency returns 503 and Retry-After of thirty seconds. The client retries at once, then again, and the dependency stays down under that load. What did the client ignore?

35.The client sends a second, hedged request when a call is still running at 200 milliseconds. On a bad day every call is that slow, so the dependency receives about double the traffic and gets slower. When do hedges become the harm?

36.The client timeout equals the healthy p99, about 300 milliseconds. The slowest healthy one percent always time out and retry, adding load that was not a processor failure. What did that timeout turn into errors?

37.Zones A and B are in different buildings. Both send card traffic through one NAT gateway. The NAT dies. Pods in both zones stay up and cannot reach the processor. What did the two zones share?

38.You fail over to the other region. Replication lag is four minutes. The last four minutes of ticket sales are missing, and the status says no data was lost. What is the lag?

39.The ticket snapshot is encrypted with a key that lives only inside the same database. The database is gone. The snapshot file is still in the bucket. What can you do with the file?

40.You notice a bad migration on day ten. Snapshots expire after seven days. The last copy from before the migration aged out yesterday. What was shorter than the time it took to notice?

41.A synthetic check of the homepage in one city returns 200. Checkout for buyers in another region is failing. The only alert is that synthetic, and it stays quiet. What did the check never do?

42.Support has no view of the incident. They tell callers the problem is the caller's bank, while checkout is down for everyone. What did support not have?

43.Severity is the count of alarms. Checkout is down for every buyer and there is one alarm, so it is filed as minor. A noisy CPU widget fires nine alarms and is called major. What should set severity?

44.The SLO target is 100 percent. Any single failure spends the budget, the budget is always gone, and people stop opening it. What can that target no longer do?

45.Checkout was down for half an hour this morning and the monthly budget is spent. A pricing experiment is still scheduled this afternoon because the date was announced. What is the spent budget for?

46.Checkout fails with too many open files. CPU is at 30 percent. The only graph the on-call has open is CPU, and it looks fine. What limit is used up?

47.The autoscaler adds pods that fail their health checks, then adds more, until the subnet runs out of addresses. A rollback pod cannot get an address either. What is the scaler spending?

48.A low-priority image resize retries without a limit while the thumbnail service is down. Those calls fill a pool shared with checkout. Purchases wait behind thumbnails. What should bound the resize?

49.A circuit opens during a thirty-second processor blip and never sends a trial call. An hour later checkout still fails fast, and the processor has been healthy for most of that hour. What is the circuit missing?

50.There is no circuit in front of the card processor. While it is erroring, callers keep full concurrency on it. Checkout workers stay busy failing, and the processor cannot recover. What would give it room?

51.The buyer has three seconds. Service A spends 2.9 seconds, then calls B without saying that a tenth of a second remains. B uses its five-second default. A has already answered the buyer with an error, and B holds a worker. What should A have passed?

52.Region failover updates DNS. The old record's TTL is thirty minutes. Clients keep the dead region's address, and the failover looks like it did nothing. What is part of the failover time?

53.DNS failover watches a static page on a box that cannot reach the database. The page stays up, so DNS never moves, and buyers keep arriving at a region that cannot sell. What is the failover check watching?

54.The restore script last ran a year ago. The schema has changed since. The script stops halfway, and the on-call has no second path. What is that script?

55.You restore last night's database and start the app. The queue was not restored to that same moment. The app replays today's messages and confirmation emails go out again. What was inconsistent?

56.The checkout page goes to a shared mailbox. Overnight nobody has that mailbox open. Buyers cannot pay for three hours, and a person first sees the mail in the morning. What did the page fail to reach?

57.The only person who knows the payment path is pulled into three incident rooms at once and finishes none of them in time. Checkout is one of the three. What should own that person until sales move?

58.The page links to a dashboard that loads only on the corporate VPN. The on-call laptop is not on the VPN. The first twenty minutes go to finding a login. What has to open from the place pages are answered?

59.The load balancer records checkout as 200 in 100 milliseconds. The phone gives up at 80 milliseconds because of a new client timeout. Buyers see a failure. The server SLO stays green. Where is the miss?

60.The budget is at zero and the rule is to stop risky changes. The next deploy is called a small fix and rewrites a hot table. Checkout locks for twenty minutes. What did the word small hide?

61.App CPU is low and reads look fine. Purchases that commit stall. The database log disk is busy enough that commits cannot finish. What is saturated?

62.One transaction holds a lock on the inventory row. The pool fills with waiters. Checkout latency is that wait, and CPU is idle. What limit are the waiters sitting on?

63.When a lock wait fails, the client retries at once. Each retry joins the same lock queue. The queue grows because of the retries, and the lock holder is still the original transaction. What are the retries adding?

64.A lock wait may run for thirty seconds. The buyer's checkout budget is three seconds. The buyer is gone, and the waiter keeps a pool slot for the rest of the thirty seconds. What should end the wait?

65.The inventory cluster has three nodes. Two of them are in zone A. Zone A fails, and the remaining node cannot form a quorum. Sales stop. Why was zone A enough to take the cluster down?

66.The only copy of tickets is a streaming replica. A bad delete hits the primary and shows up on the replica a second later. There is no earlier copy. What did the replica copy?

67.The restore script defaults to the production database host. A drill that was meant for staging restores over the live ticket data. What made production the target?

68.During the incident everyone runs diagnostics over SSH. CPU reaches 90 percent from that work. A new CPU page fires, and half the room investigates CPU. What is that CPU?

69.The status page stays blank for forty minutes. Buyers refresh checkout in a loop and add load to the failing path. What would reduce that extra load?

70.An executive joins and tells the fixer to chase a CDN theory. The evidence is a database lock. The fixer switches to the CDN because of who asked. What should decide the next step?

71.Disk usage is 20 percent. Uploads fail because the volume has no free inodes. The disk-space graph looks calm. What is full?

72.A 30-day month at 99.9 percent availability allows about 43 minutes of full outage. You have used 40 minutes, and a risky migration is planned for Friday. How much room is left?

73.During a total checkout outage, clients stop calling. The error ratio is computed on the few health checks that remain, and the month looks almost clean. What comparison catches this?

74.Checkout shows 100 percent of its CPU limit. The node has idle cores. Moving the pod to a bigger node does nothing until the limit changes. What is saturated?

75.The browser retries three times, the gateway retries each of those three times, and the service retries each of those three times. One click can become twenty-seven charge attempts. Where should the retry live?

76.One tenant is fully unable to buy and is one percent of traffic. The aggregate checkout SLO stays inside budget. That tenant completes no purchases. What does the aggregate hide?

77.You promise checkout at 99.9 percent. The card processor you must call is offered at 99.5 percent, and a sale cannot finish without it. There is no fallback. What is wrong with the promise?

78.The passive region is sized for about ten percent of traffic and has never taken a real rush. A region failure sends all traffic there. What will that region do under the rush?

79.Backups and request logs share a bucket. A lifecycle rule written to delete logs after one day deletes the backups too. What did the shared rule erase?

80.A build agent's disk-full event is a phone page. The checkout burn alert is an email. The on-call wakes for the agent and reads the checkout mail in the morning. Buyers were failing overnight. Which event deserves the phone?

81.The mitigation adds replicas. CPU falls. Checkout is still stuck on the same lock. The incident is marked mitigated because CPU recovered. What should decide that the mitigation worked?

82.A finance dashboard query uses the checkout database pool and has no timeout. One dashboard holds every connection. Purchases cannot start. The dashboard is an internal page. What did it take?

83.A worker's lease on a queue message is thirty seconds. Sending the ticket email takes two minutes. Another worker receives the same message and both send the email. What expired too soon?

84.The restore steps exist only as a script on the database host. That host is the one that failed. The on-call has nothing to read. Where should the procedure live?

85.The metrics pipeline fills up. The SLO dashboard freezes on the last green value. Checkout is failing, and the board stays green. What is the frozen board?

86.Checkout fails for about five percent of attempts, all week. Alerts that look for a sharp spike never fire. The monthly budget is gone by Thursday. What window is missing?

87.You restore orders from midnight and payments from 6 a.m. Seats look sold and unpaid. The two restores were not one moment. What does the sale need instead?

88.A mitigation serves cached prices and is left on for a week. Nobody owns turning it off. Buyers see stale prices, and the SLO counts the response as success. What does the mitigation lack?

89.At 4 a.m. checkout volume is supposed to be near zero. The error ratio divides by a tiny number and pages. The few calls that happened were fine. What should stay quiet?

90.The error budget resets on the first of the month. A risky deploy is scheduled that morning so it can use a fresh budget. Buyers had a long outage on the last day of the previous month. What did the calendar change?

91.The processor returns HTTP 200 and an empty body when it cannot issue a ticket. Your SLO counts the 200 as success. Buyers see a blank ticket. What should count as success?

92.Checkout calls twenty dependencies one after another. Each may take two seconds. A slow day can take forty seconds, and each dependency stays inside its own timeout. What budget did the hops fail to share?

93.When the tax service is down, checkout uses yesterday's tax and counts the sale as a successful SLO hit. Some buyers are charged the wrong tax. What did the SLO treat as good enough?

94.The web server's accept queue fills. New connections are refused. A health check inside the process still passes, and CPU is moderate. What is full?

95.A bot acknowledges every page and files a ticket. Nobody is woken after the ack. Checkout is down at 2 a.m., and the ticket waits for office hours. What did the ack replace?

96.The communications lead pastes a command from a screenshot into production and deletes a namespace. They were trying to make the status update more precise. Who should have run commands?

97.The quarter is spent tightening a read-path SLO. Paying for a ticket is not measured. Payment failures stay quiet while reads get faster. Which action is missing an SLO?

98.You drain a zone and the database fails over. Replacement pods cannot schedule because the remaining zone is already at its pod limit. The drill never looked at scheduling room. What did zone loss include?

99.The app's database user can delete ticket rows and can also delete backups. A bug that wipes tickets can wipe the copies in the same minute. What should the app user be unable to do?

100.Policy rotates the incident commander every fifteen minutes so the load is shared. Each handoff restarts the diagnosis. Nobody holds the plan. What should a handoff carry?

101.A game day fails the database over cleanly. The on-call has never opened the paging tool and cannot find the bridge. The script worked. What failed?

102.Eight workers each wait up to sixty seconds on a hung partner. All eight are busy. New checkouts cannot start. The partner is one dependency, and the rest of the sale would be quick. What occupies the pool?

103.The same eight workers serve checkout and the health check. They are all stuck for sixty seconds on a hung partner. The health check cannot get a worker, times out, and the orchestrator restarts the pod, dropping the in-flight sales. What did the restart add?

104.The failover runbook's first step is a DNS change only the vendor can make. Vendor support is closed tonight. Your stated recovery time assumes that call is answered. What recovery time do you have tonight?

105.Checksums are stored next to the backup and never checked until the day you restore. The disk has returned bad bytes for a month. You discover that during the outage. When should the checksum have been read?

106.Paging is suppressed for the open checkout incident so repeats stay quiet. A separate failure starts: every refund fails. The refund page is suppressed too. What should suppression still allow?

107.The fixer rolls one zone back. The commander tells buyers the region is fixed. The other zone is still on the bad release, and half of purchases still fail. What was verified too early?

108.One person is commander, the only one typing, and the only one writing buyer updates during a total checkout outage. The timeline is missing, and a second change starts with no coordination. What has to be split once the incident is real?