Model monitoring

1.Cobalt Pay's card-risk model shows a jump in PSI on ticket size the morning after a retail holiday. Chargeback labels arrive 21 days later. What can you say today about accuracy?

2.Three weeks later, precision on that holiday week matches the prior month. Ticket size had drifted. What is the honest reading?

3.Ticket size stays inside its usual band for a month. When the 21-day chargebacks mature, the rate has doubled. What changed?

4.PSI pages every hour. Chargeback labels are reviewed weekly. After a month, on-call mutes the channel. What did the hourly page train people to do?

5.The nightly train is green. It fit yesterday's closed snapshot. Serving is throwing because today's payload includes a new value, wallet_v2. What did the green train not see?

6.Serving returns errors on a null currency. Training dropped null currency, so that job is green. Where else does the contract have to be checked?

7.The error alert requires the rate to stay above 1 percent for 5 minutes. A deploy hits 4 percent for 30 seconds and recovers. The page does not fire. What did the window do?

8.The same alert's window is 30 minutes. Serving is fully down for 10 minutes and no page fires. What is wrong with the window?

9.A blog says to page when PSI exceeds 0.2. Cobalt's merchant-id hash has a natural daily PSI above 0.2. What happens each morning?

10.A noisy optional field goes from 0.1 percent null to 0.2 percent null. The alert fires on any increase. What bound can you live with?

11.A shadow scorer logs the same requests and does not change the approve-or-decline decision. Shadow errors on a new field. No cardholder is affected. What is that setup for?

12.Shadow uses the same feature store as production, and its new code writes a key the live model reads. What is not true?

13.Shadow is called inside the request, and the cardholder waits on it. The shadow model is not what you serve. What has to change?

14.You want to compare shadow and live accuracy this afternoon. Chargeback labels are not in yet. What can you compare today?

15.Shadow and live agree on 99 percent of requests. The other 1 percent is the high-value slice. What does the overall rate hide?

16.You point serving at model v3. The job that writes basket_size with the new formula stays on v4. v3's scores are nonsense. What did the rollback omit?

17.After that rollback the status page says model v3. Logs show a feature hash from v4. What is the page failing to show?

18.One request arrives with a misspelled category that has never been seen. Nothing else looks wrong. What do you do with it?

19.The mean score dips every Sunday, and it has for months. An absolute threshold pages every Sunday night. What baseline avoids that?

20.Chargeback labels are specified to be absent for 10 days. An alert fires because none arrived today. What should that alert wait for?

21.The train is green. Online feature lookup p99 is 800 milliseconds and timeouts are climbing. The train never calls that store. What is the green job not measuring?

22.The GPU train succeeds. The CPU server runs out of memory because the new artifact is larger, and the process crash-loops. The registry accepted the file. What live signal sees the death?

23.Inputs sit inside their bands. After a deploy, every score is 0.5. Labels will not arrive for weeks. What monitor can see this now?

24.Every score is 0.99 after a deploy that skipped a sigmoid. Inputs look normal. Which alert fires before any label exists?

25.A calibration chart needs chargeback labels. For the two weeks before they arrive, what can stand in, and what is it not?

26.The precision alert is a drop of 5 points. This hour has 12 mature labels. What is wrong with firing?

27.Waiting for 10,000 labels takes a month, and the harm you care about is daily. What can page in the meantime?

28.Drift on a feature with a near-zero weight pages the same way as drift on the top feature. An unused logged column drifts too. Who should be paged for that unused column?

29.The train environment loads the model. The server uses a different library build and fails a checksum. The train job is green. What failed?

30.A canary takes 5 percent of traffic. The global error rate barely moves and stays under its threshold. The canary's own error rate is 20 percent. Where is the page?

31.The canary has 30 requests and one error, about 3 percent. What do you not do?

32.Shadow shows the candidate timing out on large carts. Live cardholders are unaffected. What do you do with the promotion?

33.You roll back the model container and not the client flag. The flag still selects the new preprocessor. What is still running the new code?

34.Clients cache the feature schema for 24 hours. The server rollback finishes in a minute. Some apps still send the new layout. What do you watch?

35.The label job is late. The accuracy chart draws the gap as zero, and an alert on accuracy equal to zero pages the model owner. What should a missing point be?

36.The drift reference is the training distribution from 18 months ago. Every recent month looks drifted. What question is that reference actually asking?

37.Inputs are compared with the previous hour. Every morning looks like drift because volume and mix follow the day. What comparison matches that cycle?

38.This afternoon's recall uses only chargebacks that have already arrived, which are the fast cases. Someone wants to page on that recall. What is wrong with it?

39.A live holdout is 1 percent of traffic and is the only fresh accuracy you have. It can show a large break. What can it miss?

40.The model process is crash-looping. The gateway fills a constant score and returns HTTP 200. The nightly train is green. What do you alert on?

41.Fallback normally covers 0.01 percent of requests from timeouts. It is at 15 percent. Why is a short window appropriate?

42.A feature-store failover drill is on the calendar. Fallback is the planned behavior during the window. What do you put on the alert?

43.Someone mutes the drift alert for a week. The mute has no expiry. A month later a real break is silent. What does a mute need?

44.Shadow is sampled only from employee cards. Their baskets are not customer baskets. A clean run does not clear the model. What sample do you need?

45.The shadow logger copies the raw payload, including a payment token. What may that log keep?

46.The online key is unversioned. You roll the model back and start a recompute. Halfway through, some keys are old and some are new. What can one request read?

47.A recompute would take too long. Yesterday's feature values are still in a second column, and a flag points reads at it. What had to be built before the incident?

48.Cardholders report timeouts. The median latency alert is quiet. The 99th percentile moved. What did the median hide?

49.The 99th percentile pages on every deploy because one new instance spends two seconds on a cold start. What keeps the page for a tail that stays bad?

50.After a client release, the share of requests with country unknown jumps from 1 percent to 40 percent. What kind of response is that?

51.The same unknown-country rate moves from 1 percent to 1.4 percent, inside the noise you have seen before. What is the response?

52.The train is green and the deploy job failed. A dashboard reads the registry and shows the new model id. Serving is still answering with the old id. Which id belongs on the page you trust?

53.One server did not reload. Half of traffic is still the old model, and the average looks acceptable. What split shows the stuck server?

54.One segment's mean score drops. That segment's orders got smaller, and order size is a feature. What do you check before calling the model broken?

55.You log the score and not the features. After a bad hour you cannot explain why a score was high. What was missing?

56.A point labeled yesterday uses yesterday's decisions plus whichever labels happen to be in today. Many are still open. What should the point carry?

57.Accuracy on mature labels is steady. The alert is wired to the immature number and it pages. What should the alert use?

58.PSI on a unique account id fires all day. What should you stop computing PSI on?

59.Error rate and success rate describe the same requests, and both page. Two people acknowledge one fire. What do you keep as the page?

60.Those two pages link to runbooks that disagree about the first step. What does on-call need?

61.Shadow beats live on last month's labels and loses on this week's score-shift proxy. The proxy was the one you said matches current traffic. What do you not do yet?

62.Rollback moves a tag named prod. That tag was already updated to the bad image. What should the rollback point at, and what should the page display?

63.The health check counts HTTP 200 as success. The body is a fallback score and the status is 200. What is the check missing?

64.The streaming job that writes a feature is stuck on an old checkpoint. The feature age is 9 hours, and scores still return. The train is green. What should page before accuracy could move?

65.The freshness limit is 5 minutes on a feature that is rebuilt once a day. The alert pages all afternoon. What is wrong with the limit?

66.That daily job finishes two hours late. The values are correct, and the freshness page fires. Is the page a mistake?

67.An input-drift alert fires because a feature is repeating yesterday's value. The freshness alert is also red. What does the runbook open first?

68.Drift on ticket size is real, and the matching chargebacks will not mature for 21 days. What do you write as today's impact?

69.The wiki says to page at PSI 0.25. The alert configuration says 0.1, and it is paging. Which number is the control?

70.A newly logged column has no reference distribution. The drift job throws, and every other monitor stops updating. What should the new column's failure do?

71.Those monitors stay empty all weekend. The status board looks green because nothing fired. What is absent?

72.A checker on an internal doc link fails at 03:00 and pages the model owner. Serve errors, freshness, and scores are inside their bounds. What is that page?

73.A new category was registered in the contract change before the deploy. The unseen-value alert pages on it anyway. What should the alert already know?

74.A value shows up that was not on the list registered with the change. What is the page?

75.Shadow scores 100 percent of traffic against the shared feature store. Live p99 collapses. What budget did the mirror ignore?

76.You cut shadow to 1 percent, and the bug sits in a rare country. The sample misses it for days. What do you either change or write down?

77.You revert the feature repository. The running image was built from a different commit and is not rebuilt. The feature hash in the logs does not change. What did the revert fail to do?

78.The feature hash that served a bad decision is not in the logs. What do you add, and what is a rollback until you add it?

79.The train is green on a 10 percent sample someone set to save cost. Serving scores everyone. Offline metrics and the live score monitor disagree. What does the training page need to show?

80.Forty drift pages fire in a night. A real outage sits for two hours before anyone accepts it. What makes the next page something people will trust?

81.You are writing the template for a page someone can act on at 03:00. What does it name?

82.You roll the model id back and the error rate stays high. The feature version in the logs does not change. What is the next move?

83.You roll the feature version back. The error rate recovers. The model id never changed. What does the review name as the fault?

84.One region starts sending amounts in cents. Global PSI barely moves. That region's drift is severe. How should the drift monitor be sliced?

85.That region has little overnight traffic, and its PSI swings wildly after midnight. What keeps the night quiet without losing the region?

86.Enterprise chargebacks arrive in 2 days. Consumer chargebacks arrive in 20. A global alert expects labels the next day. What does it get wrong?

87.Consumer labels are on day 12 of a stated 20-day wait. The model owner is paged because the labels feel late. What is that page?

88.The training page shows a high AUC computed with a column serving does not have. Live decisions are worse. Which number describes today's traffic?

89.The live outcome monitor is down. In the incident channel, someone posts the training AUC as the current status. What is wrong with that status?

90.A planned price change shifts a feature the model uses, so scores ought to move. What keeps the expected move from paging, and what would be the real surprise?

91.A fixed threshold on the mean score pages on every campaign week. What do you write down so the page means something?

92.This week's shadow traffic is a campaign. Last month's shadow report was an ordinary week. What do you not treat as one test?

93.Shadow and live differ by less than the margin you would act on, and the sample is small. Where does that difference go?

94.Shadow disagrees with live. One feature is null only in shadow, because shadow reads a new store that is still empty. What do you judge?

95.You restore yesterday's model and yesterday's feature code. The request logger still parses the new schema and drops the lines. What just went blind?

96.Log volume's heartbeat drops after that rollback. The error rate has no requests to count. What is the page?

97.Every green train is shipped automatically. Serving errors start at 03:10, ten minutes after the ship. The train tile is green. What lines up with the errors?

98.Auto-ship is turned off. The next green train stays in the registry until a canary passes. What keeps serving on the last good id during that wait?

99.The canary's error rate is inside its limit. A score-shift guardrail you added is not. Do you promote?

100.That guardrail rejects every canary because the allowed shift is tiny. You are about to remove it. What do you do instead?

101.Monday's drift alert is a holiday weekend, and it is real. A label-delay alert fires beside it, but the wait is the normal 10 days and the alert expects next-day labels. What do you fix?

102.A staffing model always drifts on a public holiday. The runbook says to review it Tuesday, not to wake anyone. What is a Monday 06:00 page?

103.A fraud model on that same holiday sees a burst of merchants it has never scored. The runbook says a new-merchant burst can be an attack. Why does this one page when the staffing model does not?

104.You roll the model id back. The feature job is a stream that cannot rewind. You have no stored snapshot of the old values and no rules fallback. What can you not claim?

105.The status board shows one green light for the ML platform because the train succeeded. Serving is failing. Which lights belong in its place?

106.Ticket size has drifted and there is no mature label. A teammate wants the model owner paged to retrain tonight. What page, if any, is earned?

107.Raw ticket size drifted. The model's scores did not move, and ticket size is one of its main features. What do you suspect?

108.The rollback restored the model digest and the feature job's code. The store's TTL has already expired the old values, and a recompute needs two hours. What do you serve until then?

109.A dashboard query times out and its tile goes red at 03:00. Serve errors, freshness, and score sanity are all inside their bounds. What is the work item?