Evaluation and data

1.A return classifier looks sharp on the week it was fit and ordinary on the following Monday's mail. Which split is the first one to distrust?

2.Training rows are a random cut of March tickets. The model will score April. March and April differ because a new refund rule started on the 28th. What is the mismatch?

3.Someone proposes training through Friday and testing on a random 20 percent of that same Friday. Why does that overstate Monday?

4.You keep the last week of a quarter as the only test. That week is the annual sale. What can a strong score hide?

5.Labels for a churn model are 'cancelled within 30 days,' computed as of a snapshot. You randomly split customers, and many appear on both sides with neighboring snapshots. What leaked across time?

6.A feature is the count of tickets the customer filed this calendar week. The row is scored on Wednesday using a count that includes Thursday and Friday. What is broken for a live Wednesday?

7.The training window ends in June. You report July accuracy after retraining on July to 'make the comparison fair.' What did the July number stop meaning?

8.A rolling origin evaluates January, then February, then March, refitting only on the past each time. How is that closer to shipping than one shuffle?

9.Two models are tied on a random split. On the next unseen week, one drops hard and the other holds. Which comparison should decide the launch?

10.You notice the later week contains a new product line that training never saw. Accuracy falls only on that line. How do you report the fall?

11.A data pull for 'last month' is rebuilt every night and can gain late-arriving events. You trained on Monday's pull and tested on Friday's pull of the same month. What shifted under you?

12.Agents start using a macro on Tuesday that writes a stock phrase into tickets. A model trained before Tuesday looks worse on Wednesday's phrase-heavy mail. What changed in the input, not the truth?

13.You score a model weekly and the number drifts down for four weeks, then you discover the positive class got rarer. What comparison keeps the drift from being misread as a broken model?

14.The best model on last quarter's holdout is frozen and served all year. The business asks for a promise that the holdout number will still be true in December. What promise can you not make?

15.Fraud is about one row in a hundred. A model that calls every row legitimate is right about 99 percent of the time. What does that accuracy refuse to say?

16.Precision on the fraud flag is high and recall is low. In product language, what is the queue doing?

17.You lower the score threshold so the fraud queue gets more cases. What usually happens to precision and recall on that class?

18.A missed fraudulent transfer costs much more than a false review this quarter. Which way do you move the operating point, and what do you watch?

19.A screening tool for damaged parcels must catch nearly every damaged one. A false pull is cheap. Which error is the one you spend budget to avoid?

20.The positive class is rare, so a teammate up-samples it until the training file is half positive. The live stream is still one percent positive. What number from training is now easy to misread?

21.A confusion matrix on a rare defect shows 4 true positives, 1 false positive, 80 false negatives, and thousands of true negatives. Which cell should lead the readout?

22.Two models have the same accuracy on an imbalanced file. One catches half the rare events and the other catches almost none. What do you ask to see next?

23.A leaderboard ranks models by accuracy. Your defect rate is under two percent. How do you keep that board from picking a silent model?

24.Reviewers can only handle 30 fraud flags a day. The threshold that maximizes F1 produces 200. What is wrong with shipping the F1 point?

25.The rare class was duplicated until the loss looked balanced. Evaluation still uses the duplicated file. What does that do to the reported error rate?

26.A model is praised because specificity is 0.99. The rare disease flag in a screening set is almost never raised. How do you translate specificity here?

27.You need a single plot for an imbalanced ranker that does not depend on picking one threshold. What does that plot let you compare?

28.Management wants the fraud model 'at least 95 percent accurate' written into the contract. The base rate is 99 percent legitimate. What do you push back with?

29.A column called 'agent_already_refunded' is used to predict whether the agent should refund. It is filled before scoring in the eval extract. What is the feature?

30.Target encoding for a merchant category is fit on the full table, then the table is split into train and test. Scores look unusually strong on rare merchants. What happened?

31.The text of the final resolution is included to predict whether the customer will reopen the ticket. Resolutions are written when the case closes. Why is that suspect?

32.You remove a leaked feature and the offline score drops sharply. Leadership calls the drop a regression. What does the drop mean?

33.A free-text field contains the phrase 'refund approved by supervisor' on rows labeled approve. The model relies on that phrase. At decision time the phrase has not been typed. What will production do?

34.The same customer appears in train and test through a random row split. The model memorizes that customer's habit. What do you call the failure, and what is the repair?

35.Images for a defect model were augmented by flipping. The flip of a test image landed in training. Why can the score rise without a better detector?

36.The eval set was deduplicated after the split. Training still contains copies of many test tickets that differed by a signature line. What did dedupe-too-late miss?

37.A time-to-close feature is used to predict whether a ticket will be escalated. Closing time is unknown until the ticket ends, and escalation often happens along the way. What is wrong with the clock?

38.Preprocessing scales numeric columns using the mean of the entire file, then you split. Why is that a quieter leak than a copied label, and why does it still count?

39.A knowledge-base article is edited so that it answers the eval questions more directly. The retrieval score jumps the next day. The model weights did not change. What did?

40.The id column is a hash that includes the label code as a prefix. A tree model splits on that prefix early. How did the label enter the features?

41.You join a 'current customer value' table that was rebuilt today onto historical training rows from last year. What time does that value belong to?

42.Human raters see the model's current guess grayed in beside the ticket, then confirm or edit it. Agreement with the model rises. What bias did the screen invite?

43.A 'gold' set was built by keeping only the rows where the model and a rater already agreed. The model later scores very well on that set. What was selected?

44.The prompt used at eval time includes three solved tickets that are also scored in the average. Pull them out and the average falls. What was the fall?

45.A vendor demo reports a score on a public set that was also used to pick the checkpoint they shipped you. Why might you not be able to reproduce that score on your own tickets?

46.Synthetic tickets are generated from the gold answers, then mixed into the test file until they are most of it. The score rises. Who got easier, the model or the test?

47.Overall refund accuracy looks fine. Night-shift tickets, which are a small share of volume, are far worse. What did the overall figure hide?

48.You can only afford a few slice reads before a launch review. Which set is the minimum responsible cut for a support model?

49.One language has forty test tickets and a perfect score. How do you say that number out loud?

50.A slice with few examples swings from wonderful to poor depending on a handful of tickets. How do you report it?

51.The headline metric rises, and the highest-priority queue falls. Is the change a win?

52.Mobile chat and desktop email share one accuracy number. Mobile messages are short and full of photos the model never sees. What slice question is unanswered?

53.Errors concentrate on tickets that mention a newly launched accessory. The accessory is two percent of rows. Why can a launch bar on the global mean still pass?

54.You discover the model is worse for customers on a legacy plan that legal still has to honor. The plan is old and small. What is the wrong reason to skip the slice?

55.A slice report averages slices that have ten tickets with slices that have ten thousand. The small slices look equally precise on the page. What formatting choice caused that?

56.The weekday English slice is healthy. The weekend slice, staffed by a different vendor's agents, is not. A fix is proposed to the model only. What else could be the slice shift?

57.You add a slice for 'mentions a competitor' after you saw that those rows looked bad. The memo then says you pre-specified a failure. What is overstated?

58.A fairness read shows a gap on a region you do not serve differently on purpose. Volume there is moderate and the gap is stable across weeks. What do you owe the review?

59.The model is tuned to lift the worst slice. The worst slice rises and two healthy slices fall by more than the lift. How do you present the trade?

60.You only compute slices on the test set after the model is frozen. Training slices looked fine and were never saved. The test shows a hole. What earlier habit would have caught it sooner?

61.Three teams labeled 'urgent' with three different cheat sheets. Scores disagree by team more than by ticket text. What do you inspect before you fit anything?

62.Two raters double-label a gold set and often disagree. The model cannot beat both of them at once. What does that disagreement cap?

63.You want a single number for how often those two raters agree beyond chance. What are you reaching for?

64.Rater A marks a ticket urgent whenever a customer uses an exclamation point. Rater B follows the written safety rule. They disagree on tone-heavy mail. Which label do you train on?

65.A senior reviewer relabels only the rows the model got wrong, and leaves the model's successes untouched. What does the new gold set become?

66.Crowd raters are paid per ticket and finish in a few seconds on paragraphs that take a careful reader a minute. Agreement with your experts is poor. What do you question first?

67.Two experts disagree, and a third person who wrote the policy breaks the tie. How should that third label be described?

68.Raters are shown the customer's tenure and spend before they mark a complaint as valid. Those fields are not in the written definition. What can the labels start to copy?

69.A guideline update changes 'urgent' on Tuesday. Labels from Monday are mixed with labels from Wednesday in one training file. What did you mix?

70.You measure agreement once, on an easy batch of obvious refunds, and then assume later batches of edge cases match it. Why is that assumption thin?

71.One rater's labels are used as gold, and that rater later leaves. A new rater disagrees with the archived gold on a fresh double-label. What has to be redone before you quote a trend?

72.Raters may mark 'unsure' and almost never do, even on tickets they later say were ambiguous in review. What does a missing unsure option, in practice, force?

73.You adjudicate disagreements by keeping the label that matches the current model. The model then 'improves' on the adjudicated set. What circularity did you build?

74.A metric dashboard rewards the team when 'percent auto-resolved' rises. Agents start closing tickets the assistant touched, then reopening them off-dashboard. What is being gamed?

75.Suggested-reply acceptance is the team goal. Agents learn to accept and then paste their own text in a later field the metric does not see. What did the goal teach?

76.Precision is gamed by flagging only a handful of obvious fraud cases. The number looks excellent. The missed fraud is not in the formula you publish. What companion number stops that game?

77.The published metric ignores abstentions. The model abstains on every hard ticket and is scored only on the easy ones it answers. How does the average move?

78.A vendor chooses the threshold after seeing this month's labels, then quotes this month as the result. Next month the threshold is not refit. Why is the quote optimistic?

79.Handle time falls after the assistant ships. You then learn agents are closing tickets at the first reply to stop the clock. What metric behavior is that?

80.The loss used in training counts every class equally. The dashboard goal is catch-rate on a rare defect. Training loss goes down and the catch-rate does not. Why can that happen?

81.Leaders get a bonus when the quarterly model score rises. The easiest rise is to drop a hard segment from the scored population. What guard keeps the population honest?

82.An exact-match score is the release gate. Writers learn to copy the gold phrasing into templates the model memorizes. Customer satisfaction does not move. What was optimized?

83.You try twenty feature sets and publish the best test score as the result of the project. Why is that published score optimistic?

84.A holdout was used to pick the threshold, then the same holdout was reported as the final quality. What is biased, the model weights or the quoted operating point?

85.What is the clean role of a final test set in a model review?

86.You peek at the test score after every prompt edit and stop when it clears a bar. How many looks did the test survive?

87.Five engineers each try a fix and you ship the fix with the best test number. No validation file exists. What should you tell the review about that number?

88.Early stopping watches the test loss and restores the epoch with the best test loss. You then quote that test loss as generalization. What was the test used as?

89.A hyperparameter grid is scored on the test set. The winning width is shipped and the winning score is put in the abstract. Which sentence is fair?

90.You reuse last year's test as this year's test because the tickets were expensive to label. The model family has been tuned against that same file for a year. What is left of the file?

91.The test file is small, so you bootstrap it and quote the interval as if the model were fixed. The model was chosen as the best of many on that same small file. What does the interval miss?

92.A teammate says tuning on the test is acceptable if you promise not to look at individual rows. You still plot the score and edit. Does the promise hold?

93.Offline, the new ranker lifts nDCG. Online, add-to-cart does not move. The first result is more relevant and less often in stock. What can explain the dead online result?

94.A reply model wins an offline preference test. In the product, agents accept it less often because it is longer than the call allows. Where did the preference test diverge?

95.Offline recall of the right help article goes up. Online, agents still open the old bookmark. Logs show they do not scroll the new list. What failed to transfer?

96.A classifier's F1 rises after you add a feature the live scorer will not have until the nightly job. The daytime product cannot see it. What did the offline lift assume?

97.Human raters score answers as better when they sound fluent. Customers write back to say the fluent answer cited the wrong plan. Which score moved, and which did not?

98.Offline, you optimize classification accuracy of 'will escalate.' Online, the assistants who use the score escalate more, because the score makes them nervous. Accuracy of the prediction can stay high. What changed?

99.A search model's offline judges are employees who know the catalog codes. Shoppers do not. The employees' relevance gains do not show up in shopper success. Why might the judges be the wrong population?

100.You ship an offline win and watch the online metric for one afternoon. It is flat. A teammate wants to revert that night. What is the limit of the afternoon?

101.The offline set counts a suggestion correct when the gold article id matches. Agents call a suggestion useful when it saves them a search, even if they open a different article. Why can usefulness rise while id-match falls?

102.Offline lift is concentrated on long investigative tickets. The product volume is short status checks, where the two models tie. The traffic-weighted online metric is flat. How do you explain the flatline without calling the offline work useless?

103.You log only the offline metric in the release notes. The online guardrail on complaints was never wired up. The complaints rise after launch. What reporting choice left you late?

104.A new model wins offline on English email and is launched to every channel because the launch flag is global. The chat channel gets worse. What scoping mistake was that?

105.Shadow scoring shows the new model would have caught more fraud. Analysts never see those flags, and the fraud loss does not change. What is the most the shadow result supports?

106.The offline judge is another model from the same family. It prefers the new assistant's style. Customers prefer the old one in a blind bake-off. Which preference do you ship against?

107.You require a p-value below 0.05 on the offline set before launch, and you do not look at how large the lift is. A tiny lift on a huge file passes. Why is that a weak gate?

108.Both models are within noise of each other online. Offline, one wins by a small, stable margin on a proxy. Capacity is tight. What is a sound call?

109.Labels for the offline set were taken from the old model's auto-closes. The new model is graded on how often it matches those closes. What can a high score mean?

110.A weekly scorecard shows the offline number in green and hides the week-over-week online flatline in an appendix. What decision does that layout push?

111.You fix a slice offline and the slice metric rises. The only online test mixes all slices into one conversion number that does not move. How do you learn whether the slice fix landed?

112.The lab set is cleaned of duplicates, empty tickets, and spam. Production is not. The lab lift shrinks to nothing in production. What did the cleaning change?

113.Reviewers want one number to ship on. You have a lab lift, a flat online conversion, and a rising complaint rate in one region. Which number is the one you can ship on?

114.Two annotators disagree on whether a borderline policy exception applies. You drop every disagreed row from the test and the model score rises. What did the drop do?

115.A later week arrives with a new macro in the text and a new mix of issues. Train-week accuracy was strong. You retrain on the later week before reporting it. What question did you erase?

116.Class imbalance, a leaked closing note, and a test set you tuned on can all produce a confident wrong ship number. What do they share?