Study design
An applied AI scientist interview on what a project locked before the run, and what the result cannot claim.
These prompts are original practice. They are not questions from a named employer or a live hiring loop.
1.On the demand-forecast walkthrough, the 14-day horizon and the Sunday cutoff were written down before the April test weeks were scored. What does that lock support?
2.The support-ticket tagger hashed the March 3 queue taxonomy before training. A listener asks what that hash locked. What is the honest statement?
3.For speech endpointing, a 400-millisecond hangover was chosen on development audio and frozen before any test call was scored. What was locked?
4.Parts recommendation froze the candidate list as the parts stocked on Monday, before any hit rate was computed. What does that freeze fix?
5.Sensor-failure labels used a 72-hour window written in the protocol before the first fit. The window was not edited after the scores. What is locked?
6.The delivery-time model scored lateness against the promise printed at checkout. That promise rule was fixed before the holdout week. What was locked?
7.The fraud-score project set the review threshold from last quarter's cost ratio before scoring the holdout month. What is the lock?
8.The note tagger froze tag manual version 4 before any note was labeled for the test week. What does that version lock?
9.The demand project said the split was by week, but nobody recorded which weeks were held out before the first score. What was locked?
10.Ticket labels were the queue the agent picked at close, and that rule was stated before training. Some tickets were later relabeled to match the model. What remains locked?
11.Speech endpointing held out entire speakers, and the test speaker list was written before any endpoint score. What is the lock?
12.Parts recommendation scored the top three suggestions. Three was written in the decision note before the holdout jobs were scored. What was locked?
13.The sensor plan said features stop at the last reading at least six hours before the label time, and that line was written before the fit. What is locked?
14.Delivery-time training used routes finished before the holdout Monday, and that Monday was named before scoring. What was locked?
15.Fraud labels were a chargeback filed within 45 days, and 45 was in the protocol before the holdout month was scored. What is the lock?
16.The note tagger split by clinician, and the test clinicians were listed before any tag score. What was locked?
17.Demand features were frozen as the set in commit 18f, recorded before the test month was scored. What does the commit lock?
18.The ticket tagger saved a ticket-id list that cut train from test, and the list was saved before training. It was not rebuilt after the scores. What is locked?
19.Parts recommendation compared last spring's model on last spring's jobs with a new model on this spring's jobs. Hit rate rose. What comparison is real?
20.The sensor project compared a new model on cleaned logs with the old model on raw logs. The failure score rose. What was actually compared?
21.Delivery time compared a new model with a new depot map against the old model with the old map. Mean error fell. What can you attribute?
22.The fraud project compared a new score on this month's payments with the old score on last month's payments. The top bucket caught more fraud. What is the real comparison?
23.The note tagger compared a new model on notes labeled with manual version 5 against the old model on version 4. Agreement rose. What moved?
24.Demand forecasting compared a new model that added a promo flag with the old model that had none. The test month also used a different promo calendar. Error fell. What is real?
25.The ticket tagger compared a new model on English-only tickets with the old model on the mixed-language queue. Accuracy rose. What comparison was run?
26.Speech endpointing compared a new detector on headset audio with the old detector on speakerphone audio. The endpoint score rose. What is the real contrast?
27.Parts recommendation compared a new model trained after dropping incomplete jobs with the old model trained on all jobs. Hit rate rose. What changed at once?
28.The sensor comparison used a new model and also moved the failure label from 24 hours to 72 hours. The score rose. What can you say?
29.Delivery time compared a new model on parcels under ten pounds with the old model on all parcels. Error fell. What is the real comparison?
30.Fraud scoring compared a new model that excluded gift-card payments with the old model on all payments. Precision in the reviewed bucket rose. What moved?
31.The note tagger compared a new model on typed notes with the old model on a mix of typed and dictated notes. Agreement rose. What is real?
32.The demand team compared a new model fit only on stores open all year with the old model fit on every store, including remodel weeks. Error fell. What changed together?
33.Ticket tagging compared a new model trained only on the first reply with the old model trained on the whole thread. Accuracy rose. What is the confound?
34.Speech endpointing compared a new detector with a 200-millisecond hangover against the old detector with a 500-millisecond hangover. The score rose. What moved?
35.Parts recommendation compared a new model scored only on jobs with a photo against the old model scored on all jobs. Hit rate rose. What comparison is real?
36.Sensor failure compared a new model that reads the maintenance log with the old model that reads only the vibration feed. The score rose. What changed at once?
37.Fraud labels for training included the analyst's note, and that note was written after the analyst saw the model's score. What did that order do?
38.Some test notes were tagged by copying the model's suggested tag, and those copies were used as the answer key. What did the copy do?
39.The demand model used this week's actual units as a feature while predicting this week's units. What did that feature do?
40.Ticket text for the tagger included the queue name in the subject line, and the label was that same queue. What did the subject do?
41.Endpoint labels were marked by a listener who could see the detector's mark on the waveform. What did that view do?
42.The parts label was the part the technician installed, and the install screen had already shown the model's top suggestion. What did that screen do?
43.Sensor-failure labels came from work orders opened by the same alarm the model uses as a feature. What did that shared alarm do?
44.Delivery-time training copied the door-scan time into a feature called latest event, and the target was that same scan. What did the copy do?
45.A batch of fraud labels was filled by a rule that flags any payment over a fixed amount, and the model was scored against that rule. What did the rule do?
46.Note tags for the test day were the model's draft, which a clerk accepted without reading the note. What did that acceptance do?
47.Demand rows included a units-sold column that was revised after the week, and the target used that same revised number. What did the revision do?
48.The ticket tagger's test set included tickets copied into the training export under a second id. What did the second id do?
49.Endpoint test clips were cut on the detector's own endpoint, then scored for how often the clip matched that endpoint. What did the cut do?
50.When the technician left the part field blank, the parts label was filled from the recommendation log. Those rows stayed in the test hit rate. What did the fill do?
51.Only sensors that had already alarmed were sent to inspection, and inspection is what created the failure label. What did that selection do?
52.On the delivery-time model, late arrivals were more often given a hand-checked time, and that check used the model's estimate as the default. What did the default do?
53.Fraud training used confirmed labels, and confirmed meant the reviewer clicked a button that copies the model's reason code. What did that click do?
54.Notes from the pilot clinic were labeled by the person who wrote the model's rules, and the key was those rules. What did that grading do?
55.The ticket tagger dropped tickets longer than 4,000 characters. The plan, written before scoring, said those are pasted logs and the queue decision excludes them. What is the right reading?
56.Speech endpointing kept calls with long silences. The protocol, written before the score, said the product must endpoint those calls. Why did they stay?
57.Parts recommendation dropped jobs where the installed part cost more than the truck's approval limit. The limit was in the decision note before hit rate. Why is that drop in bounds?
58.The sensor project kept units with extreme vibration. The plan, written before the score, said those are the units maintenance already watches. Why keep them?
59.Delivery time dropped routes canceled before the truck left. The protocol, written first, said a canceled route has no arrival to score. What is that drop?
60.The fraud project kept tiny payments. A ledger note, dated before the holdout score, said small payments stay in the review decision because they match a known mule pattern. Why keep them?
61.The note tagger dropped notes that were only a signature stamp. The manual, frozen first, said a stamp is not a taggable note. What is the right account?
62.Demand forecasting kept the holiday spike weeks. The planning note, written before the error, said the order decision includes those weeks. A colleague wants them dropped because they dominate the error. What do you say?
63.The ticket project dropped three tickets after they caused the largest losses. No rule about them was written beforehand, and accuracy then rose. What is the status of that drop?
64.Speech endpointing dropped clips where two listeners differed by more than one second. The labeling guide, written first, said those clips have no agreed endpoint. Is the drop in bounds?
65.Parts recommendation kept rare parts that appear about once a year. The decision note, written before the score, said a rare part still counts if it is stocked. Why is that the stated choice?
66.The sensor team dropped a week when a logger wrote zeros for every unit. The incident report, filed before modeling, said the logger failed and the zeros are not vibration. What may be dropped?
67.Delivery time kept the storm-week routes. The service note, written before the error, said the promise still applies on storm weeks. Why keep them?
68.Fraud scoring dropped test payments sent on the lab's own cards. The protocol, written first, said lab cards are not customer payments. What is the right reading?
69.The note tagger kept very long notes. The sampling plan, written first, said length is not a reason to exclude a note. A high error on them tempts a drop. What is the stated choice?
70.The demand project dropped the top one percent of store-weeks only after that drop cut the error. No reason for those weeks was written beforehand. What can you say?
71.Ticket tagging kept tickets with a single word. The queue guide, issued before scoring, said a one-word ticket is still routed. Why is that keep the stated one?
72.Speech endpointing dropped calls with clipping in the first half-second. The audio note, written before scoring, said the recorder truncated those files, so they have no true start. What is the drop?
73.Sensor failure reported accuracy of 97 percent. Failures are about two percent of unit-weeks, the decision is to catch them, and the model calls almost everything healthy. What is wrong with the report?
74.The delivery model reported a lower mean absolute error. Parcels more than a day late got worse, and the promise decision is about those parcels. What should the metric match?
75.The fraud score reported a higher area under the curve. The desk can review 200 payments a day, and the share of fraud in those 200 fell. What matches the desk?
76.The note tagger reported overall tag accuracy. The decision is to catch the rare callback tag, and those callbacks were mostly missed while overall accuracy rose. What is hidden?
77.Demand forecasting reported a better average error across store-weeks. The buy is judged on the worst stockout weeks, and those weeks got worse. What matches the buy?
78.The ticket tagger reported accuracy averaged across queues. The outage queue is the one that pages someone, and its recall fell while that average rose. What matches the page?
79.Speech endpointing reported a higher frame match on silence. Cutoffs in the middle of a word rose, and the product treats those cutoffs as failures. What matches the product?
80.Parts recommendation reported a higher hit rate at position 10. Technicians see only three suggestions, and the needed part left that top three more often. What matches the screen?
81.The sensor model reported better average precision. The night shift inspects only alarms above a fixed score, and failures above that score were caught less often. What matches the shift?
82.Delivery time reported a better median error. The decision flags routes more than two hours late, and more of those routes were missed. What matches the flag?
83.Fraud reported a lower average score on payments that later charged back. The desk holds only the top slice, and fraud in that slice was unchanged. What matches the hold?
84.The note tagger reported a higher exact match of every tag on a note. The decision only needs the refill tag to be present, and refill tags were dropped more often. What matches the decision?
85.The demand model reported a lower error in units. The replenishment call is the direction of the week, up or down, and the direction was wrong more often. What matches replenishment?
86.Ticket tagging reported higher accuracy on the tickets it marked confident. Routing still sends the other tickets, and those were wrong more often. What is hidden?
87.Speech endpointing reported a shorter average delay before declaring the end of speech. Early cutoffs, which the product treats as failures, became more common. What matches the product?
88.Parts recommendation reported a higher average overlap with the installed parts. Misfits on the first suggestion, a part for the wrong model year, became more common. What matches the failure?
89.Sensor failure reported higher recall at a threshold chosen after looking at the test week. Maintenance uses the threshold in the protocol, and recall at that line fell. What matches maintenance?
90.The delivery model reported a better error on the average parcel. A message goes out only when the model is sure, and those sent messages were late more often. What is hidden?
91.The note tagger raised agreement with the manual on the test week. The team says clinicians now follow up faster. What can the result claim?
92.The demand model cut error on the test month for stores already in the training history. The write-up says a store opening next quarter will have the same error. What is outside the result?
93.The ticket tagger improved accuracy on the billing and outage queues in the test week. The summary says a legal queue that was not in the data will tag the same way. Where does the claim stop?
94.Speech endpointing improved on English headset calls in a quiet room. The slide says the same score holds for a loud warehouse and for a second language. What is off the result?
95.Parts recommendation raised hit rate on brake jobs in the holdout. The note says electrical jobs will hit at the same rate. Electrical jobs were not in the holdout. What can the hit rate claim?
96.The sensor model improved catch rate on the two press lines in the test. The report says the oven line, which has no sensor of this type, has the same catch rate. What blocks that sentence?
97.The delivery model reduced error on ground routes inside the mapped cities. The summary says air routes between those cities improved by the same amount. Air routes were not scored. What stands?
98.The fraud score lifted precision on card-not-present payments in the holdout month. The review says account-takeover cases improved too. Those cases were not labeled. What can precision claim?
99.Note tagging improved on daytime notes from one clinic. The conclusion says night-shift notes and the other clinics improved as well. They were not in the test. What is the claim?
100.The demand forecast cut error for items that sold every week of the test. The deck says items that sold once have the same cut. Those items were excluded from the score. What is refused?
101.The tagger raised accuracy on tickets in the two languages present in the test file. Someone says a third language will match. No third-language ticket was scored. What do you tell them?
102.Speech endpointing improved on calls that were one short utterance. The claim says long calls with barge-in improved too. Barge-in calls were not in the test. Where does the result stop?
103.The parts model raised hit rate when the job card listed a model year. The summary says jobs with the model year missing improved too. Those jobs were dropped before scoring. What can you claim?
104.Sensor catch rate rose on failures that maintenance confirmed inside the test window. The write-up says unconfirmed glitches were caught too. Glitches were not labeled. What is the boundary?
105.Delivery error fell for parcels with a complete scan history. The report says parcels that skipped the depot scan improved by the same amount. Those parcels were not in the scored rows. What holds?
106.Fraud precision rose on domestic card payments in the holdout. The brief says cross-border payments moved the same way. Cross-border payments were not in the holdout file. What is the claim?
107.The tagger improved agreement on notes typed in full sentences. The conclusion says shorthand fragments improved too. Fragments were excluded before the test score. What can agreement claim?
108.The demand forecast reduced error on items whose price stayed stable in the test month. The close says items with a mid-month price change improved too. Those items were removed before scoring. What is refused?