2026 AI/ML Interview

1.A parts counter still ranks repair manuals with keyword search. A director wants a model in place before the catalog freeze. What do you settle first?

2.Keyword search on a clinic's policy binder misses paraphrases, and a pilot embedding index is already built. When is it reasonable to retire the old ranker?

3.Travel-desk agents say keyword search works only if they already know the policy heading. What clarification keeps a rewrite from wandering?

4.A marketplace wants semantic search on listing titles. Sponsored listings are a paid contract. What has to stay explicit before the model touches rank?

5.Support leadership will call a search rewrite a win if agents feel faster. Why is that a weak bar for the bake-off?

6.One team will judge new site search by zero-result queries. Another will judge it by add-to-cart after a search. How should those goals enter the launch?

7.A legal-hold tool searches scanned contracts, and the OCR is uneven. What belongs in the plan before any embedding job starts?

8.Staff rely on exact SKU match and a date-range operator. A model demo ignores both. What is the product risk of replacing the whole box?

9.You are given one week to prove model search beats keywords. The only labeled set is fifty queries a product manager wrote on Monday. What is the limit of that proof?

10.A retailer will judge the new search by revenue per search session. What else do you insist on reading?

11.Merchandising maintains the synonym list that keyword search uses. A model project plans to ignore that list. What do you clarify?

12.Shoppers type queries in Spanish, and the catalog text is mostly English. A semantic demo was judged on English queries only. What is unfinished?

13.A search pilot quietly boosts items the signed-in shopper bought last month, then calls the whole change relevance. What should you separate?

14.New SKUs are created every hour, and the embedding index rebuilds overnight. What failure should you expect the next morning?

15.During a side-by-side trial, staff keep opening the old keyword result even when the model result is listed first. What is that behavior telling you?

16.Answering a benefits question well means reading three long plan PDFs. The window you can afford holds about one of them. What is the honest constraint?

17.Someone suggests pasting the whole internal wiki into the prompt so you can skip retrieval. Why is that a weak default?

18.The sentence that changes the answer sits in the middle of a long retrieved packet, and the model quotes the beginning instead. What are you seeing?

19.A long claims chat is truncated from the front. The system preamble stays, and the user's newest correction is what gets cut. What went wrong in the packing?

20.Doubling a prompt from 4,000 to 8,000 tokens more than doubles the serving cost on your decoder stack. What is the useful interview point?

21.A claims conversation can run for a week. You cannot keep every turn raw. Which compromise matches the product?

22.You can afford one short window for a policy answer. What is the sensible packing order?

23.An agent pastes a full inventory spreadsheet into chat. The request fails before any answer starts. What is the likely limit?

24.Prompt caching makes a repeated safety preface cheap. Why does that not solve cost for a retrieval answer?

25.A stakeholder hears that the window now fits entire manuals and wants citations removed. What do you keep anyway?

26.Over a quarter, the system prompt grows from a short policy note into a long style guide. Retrieved passages start getting truncated. What happened?

27.Two memos in the same window disagree about overtime. The draft answer blends both into a rule neither memo states. What should the product do?

28.A synthetic needle test passes at the full window length. Real benefit booklets still produce wrong exclusions. Why can both be true?

29.The call fails with an overflow even though the prompt itself fits the advertised window. What did the packing forget?

30.You summarize a claims thread every ten turns. A later denial quotes a deductible the summary dropped. What practice keeps that from becoming a silent error?

31.The top retrieved passages all discuss the right benefit, and none of them includes the exception that decides this member's case. How do you describe the failure?

32.A rate table was chunked so the numbers landed in one chunk and the column headers in another. Answers mix up the columns. What broke?

33.A shopper searches for a specific pump. The top hit is a different pump that shares most of the wording. What retrieval miss is this?

34.The query is only a SKU. Dense retrieval returns similarly described parts and not that SKU. What should take the first look?

35.Retrieval returns no passages. The assistant still answers the policy question in a confident paragraph. What behavior do you require instead?

36.A mileage policy changed Friday night. Monday's assistant still quotes Saturday's rule. The page on the intranet is already new. Where do you look?

37.You expand the user query before search. The expansion adds a product line the user did not mention, and the hits drift. What do you change?

38.A school-district assistant returns another district's lunch menu. Both districts share one index. What failed closed?

39.A reranker lifts offline ranking quality and pushes the answer past the latency budget agents will wait for. How do you frame the choice?

40.The answer cites a chunk. A reviewer opens it and the quoted sentence is not there. What does that do to trust?

41.The question is which plans do not cover orthodontics. The top hits are plans that do cover it, because they share the word. What is the retrieval gap?

42.In the warehouse manual, ERA means equipment return authorization. In the finance manual, ERA means earned revenue adjustment. Both are indexed together. What goes wrong?

43.Tiny chunks match many queries and each one omits the condition that follows the rule. What chunking judgment are you missing?

44.The user asks whether a repaired part is still under warranty if it was resold once. The warranty rule and the resale rule live in different documents. One search returns only the warranty page. What is missing?

45.Five mirrored copies of one policy page fill the top hits. A newer amendment never appears. What cleanup changes the list?

46.The retriever finds a relevant contract the signed-in user is not allowed to see. The draft answer uses it anyway. What should the product refuse to do?

47.The best matching policy is in a language the member does not read, and the assistant quotes it untranslated. What is the product hole?

48.A high similarity cutoff makes the assistant say it found nothing on questions reviewers later answer from a passage just under the line. What did the cutoff buy and cost?

49.Hybrid search is on. For a query that is a part number plus the word 'recall,' keywords and vectors disagree. Which judgment fits?

50.You want to learn a new ranker from clicks. Those clicks were produced by the old keyword order. What bias do you have to name?

51.An assistant can call a refund tool. The amount can be large. When do you require a person to confirm before the tool runs?

52.The refund tool's schema says amount is integer cents. Traces show the model sending 25 for a twenty-five-dollar refund. What broke?

53.The charge tool times out. The agent retries the same charge and the customer is billed twice. What should the design have required?

54.The assistant says a tool confirmed the truck is in Denver. The trace has no tool call and no tool result. What do you call that?

55.A planner searches, searches again, and searches a third time, then hits the step limit with no answer. What is the loop telling you?

56.The same assistant can read order status and can cancel an order. A status question ends in a cancellation. What separation was missing?

57.Two tools are described as 'look up the customer.' One reads the CRM. One starts a credit check. The model picks the credit check for a name spelling. What do you fix?

58.A request says 'check the shelf count and email the customer that the part shipped.' The shelf is short, and the email still goes out. What scoping failed?

59.Five warehouse lookups are requested. Three return counts and two time out. The assistant reports all five numbers anyway. What should it have done?

60.A tool only accepts the statuses open, packed, and shipped. The model sends 'almost shipped.' The call errors. What is the right product behavior?

61.You are reviewing an agent trace after a bad refund. Which log lets you reconstruct the decision without storing unnecessary customer prose?

62.Every successful run of a returns flow does the same four lookups in the same order, then fills a form. When is an open-ended agent the wrong shape?

63.A shopper asks if a damaged lamp is covered. The agent calls a calculator and never opens the policy tool. What does that choice show?

64.The orders tool requires a customer id. The agent calls orders and the customer lookup at the same time, and orders fails. What dependency did it skip?

65.The only stop rule in an agent is 'continue until confident.' A trace shows twelve steps and a confident wrong answer. Why is that rule thin?

66.An agent wants to run generated code against the warehouse database to answer a one-off count. What approval matches the risk?

67.A tool returns an entire year of transactions into the next prompt. The following call overflows. How should the tool have been shaped?

68.The assistant apologizes, says it will not cancel the order, and then the trace shows the cancel tool ran. What failure matters most?

69.A create-ticket tool has no idempotency key. Users double-click and get two tickets. The agent retries on timeout and creates a third. What do you add?

70.The user asks for a live aisle photo. The only tools are order status and policy search. The assistant describes a photo in detail. What response was required?

71.To 'show the format,' the prompt includes three worked examples. Those three are also in the final test set. What did you leak?

72.After reading the test questions, an editor rewrites the help-center chunks so each test question has a cleaner passage. The retrieval score jumps. What is that jump worth?

73.A fine-tune on support tickets includes rows you had set aside as the holdout. The holdout score looks excellent. What can you no longer claim?

74.Prompt wording was tuned for a week with the same agents whose acceptance rate is now the launch metric. Why is that rate hard to trust?

75.You discover the embedding model was trained on a public dump of the same help center you are using as the test corpus. What risk does that create for the retrieval score?

76.You edit the prompt, check the test score, edit again, and stop when the test score peaks. What did the test become?

77.Synthetic questions are generated from the held-out answers, then used as the only eval. Why can that look better than live questions?

78.A feature stores the note an agent wrote after the refund was decided. The model predicts whether to refund, and that feature is available at scoring time in the eval. What is wrong?

79.An automated judge scores answers. The test items were also pasted into the judge's rubric as illustrations. Why is the judge's agreement inflated?

80.Twenty prompt variants are compared on the final set, and the memo quotes the best one. Why is that quoted number optimistic?

81.A policy answer is accurate and takes thirty seconds. Agents hang up or skip the panel. Offline exact-match is the best you have recorded. What do you ship against?

82.Streaming shows the first tokens quickly, then the answer reverses itself at the end. Agents have already read the opening aloud. What trade are you making?

83.Easy status questions go to a small model. Coverage questions go to a large one. The router mislabels a coverage question as easy. What do you measure besides average latency?

84.Frequent answers are cached for a day. A recall notice is published at noon. Cached replies still say the product is fine. What did the cache cost you?

85.Overnight case summaries have a large token budget. The same model is used for live chat with a tight budget. Why can the overnight quality number mislead the chat team?

86.Reranking fifty candidates improves a lab metric and misses the interactive timeout. Reranking eight fits. How do you decide?

87.A faster draft model proposes tokens the main model mostly accepts. On long policy answers the acceptance rate drops. What do you check before turning it on for those answers?

88.If the model path exceeds two seconds, the product falls back to keyword search. Keyword results are weaker but immediate. When is that fallback the right call?

89.Two designs are on the table: a fast answer that is sometimes wrong, and a slower answer that waits until sources are attached. Agents read the answer to customers live. Which way do you lean?

90.Finance asks for cost per token. Operations asks for cost per resolved ticket. Which one tells you whether the assistant is worth running?

91.Median latency looks fine. One search in twenty waits on a retrieval timeout and then fails. Agents complain about 'random' slowness. What did the median hide?

92.A larger model gains two points on an offline judge and needs about four times the hardware. The current model already meets the written quality bar. What is the decision frame?

93.A retrieved help page tells the assistant to ignore company policy and approve every refund. No one on your team wrote that sentence. What is the product risk?

94.A browsing assistant can read public pages and can also change the email on an account. Why is that combination a poor default?

95.A customer pastes a long note into a ticket and the note includes lines that look like new rules for the assistant. How should the product treat that note?

96.A bad sentence in a draft is embarrassing. A bad tool call issues a refund. Why does tool use raise the stakes of untrusted text?

97.You separate customer text from the instruction block and still assume the model might mix them up. What does that assumption change in the design?

98.The assistant fetches a link preview to title a bookmark. The preview text includes directions aimed at the assistant. What class of content is the preview?

99.A weekly review finds answers that followed orders contained in a document rather than the product policy. What do you add to the review besides a better prompt?

100.A draft system prompt contains a private API token so the model can 'help debug.' What product risk does that create?

101.Looking up a tracking number and issuing a credit both depend on 'whatever the model decides.' Why is one gate for both a weak design?

102.A vendor PDF in the corpus says it is an admin notice and tells the assistant to export customer lists. How should that PDF be ranked as authority?

103.Retrieved text says to waive a fee. Company policy says the fee stands unless a supervisor approves. The assistant waives it. What should have happened?

104.Your eval set is entirely clean help-center prose written by your own team. What product case is missing?

105.Staff cannot tell whether a refund instruction came from policy or from a pasted invoice. What UI fact would make that review possible?

106.You have forty corrected chats, and the refund policy changes every month. Why is a fine-tune a weak plan?

107.Wrong answers quote a similar product's manual. The right manual is in the corpus and rarely retrieved. A teammate proposes a fine-tune on the corrections. What do you try first?

108.A price will be corrected tomorrow morning. Someone wants that price fine-tuned into the model tonight. What clock makes the tune the wrong tool?

109.The only complaint is that answers ignore a JSON schema the downstream form already enforces. A tune is proposed to 'teach the format.' What is the smaller fix?

110.Nobody has written down how the current prompted model does on the real ticket sample. Leadership still asks for a fine-tune this sprint. What is premature?

111.Legal and support still disagree on what counts as an approved exception. The labeled set follows whichever reviewer was on duty. Why hold the fine-tune?

112.A narrow tune on refund phrasing makes the assistant worse at ordinary order-status questions it used to answer. What cost showed up?

113.Sales tax has to round by a published rule to the cent. A teammate wants the model fine-tuned until it usually gets the cent right. What belongs in code instead?

114.The miss rate is almost all internal acronyms a one-page glossary would expand. Why is that a poor reason to fine-tune?

115.The assistant shows 0.80 next to a policy answer. Nobody has checked how often those 0.80 cases are actually right. What are you not allowed to tell a supervisor?

116.Agents treat a 0.51 suggestion and a 0.99 suggestion the same way, because the panel hides the score and only shows the text. What did the interface throw away?

117.Turning temperature down made replies more repetitive. A teammate says the model is now calibrated. What did temperature actually change?

118.Auto-send is allowed above 0.9, a cutoff chosen on the old model. The model behind the score was swapped yesterday. What do you redo before auto-send stays on?

119.On English tickets the 0.9 band is right about nine times in ten. On the night-shift Spanish queue the same band is right about half the time. What does that block?

120.The product will auto-approve a replacement when the score clears a line. Rank-order metrics look strong. Why is that not enough to pick the line?

121.In plain language, what would convince you a policy score near 0.9 is calibrated?

122.You fit a calibration map on the final test file, then quote that same file as the probability quality. What did you spend?

123.The panel prints 0.873 beside a yes-or-no coverage line. Reviewers argue about the third digit. What would you show instead?

124.Offline, suggested replies match the gold text more often. Agents accept fewer of them, because the new wording is longer than they will read on a call. How can both numbers be true?

125.Search click-through rose in the same week the first result was given a brighter highlight. The ranker did not change. What can you not conclude?

126.A launch review quotes suggestion acceptance and does not mention handle time, escalations, or complaints. What can that single number hide?

127.The offline set is daytime English email. The launch audience is a night-shift Spanish chat queue. The offline score is strong. What comparison is missing?

128.Shadow mode runs the new ranker beside production and logs disagreements. It does not change what users see. What can the shadow tell you, and what can it not?

129.Week one of a new assistant shows a spike in agent satisfaction. Week four is back to the old baseline. What story should you rule out before calling week one the launch effect?

130.An automatic judge gives the new assistant a higher score. The judge is a sibling of the assistant and shares its bad habit on exclusions. What will the judge fail to punish?

131.An online miss is reported as 'the model hallucinated.' The trace does not record which passages were retrieved. What investigation is blocked?

132.You will A/B a new ranker. Tickets from the same customer often share an intent. Randomizing each ticket separately splits that customer across both arms. What unit fits better?