Training pipelines

1.The nightly train for Brine Market's out-of-stock model fails at 01:40. The scheduler paints the whole DAG red. The log's last line is inside the step that reads shipments. What do you open first?

2.The exception is that cold_chain_minutes is not on shipment_features. Yesterday's run selected it. What broke?

3.The producer renamed cold_chain_minutes to chill_minutes in the same hour as the train, and left no alias. The job is written to stop when the expected name is absent. What is the right outcome?

4.A helper reads cold_chain_minutes with a default of 0, so the rename does not crash. The run finishes green. What did the model fit?

5.The producer can keep the old column name alive for two weeks while consumers move to chill_minutes. What is that two weeks?

6.Column names match the checked list, but cold_chain_minutes arrives as text. The trainer casts with a strict integer parse and throws on an empty string. What did the name-only check miss?

7.The schema check passes. After a bad load, 70 percent of cold_chain_minutes is null and the rest is real. The trainer still fits. What check is still quiet?

8.Monday's partition for ship_date 2026-10-31 never arrived. The trainer reads the latest partition it can find and fits. The row count matches Sunday. What did it train on?

9.The warehouse task failed. The trainer caught the error and exited 0, so the DAG is green. What should that exit mean?

10.A killed writer left a partial parquet directory. The trainer reads it, fits, and records an AUC of 0.81. What do you refuse?

11.The 01:40 attempt is still writing the prefix models/brine/current when the retry starts and writes the same prefix. What can a later publish pick up?

12.Each attempt writes only under runs/ plus its own start time, and a publish step runs after that write completes. What does a retry write?

13.The train runs out of memory after a batch-size change that merged at 18:00. Saturday's green run used the same machine and the same partition size. What failed?

14.The same batch size runs out of memory only on the day after a holiday, when the partition is about four times larger. What has to follow the input size?

15.The trainer image is brine-train:latest. Last night's pull brought a new numeric library and the job died on import. What would have kept Saturday's image?

16.The warehouse token expired at 00:30. The stack trace is an authentication error. A retry at 02:00 uses the same token. What happens?

17.The train step failed, and a later step still registered yesterday's checkpoint under today's run id. What edge is missing?

18.On-call reruns the red job at 09:00 against the live feature table. The 01:40 schedule would have read the midnight snapshot. The 09:00 table includes a morning repair. What is the rerun?

19.promo_flag is in the schema and is null on 40 percent of rows after a bad load. The missing-column check stays quiet. What failed?

20.Every value of lane_code is the four characters n-u-l-l. The schema says string, so the type check passes. What should fail the run?

21.Training SQL counts substitutions on the calendar day. The online feature counts a trailing 24 hours from the request. Both are named subs_1d. Where is the skew?

22.Training lowercases city. The server passes Austin through unchanged. The fitted levels include austin and not Austin. What happens on a score request?

23.Training joins the shopper's city as of the label date. Serving joins the shopper's city as of the request. A shopper who moved is scored with the new city. What did training do that serving does not?

24.Training drops rows with a null postal code. Serving fills a null postal code with the store's code. Live traffic is full of those nulls. What did the model never see?

25.Training standardizes basket dollars with the mean of the training snapshot. Serving divides by 42, a mean copied from a notebook two quarters ago. What disagrees?

26.The one-hot encoder was fit on the training vocabulary and emits 36 columns. This morning a bag_type of paper_tall appears. The server still emits 36 columns and drops the new level. The request scores. What is the quiet failure?

27.The decision is the delivery. The feature returns_already is computed at training time with no cutoff, so it includes a return filed ten days later. The label is that same return. What is inside the feature?

28.Target encoding of brand_id uses the average label on the full training frame, including the row itself. A brand with one row gets a code equal to its own label. What leaked?

29.Each shopper has rows across many weeks. The split is a random cut of rows. A May label sits in the test set while that shopper's April row is in the train, and the April features include May activity. What did the split fail to hold out?

30.You switch to a split by shopper. The feature lifetime_return_rate is still computed on all of that shopper's history, including weeks after the decision. What is still leaked?

31.The label is returned within 14 days. The feature return_count_14d is filled with returns filed on day 20 because the label job waits 30 days. What did the longer wait put in the feature?

32.The training query is select star from features with no as-of time. Two reruns a day apart both succeed. What can you not claim?

33.The logged snapshot id is the view name features_current. Someone edits the view. The id in the registry does not change. What stayed stable?

34.Iceberg snapshot 88421 is on the model record. A retention job drops snapshots older than three days. The model is the one you would restore next week. What did retention remove?

35.The SQL filters ship_date to current_date minus one day. A delayed rerun on Tuesday reads Monday. The run you wanted to rebuild was Sunday's schedule. What should be the parameter?

36.The run log says data_date is 2026-10-31, and one CTE still filters on current_date. The log says Sunday. That CTE reads Tuesday. Why is the log wrong?

37.Shoppers are read from snapshot s1 at 02:00. Orders are read from snapshot s2 at 06:00 after a repair. The contract says one cutoff. What should the run do?

38.The snapshot is a csv under Downloads, and the path is written in a doc. The file is overwritten on the laptop before promotion. Which rows can you show for that train?

39.The frame hash was taken after an unseeded shuffle, and the hash includes row order. Two identical sets hash differently. What do you hash?

40.The git commit and the table snapshot are on the run. The package that builds subs_1d is not. A feature change merged after the schedule fired and before the job started. What can you not explain later?

41.Offline AUC moved from 0.74 to 0.76 in the same change that redefined session_depth from pages to seconds. What is that 0.02?

42.The registry accepts the new ranker. The feature service still writes session_depth in pages. Live requests feed pages into a model fit on seconds. What has to move together?

43.The model signature expects 40 floats. After the feature change the server sends 41. Where should that stop?

44.You roll back to model v17, which was fit with bag_fee coded as 0 when the fee was absent. Serving still writes a 0 for old models. What has to be true of that 0?

45.The training owner wants to promote at 17:00 on Friday. The feature owner has not accepted the new definition of session_depth. Who is missing?

46.An automatic gate checks AUC, calibration, and a global recall floor. It passes. New sellers are 4 percent of rows and are not in the gate. What is the human sign-off for?

47.The approve button reads latest. Two people approve five minutes apart, and two different builds were latest. What should the button show?

48.Promotion is a copy of the newest file in the bucket, done by whoever holds the key. The registry has no approval event. What can you not answer later?

49.Serving points at model v18. The image's preprocessor is the v19 transformer. You point serving back at v17. What did that pointer change leave in place?

50.v17's weights are in the registry. The online store has only values written by the v19 feature job, under the same keys. You send traffic to v17. What does v17 score?

51.You revert the feature-library commit and leave the new model serving. That model requires bag_fee. Requests start failing on a missing input. What did the one-sided revert break?

52.A canary of the new model is supposed to be the only reader of the new feature path. The online store uses one unversioned key, and the canary job overwrote it. What did the rest of the traffic read?

53.The class name is in the training filename, damaged_8841.jpg, and a text feature includes that filename. The holdout is a random cut of files. What can the model use instead of the photo?

54.Two listing photos are 95 percent the same pixels. One is in train and one is in test. The split key is the file id. What does the holdout score overstate?

55.The feature days_until_return is the return timestamp minus the order timestamp. The label is whether the shopper returned the order. What is the feature at decision time?

56.Training rows are shoppers who finished a feedback form. Serving scores every checkout. The gate's AUC is on the form-fillers. Who is the gate silent about?

57.The label table is empty because label_ready never flipped. The trainer still fits, treats every label as 0, and registers a model. What assertion is missing?

58.A dropped filter trains on unlabeled rows filled with 0. The row count is about ten times the usual band. What do those zeros become?

59.Corrections land in the label table after the training snapshot. A rebuild the next week does not match the promoted run, and you cannot say which labels that run used. What should the snapshot have frozen?

60.The promotion gate scores a golden set built with the old feature code. The candidate was fit on the new code and looks worse. What is wrong with the comparison?

61.The golden set is rebuilt every night from live rows, including this week's outage. It no longer sits still across promotions. What do you keep if you want a stable gate?

62.The search log names the winning tree and not the sample that tree was fit on. The sampler was unseeded. What can a later run not rebuild?

63.The job reads a stream from latest and has no end offset. The run record says it trained on Tuesday. What set is Tuesday?

64.subs_1d is an embedding from a second model. The run logs the ranker commit and not the embedder id. Next week the embedder is retrained. What do you lose for the old ranker?

65.You promote the ranker and leave the previous embedder in serving. The widths match. Offline AUC was computed on the new embeddings. What do live scores use?

66.Raising the block threshold will refuse more good orders. The model owner can sign the artifact. Who signs the threshold?

67.The third retry of a red train starts, and the page clears because the run is in progress. The third try fails on the same missing column. When should that page clear?

68.The metric step writes a healthy AUC, then the publish step fails. Production is unchanged. The status page shows the AUC tile in green. What is that tile?

69.The order id is an int in one table and a string in the other. The join matches nothing, every feature is null, and the model sits on the majority-class baseline. What should have failed first?

70.A refactor drops the join condition and the join becomes a cross product. The row count explodes. The match rate is no longer a useful signal. What sees the explosion?

71.Training's scaler skips nulls. Serving fills a null fee with 0 and then scales, so a missing fee becomes a real standardized value. What has to be the same library call?

72.Training maps rare bag_type values to other using a frozen frequency table. Serving leaves rare values as themselves, then the embedding misses and the request errors. What does the server have to apply?

73.vocab.json sits beside the weights and is overwritten on every train. You roll the weights back to v17. The file on disk is v19's vocabulary. What should hold v17's vocabulary?

74.hour_of_day is UTC in the warehouse and local time on the server. A 09:00 order is a different number on each path. What belongs in the contract?

75.The label is clicked within one hour. The train keeps only clicks that had arrived by the job start, so early-morning clicks still in the queue are labeled 0. What is biased?

76.The snapshot is a list of object keys. One key is lane/current.parquet, which the producer overwrites. The other keys are immutable. Which pin is weak?

77.The reproduce script checks out the git tag and then installs the feature package with no version pin. What is not the run you think you are rebuilding?

78.The promotion test reads a path that defaults to the training split. The gate score matches the training score. What is circular?

79.The same person may approve promotion and may lower the AUC floor in one change. A failing model can be made to pass. What do you separate?

80.In a staging drill you roll the model id back and leave the feature flag on the new transformer. The checklist says the model was rolled back. What is unfinished?

81.The feature-flag service is down. The client defaults to the new transformer, which is the broken one. What does a flag rollback do during that outage?

82.A morning repair rewrites Sunday's labels after Sunday's job has already read its snapshot. The in-flight run used the pre-repair rows. What do you not do quietly?

83.A repair deletes rows whose returns were entered in error. The promoted model was fit on labels you no longer believe. What is the honest next step?

84.The model card lists features A, B, and C. The artifact signature is A, B, and D. A reviewer checks the card. What should serving check?

85.A column stays named fee_cents while the unit changes from cents to dollars, and the feature version does not change. What is the unchanged name doing?

86.On a feature-store timeout, serving uses a fallback model trained with fewer features. The promotion gate never calls that path. What did sign-off skip?

87.The GPU pool rejects the nightly job. Someone fits on a laptop and uploads the file into the production bucket. There is no snapshot id. What is the upload?

88.The bucket policy still allows developer laptops to put objects into the promotion prefix. What closes that door?

89.The SQL text is stored on the run. A warehouse function that SQL calls is replaced later, and a replay does not match. What else was unpinned?

90.A date filter compares a string to a timestamp, so it does not prune, and the job reads two years. The label mix shifts. What band catches the oversized read?

91.Last year's byte band pages every night after traffic tripled. Someone sets the band to anything, and the next missing partition fits inside it. What did that edit remove?

92.Training uses an exact count of distinct shoppers per device. Serving stores a sketch under the same feature name. What is true of the two numbers?

93.Sample weights are one over a propensity. The propensity model was fit on rows that include the outcome you are about to predict. What do the weights carry?

94.Three nightly failures are a missing column that appears only when an upstream commit marker wins a race. The fourth retry is green. What does rerunning until green hide?

95.The sensor succeeds when the directory for ship_date 2026-10-31 exists. The producer has created the directory and is still writing part files. The train starts. What should the sensor wait for?

96.The promotion diff of features compares names and shows no changes. The transform behind session_depth changed from pages to seconds. What is absent from that diff?

97.You revert the feature commit. The online store still holds values computed by the bad code, and the next full refresh is six hours away. When has the rollback actually happened?

98.Both feature versions are stored. The model rollback reads v17's keys. New writes still go to v19, and v17's keys expire after one hour because they inherited the new TTL. What is shorter than the policy you need?

99.Champion against challenger promotes the challenger automatically when AUC wins. This week the feature version also changed. What do you turn off for that case?

100.A check requires every label column to be non-null. A new optional head is null by design, and the nightly job goes red. What should the check cover?

101.That null check is skipped in the same commit as a feature-definition change. The run goes green and the skip is promoted with the change. What is the skip?

102.During an incident you need model v14. Its weights are in the registry. The feature-code image that built v14's vector was deleted six versions ago. What can you not do?

103.The train writes the artifact, then writes the success row. The process dies between the two writes. The file is complete and the run is not marked success. What is that state?

104.You reverse the order: the success row is written, then the artifact. The process dies between them. What can a deployer believe?

105.Two customers' rows land in one batch because a partition filter was dropped. The score is the smaller problem. What else happened?

106.The trainer pull request shows the code and not the snapshot. Reviewers approve. The row count tripled because a warehouse filter was removed outside the diff. What could they not see?

107.You are writing the on-call line for a red nightly train of the out-of-stock model. Which line is the one to leave?

108.A shadow path scores the new out-of-stock model on live checkouts and writes nothing the shopper sees. Its features come from a branch the server does not run. Shadow AUC looks strong. What can you not conclude?