Production readiness

Question 1

Six months after launch, nobody can rebuild the model that is in production. Which missing record is most often the reason?

Accepted Answer

The exact snapshot of training data that was used.

Question 2

What should be recorded for every training run so the model can be reproduced?

Accepted Answer

The code commit, data version, configuration and hyperparameters, environment and package versions, random seeds, and the resulting metrics and artifacts.

Question 3

Two training runs with the same code, data, and seed give models whose accuracy differs by 0.4 points on GPU. What is a likely reason?

Accepted Answer

Some GPU operations are not deterministic, for example the order of floating-point additions in parallel reductions.

Question 4

Why pin package versions, for example with a lock file or container image, for training and serving?

Accepted Answer

A minor library update can change default settings, preprocessing, or numerics, which quietly changes model behavior.

Question 5

Which seeds should be set for a reproducible run?

Accepted Answer

Seeds for every source of randomness: the language's random module, the numeric library, the deep learning framework, data shuffling, and data loader workers.

Question 6

Why should the train, validation, and test split be saved as fixed IDs rather than recomputed with a random split each time?

Accepted Answer

New rows added later would shift a recomputed split, so rows can move between sets and comparisons across runs stop being fair.

Question 7

What does a model registry provide?

Accepted Answer

A record of model versions with their lineage, metrics, and stage, such as staging or production, so deployment and rollback use a known artifact.

Question 8

What is data lineage, and why does it matter for a model?

Accepted Answer

A record of which sources and transformations produced the training data, so you can trace a problem back or find every model affected by a bad source.

Question 9

What is training-serving skew?

Accepted Answer

A difference between how features are computed or what data looks like in training versus in production.

Question 10

Training computed 'average order value' in SQL in the warehouse. Serving computes it in Python with a slightly different rule for refunds. What is the risk, and the fix?

Accepted Answer

The feature means different things in the two places; compute it once with shared code, or from a feature store, for both.

Question 11

How can you detect training-serving skew after launch?

Accepted Answer

Log the features actually used at serving time and compare their values and distributions with those computed offline for the same entities.

Question 12

What is a point-in-time correct join when building training data?

Accepted Answer

Joining each training example only with feature values as they were known at that example's time, never later values.

Question 13

A fraud model uses 'number of chargebacks on the account', computed from today's table, for transactions from a year ago. What is wrong?

Accepted Answer

Today's count includes chargebacks that happened after those transactions, which the model will not know at decision time.

Question 14

A feature is available offline right away, but in production it arrives from a batch job 6 hours late. What should training do?

Accepted Answer

Simulate the delay by using the value as it was 6 hours before each example, so the model learns with the freshness it will really have.

Question 15

What should a pre-launch evaluation for a model include beyond one overall metric?

Accepted Answer

Results by important slices such as region, device, and new versus returning users, a comparison with the current model, and checks on latency and cost.

Question 16

A new model is 1 point better overall but 6 points worse for users on older Android phones, who are 8 percent of traffic. What is the right response?

Accepted Answer

Treat it as a blocking regression, find out why that slice got worse, and decide with the product owner whether it is acceptable.

Question 17

What is a validation gate in a model release pipeline?

Accepted Answer

An automated check a candidate model must pass, such as beating the current model on key metrics without slice regressions, before it can be promoted.

Question 18

A nightly retraining job produced a model with half the usual accuracy, and it was deployed automatically. What safeguard was missing?

Accepted Answer

An automated comparison against the current model and sanity checks on the data and metrics, blocking deployment when they fail.

Question 19

What are data validation checks on a training pipeline's input?

Accepted Answer

Checks that row counts, schema, null rates, value ranges, and category sets are within expected bounds before training starts.

Question 20

An upstream team renames a category value from 'mobile' to 'Mobile'. The model's one-hot feature for mobile is now always zero. What would have caught it?

Accepted Answer

A check that the set of category values matches what training saw, plus an alert on the share of unknown values at serving time.

Question 21

What kinds of unit tests make sense for ML code?

Accepted Answer

Tests for feature functions on known inputs, shape and type checks, tests that preprocessing matches between training and serving, and a tiny end-to-end training run.

Question 22

What is a behavioral test for a model, as opposed to a unit test?

Accepted Answer

A check of how the model responds to designed inputs, such as 'changing only the name should not change the credit decision' or 'adding not should flip the sentiment'.

Question 23

Why check a model's results across groups such as age bands or regions before launch?

Accepted Answer

A model can have very different error rates for different groups, which can harm people and create legal or trust problems.

Question 24

What is a model card?

Accepted Answer

A short document stating the model's intended use, training data, evaluation results including by group, known limits, and owners.

Question 25

What is a shadow deployment?

Accepted Answer

The new model receives copies of real requests and its predictions are logged, but users only see the current model's results.

Question 26

What can a shadow deployment not tell you?

Accepted Answer

How users would react to its outputs, such as whether they click more, since they never see them.

Question 27

What is a canary release?

Accepted Answer

Sending a small share of real traffic to the new model, watching errors, latency, and key metrics, and widening the share only if they hold.

Question 28

A service handles 1 million requests a day. A canary at 5 percent receives about how many requests a day?

Accepted Answer

50,000

Question 29

What is a blue-green deployment?

Accepted Answer

Running two full environments, switching traffic from the old one to the new one at once, and keeping the old one ready to switch back.

Question 30

A new model went out, and conversions dropped. How fast should rolling back be, and what makes that possible?

Accepted Answer

Minutes, by keeping the previous model version deployable and switching to it with a config change or flag, without a new build.

Question 31

A new model needs a new feature, and the feature pipeline was changed at the same time. Why does that make rollback harder?

Accepted Answer

If the old model's features were changed or removed, switching back to the old model may also fail; keep both feature versions available during the rollout.

Question 32

Why use a feature flag to turn the model on and off?

Accepted Answer

It separates deploying the code from releasing the behavior, so you can turn the model off or limit it to some users without a redeploy.

Question 33

What is a sensible fallback when the model service times out?

Accepted Answer

Return a safe default, such as a rules-based result, a cached prediction, or the popular items, and log that the fallback was used.

Question 34

A request calls three services one after another, taking 120 ms, 80 ms, and 200 ms. What is the total latency, and what if they could run in parallel?

Accepted Answer

400 ms one after another, about 200 ms in parallel

Question 35

Why are latency targets usually written as p95 or p99 rather than the average?

Accepted Answer

The slow tail is what frustrates users, and an average can look fine while 1 in 100 requests takes seconds.

Question 36

A page makes 20 independent calls to a model service. What is the chance that at least one call is slower than that service's p99?

Accepted Answer

About 18 percent

Question 37

A model takes 50 ms per request on one worker, one request at a time. About how many workers are needed for 500 requests per second, with no batching or headroom?

Accepted Answer

25

Question 38

Why does dynamic batching raise throughput on a GPU model server?

Accepted Answer

It groups requests that arrive close together into one batch, so the GPU does more work per pass, at the cost of a short wait.

Question 39

A model's predictions for a user only change once a day. How can you cut serving cost and latency?

Accepted Answer

Precompute predictions in a daily batch job and serve them from a fast key-value store.

Question 40

When is online, real-time scoring worth the extra complexity over batch scoring?

Accepted Answer

When the prediction depends on information from the current request or the last few minutes, such as a transaction being scored for fraud.

Question 41

What is an error budget, given a monthly availability target of 99.9 percent over 30 days?

Accepted Answer

About 43 minutes of allowed downtime that month.

Question 42

A request depends on two services in series, each available 99.9 percent of the time, independently. What is the combined availability?

Accepted Answer

About 99.8 percent

Question 43

Why set a timeout on calls to the model service?

Accepted Answer

Without one, a slow model can tie up callers' threads and connections until the whole product slows down.

Question 44

Clients retry failed model calls immediately, three times each. During a brief overload, what can happen?

Accepted Answer

Retries multiply the load on an already overloaded service and can keep it down; retries need backoff with random jitter and a limit.

Question 45

What does a load test before launch tell you?

Accepted Answer

How latency and errors behave as traffic rises, and where the service breaks, so capacity can be planned with headroom.

Question 46

Autoscaling adds model servers when CPU is high, but GPU-bound servers stay overloaded. What is wrong?

Accepted Answer

The scaling signal does not match the real bottleneck; scale on GPU use, queue length, or request latency instead.

Question 47

Why does a large model's startup time matter for production?

Accepted Answer

If loading takes several minutes, new servers cannot help with a sudden spike in time, and restarts after crashes are slow.

Question 48

What is the difference between a liveness check and a readiness check for a model server?

Accepted Answer

Liveness says the process is running and should not be restarted; readiness says the model is loaded and the server can take traffic.

Question 49

What is data drift?

Accepted Answer

A change in the distribution of the model's inputs compared with the training data.

Question 50

What is concept drift?

Accepted Answer

A change in the relationship between inputs and the target, so the same inputs now lead to different outcomes.

Question 51

Labels for a loan model arrive 90 days after each decision. How can you monitor the model before those labels come in?

Accepted Answer

Watch input drift, prediction distribution, approval rates, and early proxy signals such as first missed payment, then confirm with real labels as they arrive.

Question 52

The share of transactions a fraud model flags jumps from 2 percent to 9 percent overnight, with no model change. What is the first thing to check?

Accepted Answer

Whether an input feature broke, such as a pipeline sending nulls or a change in units, before assuming the world changed.

Question 53

What does the population stability index (PSI) measure?

Accepted Answer

How much a variable's distribution in one period differs from a reference period, using binned shares.

Question 54

Drift alerts fire every day on dozens of features, and the team ignores them. What is wrong with the setup?

Accepted Answer

Thresholds are too sensitive or not tied to impact; focus alerts on the most important features and on shifts that affect predictions or business metrics.

Question 55

Which signals belong on a model's production dashboard?

Accepted Answer

Request rate, latency percentiles, error and fallback rates, input and prediction distributions, model quality as labels arrive, and the business metric it supports.

Question 56

When should a model be retrained?

Accepted Answer

On a schedule set by how fast its data changes, or when monitoring shows quality dropping, with the new model passing the same gates as any release.

Question 57

A recommendation model is retrained daily on clicks from items it recommended. What long-term risk does that create?

Accepted Answer

A feedback loop: it keeps learning that its own picks are popular, narrowing what users see and starving new items of data.

Question 58

Why log the model version with every prediction?

Accepted Answer

So any prediction, complaint, or metric change can be tied to the exact model that produced it, especially during rollouts.

Question 59

What should a prediction log include so problems can be investigated later?

Accepted Answer

A request ID, timestamp, model version, the features used, the prediction and score, and the decision taken, within privacy limits.

Question 60

A model makes credit decisions. What extra requirement often applies in production?

Accepted Answer

Being able to explain individual decisions, give reasons to declined applicants, and keep records for audits.

Question 61

How does an engineer typically give per-prediction explanations for a tree ensemble?

Accepted Answer

With feature attribution methods such as SHAP values, which split a prediction into contributions from each feature.

Question 62

Who should be on call for a model in production, and what should they have?

Accepted Answer

A named owner with alerts, a runbook covering common failures, and the access to roll back or switch to a fallback.

Question 63

What should a model runbook cover?

Accepted Answer

What each alert means, how to check the main dashboards, how to roll back or turn on the fallback, who owns upstream data, and how to escalate.

Question 64

After an incident where a bad model shipped, what should the follow-up focus on?

Accepted Answer

What allowed it through, such as a missing gate or test, and concrete changes so the same kind of failure is caught next time.