Card 1 of 50
Front. Production-ready. What does production-ready mean for a model?
These cards are original study material. They are not questions from a named employer or a live hiring loop.
All 50 cards in this deck
1.Production-ready
What does production-ready mean for a model?
Reproducible, passes agreed quality and slice checks, meets latency and cost budgets, is monitored, has an owner, and can be rolled back quickly.
2.Reproducibility
What is reproducibility for a training run?
Being able to rerun training and get the same model, or one within known variation, from recorded inputs.
3.Run-record
What should be recorded for each training run?
Code commit, data version, configuration and hyperparameters, environment and package versions, seeds, metrics, and artifacts.
4.Seeds
Which random seeds matter for reproducibility?
Every source of randomness: language, numeric library, framework, data shuffling, and data loader workers.
5.Nondeterminism
Why can GPU training differ run to run with the same seed?
Some parallel operations add numbers in varying order, and floating-point results depend on order.
6.Data-versioning
What is data versioning?
Recording an exact, retrievable snapshot of the data used for training, so it can be reused later.
7.Env-pinning
What is environment pinning?
Fixing exact package versions, for example with lock files or container images, so library updates do not change behavior unnoticed.
8.Experiment-tracking
What is experiment tracking?
Logging each run's settings, metrics, and outputs in one place so runs can be compared and found later.
9.Model-registry
What is a model registry?
A catalog of model versions with their lineage, metrics, and stage, used as the source for deployment and rollback.
10.Lineage
What is lineage?
A record of which data, code, and steps produced a model, so problems can be traced and affected models found.
11.Data-validation
What is data validation?
Automated checks on incoming data, such as row counts, schema, null rates, ranges, and category sets, before training or scoring.
12.Schema-check
What does a schema check catch?
Missing, renamed, or retyped columns, and new or changed category values from upstream.
13.Unit-tests
What should unit tests cover in ML code?
Feature functions on known inputs, shapes and types, preprocessing consistency, and a tiny end-to-end training run.
14.Behavioral-tests
What is a behavioral test for a model?
A check that the model responds as expected to designed inputs, such as staying stable when an irrelevant detail changes.
15.Validation-gate
What is a validation gate?
An automated check a candidate model must pass, such as beating the current model without slice regressions, before promotion.
16.Slice-eval
Why evaluate slices before launch?
A model can improve on average while getting much worse for one group of users or inputs.
17.Fairness
What is a fairness check?
Comparing error rates and outcomes across groups to find harmful differences before launch.
18.Model-card
What is a model card?
A short document covering a model's intended use, training data, evaluation results, limits, and owners.
19.Skew
What is training-serving skew?
A difference between how features or inputs are produced in training and in production.
20.Shared-features
How do you prevent training-serving skew?
Compute features with one shared implementation or a feature store for both, and log served features to compare.
21.Pit-join
What is a point-in-time correct join?
Using only feature values known at each example's time when building training data.
22.Golden-set
What is a golden set for deployment checks?
Fixed inputs with expected outputs, run against each deployed endpoint to confirm it matches the tested model.
23.Shadow
What is a shadow deployment?
The new model receives copies of live requests and logs its outputs, but users only see the current model's results.
24.Canary
What is a canary release?
Sending a small share of traffic to the new model, watching key signals, and widening only if they hold.
25.Blue-green
What is a blue-green deployment?
Running old and new environments side by side and switching traffic between them, so rollback is a switch.
26.Ab-test
What does an A/B test add after a canary?
A properly sized comparison of user-facing metrics, to decide whether the new model is actually better.
27.Rollback
What makes a fast rollback possible?
Keeping the previous model, its features, and its config deployable, and switching with a flag or config change.
28.Feature-flag
What is a feature flag?
A setting that turns behavior on or off, or for some users, without a code deploy.
29.Rollback-criteria
Why write rollback criteria before launch?
Agreed thresholds make the decision quick and objective during an incident.
30.Fallback
What is a fallback for a model service?
A safe default used when the model fails or is too slow, such as rules, a cached answer, or popular items.
31.Timeout
Why set timeouts on model calls?
So a slow model cannot tie up callers and slow the whole product.
32.Retry-backoff
What is exponential backoff with jitter?
Waiting longer after each failed retry, plus a random delay, so retries do not pile onto an overloaded service at once.
33.Circuit-breaker
What is a circuit breaker?
A switch that stops calling a failing dependency for a while and uses a fallback, then tries again later.
34.P99
Why measure p95 and p99 latency?
The slowest requests are what users notice, and averages hide them.
35.Slo
What is a service level objective (SLO)?
A target for a measurable service quality, such as 99.9 percent of requests succeeding under 300 ms each month.
36.Error-budget
What is an error budget?
The amount of failure an SLO allows. At 99.9 percent over 30 days, about 43 minutes.
37.Load-test
What does a load test show?
How latency and errors change as traffic rises, and where the service breaks.
38.Capacity
How do you size a serving fleet roughly?
Peak requests per second divided by requests per second per server, plus headroom for spikes and failures.
39.Health-checks
What are liveness and readiness checks?
Liveness says the process is running. Readiness says the model is loaded and can take traffic.
40.Data-drift
What is data drift?
A change in the distribution of model inputs compared with training data.
41.Prediction-drift
What is prediction drift?
A change in the distribution of model outputs, such as the share of cases flagged.
42.Concept-drift
What is concept drift?
A change in the relationship between inputs and outcomes, so accuracy falls even when inputs look the same.
43.Psi
What is the population stability index (PSI)?
A measure of how much a variable's binned distribution has shifted from a reference period.
44.Label-delay
How do you monitor a model when labels arrive late?
Watch input and prediction drift and early proxy signals, then confirm with real labels as they arrive.
45.Monitoring-layers
What three layers should model monitoring cover?
Service health (latency, errors), data health (inputs, drift), and model quality (accuracy, business metrics).
46.Alert-fatigue
What is alert fatigue?
Too many low-value alerts, so real problems get ignored. Alerts should be tied to meaningful impact.
47.Retraining
When should a model be retrained?
On a schedule matched to how fast its data changes, or when monitoring shows decay, and always through the same release checks.
48.Runbook
What is a runbook?
Written steps for responding to known alerts and failures, including how to roll back and whom to contact.
49.Ownership
Why does every production model need an owner?
Data, dependencies, and the world change. Someone must notice decay, approve retrains, and handle incidents.
50.Postmortem
What is a blameless postmortem?
A review after an incident that finds what allowed it and lists concrete fixes, without assigning personal blame.