Model monitoring

What is data drift?

A change in the distribution of model inputs compared with a reference, such as training data or a recent stable period.

What is concept drift?

A change in the relationship between inputs and the outcome, so the same inputs now mean something different.

What is label shift, or prior shift?

A change in how often each outcome occurs, such as the fraud rate doubling, while inputs given the outcome stay similar.

What is the population stability index (PSI)?

A drift score that bins a feature, compares the share of rows in each bin between reference and current data, and sums the weighted log ratios.

What PSI values are commonly treated as small and large shifts?

Below 0.1 is usually small, 0.1 to 0.25 moderate, and above 0.25 large. These are rules of thumb, not tests.

What does the Kolmogorov-Smirnov test compare?

The largest gap between two cumulative distributions of a numeric feature.

Which test is common for drift in a categorical feature?

The chi-square test on category counts, or PSI over categories.

Why do drift tests flag almost everything at high volume?

With millions of rows, tiny harmless differences are statistically significant. Effect-size measures such as PSI are more useful.

What is a reference window?

The period of data that current traffic is compared against, such as the training snapshot or the same weekday last week.

Inputs follow the day, so every morning looks like drift against the previous hour. What fixes that?

Compare with the same hour on a matching earlier day, so normal daily patterns cancel out.

A region has little overnight traffic and its PSI swings wildly. What helps?

Require a minimum row count before scoring, or widen the window for that region, so small samples do not page.

How should expected events such as holidays be handled?

Annotate or suppress them in advance, or compare with the same event last year, so known shifts do not page.

What do schema checks on live inputs catch?

Missing fields, wrong types, renamed columns, and values in the wrong unit, often before they reach the model.

Why monitor the null rate per feature?

A sudden rise in missing values usually means an upstream break, and the model may treat nulls badly.

What is an unseen-category alert?

An alert when a categorical feature takes a value the model never saw in training.

A new category was registered in the data contract before deploy. Why should its alert not page?

The alert should read the contract, so known, approved values are expected, and only unregistered ones page.

What is a range check?

Flagging values outside the bounds seen in training or allowed by the contract, such as negative prices.

What is feature freshness?

How old the newest value of a feature is at serving time. Stale features make the model act on outdated information.

What is training-serving skew?

A feature computed differently at serving than in training, so the model sees inputs unlike those it learned from.

How is training-serving skew detected?

Log the features served, recompute them with the training code for the same rows, and compare the values.

Why monitor the distribution of model scores?

Score shifts appear immediately, long before labels arrive, and often reflect input or pipeline problems.

What is label delay in monitoring?

The wait before true outcomes are known, such as 21 days for chargebacks, during which accuracy cannot be measured directly.

What can stand in for accuracy before labels arrive?

Score distribution, input drift, early partial labels, and business signals. They are warnings, not measures of accuracy.

What is a score-shift proxy not?

A measure of correctness. Scores can move for good reasons, and a model can be wrong without its scores moving.

What is measured once labels arrive?

The real metrics, such as precision, recall, AUC, and calibration, by time window and segment, joined back to the predictions.

Why log a prediction ID with every score?

So labels arriving later can be joined to the exact prediction, model version, and features that produced it.

Why monitor calibration?

Downstream rules often rely on the probability itself. A model can rank well while its probabilities drift off.

Why monitor by segment?

A problem in one region or product can be invisible in the overall average.

What is shadow deployment?

Running a new model on live traffic and logging its predictions without using them to make decisions.

Why must the shadow call be off the request path?

If the user waits on the shadow model, its latency and failures affect a model you are not even serving.

What is a canary release for a model?

Sending a small share of real traffic to the new model, watching its metrics, then widening or rolling back.

What is a champion-challenger setup?

The live model (champion) is compared continuously against candidates (challengers) on the same traffic.

How does an A/B test differ from shadow?

In an A/B test both models make real decisions for random groups, so their effect on outcomes can be measured.

What are the four golden signals?

Latency, traffic, errors, and saturation.

Why watch latency percentiles, not the mean?

The mean hides slow requests. p95 and p99 show what the slowest users experience.

p99 pages on every deploy because a new instance's first requests are slow. What helps?

Warm instances before they take traffic, or exclude a short warm-up window, so a known startup tail does not page.

What are an SLI and an SLO?

An SLI is a measured indicator, such as the share of requests under 200 ms. An SLO is the target for it, such as 99.5 percent.

What is an error budget?

The amount of failure an SLO allows over a period. Spending it fast is a reason to slow releases.

What is a burn-rate alert?

An alert when the error budget is being used up faster than planned, often checked over a short and a long window together.

What should page someone, and what should just open a ticket?

Page only for user-facing harm that needs action now. Slow drift and minor checks go to a ticket or dashboard.

Why alert on symptoms rather than causes?

Symptoms such as wrong or slow responses are what users feel. Cause-level alerts are noisy and miss unknown causes.

A dashboard query times out and its tile goes red, but serving, freshness, and score checks are in bounds. What is broken?

The dashboard, not the model. Fix the query, and do not page on a monitoring tool's own failure as if it were the model's.

What is alert fatigue?

So many low-value alerts that people stop reacting to them, including the real ones.

What should a runbook for a model alert contain?

What the alert means, how to check it, likely causes, how to roll back, and who to contact.

Why mark deploys on monitoring charts?

So a change in errors or scores can be lined up with the deploy, model ship, or data change that caused it.

What makes a model rollback fast?

Keeping the previous model version, its preprocessing, and its config ready to serve, and switching with one pointer change.

Why can a model's own decisions distort its monitoring?

If it blocks transactions, outcomes for blocked ones are never seen, so labels only cover what it let through.

How is drift monitored for text or image inputs?

Track embedding distributions, input length, language mix, and cluster shares, since raw values cannot be binned easily.

What extra signals are monitored for an LLM feature?

Refusal rate, output length, format errors, safety-filter hits, user feedback, token usage, and cost.