Card 1 of 50
Front. Decision. Why start an applied project from the decision the model supports?
These cards are original study material. They are not questions from a named employer or a live hiring loop.
All 50 cards in this deck
1.Decision
Why start an applied project from the decision the model supports?
The decision sets the target, the timing, the metric, and the cost of each error. Without it, accuracy has no meaning.
2.Target
What is a target definition?
An exact statement of what is predicted, for which unit, over which window, such as 'card is charged back within 60 days of purchase'.
3.Unit
What is the unit of analysis?
The thing one row represents, such as a customer, order, or customer-week. Splits, metrics, and claims must all use it consistently.
4.Prediction-time
What is the prediction time?
The moment the model must make its prediction. Only information available before it may be used as features.
5.Feature-cutoff
What is a feature cutoff?
The timestamp after which no data may enter the features for a row, normally the prediction time.
6.Post-outcome
What goes wrong if a feature was written after the outcome was known?
It leaks the label. The model looks strong offline but that information will not exist when it runs live.
7.Label-delay
What is label delay?
The time between a prediction and when its true outcome is known, such as weeks for chargebacks. Recent rows may not have labels yet.
8.Lock-first
What should be written down before the test data is scored?
The target, cutoff, splits, exclusion rules, primary metric, threshold rule, and success criterion.
9.Exclusions
Why fix exclusion rules before scoring?
Dropping rows after seeing results can remove the cases the model fails on and inflate the score.
10.Backtest
What is a time-based backtest?
Train on data up to a date, test on the period after it, and repeat for several dates to see how results vary over time.
11.Random-split-risk
Why can a random split overstate performance?
Near-duplicate rows from the same customer or week land on both sides, and future patterns leak into training.
12.Same-period
Why compare old and new models on the same period's data?
Comparing last year's model on last year's data with a new model on this year's data mixes the model change with the data change.
13.Baseline-current
What is the most important baseline in an applied project?
The current process, such as existing rules or human judgment, measured on the same data.
14.Baseline-simple
Why also include a simple baseline?
A rule or logistic regression shows how much the complex model actually adds.
15.Base-rate
What is the base rate?
How often the positive class occurs. It sets what a naive model can score and how precision behaves.
16.Accuracy-paradox
Failures are 2 percent of rows and a model reports 97 percent accuracy. Why is that weak?
Predicting 'no failure' for every row scores 98 percent. Accuracy says nothing about catching the failures.
17.Precision
What does precision tell a team acting on alerts?
Of the cases flagged, what fraction were truly positive, which sets how much review work is wasted.
18.Recall
What does recall tell a team acting on alerts?
Of the true positives, what fraction were flagged, which sets how many cases are missed.
19.Pr-auc
Why use PR AUC over ROC AUC for rare events?
ROC AUC can look high even when precision is poor, because true negatives dominate. PR AUC focuses on the positive class.
20.Cost-errors
Why weigh errors by cost?
A missed fraud and a false alarm rarely cost the same. The threshold should reflect the real costs.
21.Threshold-val
Where should the decision threshold be chosen?
On validation data, by a rule fixed in advance, then applied unchanged to the test set.
22.Capacity
How does team capacity set the threshold?
If reviewers can handle 200 cases a day, the threshold may be the score that flags about 200, and precision at that point is the key number.
23.Calibration
What is calibration?
Whether predicted probabilities match observed rates: of cases scored 0.3, about 30 percent should be positive.
24.Lift
What is lift?
How much more often positives occur in the model's top-scored group than in the population overall.
25.Ci
Why give a confidence interval with the test metric?
A test set is a sample. The interval shows how much the number could move with another sample.
26.Segments
Why report results by segment?
An overall gain can hide a loss in an important group, such as new customers or one region.
27.Simpson
What is Simpson's paradox?
A trend that appears in every subgroup can reverse when the groups are combined, because the group mix differs.
28.Selection-bias
What is selection bias in training data?
The labeled rows are not representative, for example only cases that a past rule sent for review.
29.Feedback-loop
What is a feedback loop?
The model's decisions change which outcomes are observed, such as only approved loans getting repayment labels.
30.Survivorship
What is survivorship bias?
Studying only cases that made it through some filter, such as customers who stayed, and missing those who left.
31.Label-noise
What is label noise?
Wrong or inconsistent labels in the data. It caps achievable accuracy and can bias the model.
32.Guidelines
Why write labeling guidelines before annotation?
So different annotators label the same case the same way, and the labels mean what the target says.
33.Label-influence
What goes wrong if annotators see the model's score before labeling?
The labels lean toward the model, so agreement with the model is inflated.
34.Missing
Why check how missing values arise?
Missingness can depend on the outcome, such as a field filled only after a problem, which leaks information.
35.Shift
What is distribution shift?
Live inputs or outcome rates differ from the training data, so offline results may not hold.
36.Seasonality
Why does seasonality matter for a test period?
A test window in a quiet season may not show performance during peaks or holidays.
37.Offline-online
Why can offline gains fail to appear online?
Offline metrics are proxies. Live users, changed behavior, delays, and shift can all differ from the test set.
38.Ab-test
What does an A/B test add over an offline evaluation?
Randomly assigning users or units to old or new versions measures the real effect on the outcome that matters.
39.Holdout-group
What is a holdout group?
A random set of units kept on the old process, so the new process's effect can be measured against it.
40.Guardrail
What is a guardrail metric?
A metric that must not get worse, such as complaints or latency, even if the primary metric improves.
41.Proxy-claim
A tagger agrees better with a manual on test data. Can the team claim staff now act faster?
No. The result shows agreement with the manual. A claim about staff speed needs a measurement of staff speed.
42.Correlation
Why can't an observational comparison prove the model caused a gain?
Other things may differ between the groups, such as timing or who chose to use it. Only randomization rules them out.
43.Novelty
What is a novelty effect?
Users react to something new at first, so a short test can overstate or understate the lasting effect.
44.Sample-size
Why plan the sample size before a test?
To make sure the test can detect an effect of the size that would matter, and to avoid stopping early on noise.
45.Sensitivity
What is a sensitivity analysis?
Rerunning the analysis with different reasonable choices, such as cutoffs or exclusions, to see whether the conclusion holds.
46.Never-checked
Why list what the project never checked?
So readers know the limits: untested segments, periods, or outcomes where the result may not apply.
47.Success-criterion
What is a success criterion?
A threshold set before the work, such as 'recall at least 0.6 at 200 alerts a day', that decides whether the model ships.
48.Model-card
What belongs in a model card?
Intended use, data and time range, metrics by segment, known limits, and cases where the model should not be used.
49.Reproduce
What makes an applied analysis reproducible?
Versioned code, a fixed data snapshot or query with a date, recorded seeds, and the exact config that produced the numbers.
50.Monitor-plan
Why plan monitoring before launch?
Labels may arrive late, so the team needs to know in advance which early signals will show the model is going wrong.