Predictive modeling

Why separate the feature window from the label window?

Features summarize information available before prediction; labels describe the subsequent outcome. Overlapping them can let the model observe evidence of the answer, producing an evaluation that cannot be reproduced at serving time.

Why exclude or censor recent examples whose outcome window has not finished?

Their labels are incomplete. Calling an unresolved 30-day outcome negative after only three days biases training and evaluation; wait for maturity or use a method explicitly designed for censoring.

A churn model uses account-closure reason. What is the failure?

The field is normally recorded after the event being predicted. It leaks the target and can create excellent offline scores with no actionable lead time; inspect feature availability, not just column timestamps.

Why split by customer rather than row when customers have repeated records?

Rows from the same person can share identifiers and stable patterns, letting the model recognize test entities. A group split estimates performance on new entities; a time split may be needed for future predictions on existing ones.

When is a random train/test split unrealistic?

When deployment predicts future data affected by temporal drift, evolving products, or delayed features. Train on earlier information and evaluate later periods, matching the production retraining and prediction cadence.

What does a gap between training and validation periods protect against?

Overlapping feature/label windows or delayed labels can share information across the boundary. Choose the gap from the data-generation process; an arbitrary gap does not cure features that already leak future information.

Where should imputation and scaling be fitted during cross-validation?

On each fold's training portion, then applied unchanged to its validation portion. A pipeline keeps this boundary explicit; fitting preprocessing on the full dataset lets validation information influence training.

How can target encoding leak labels inside the training data?

Encoding a category with its own row's target or a full-data target mean makes the feature partially reveal the answer. Use out-of-fold or ordered training encodings and fit deployment mappings only from permitted training labels.

What is the difference between validation and a final test set?

Validation supports model and threshold choices; the held-out test estimates performance after those choices are fixed. Repeatedly choosing models from test results turns that set into another validation set.

What problem does nested cross-validation address?

An inner loop selects hyperparameters and an outer loop evaluates the complete selection procedure on unseen folds. It reduces selection optimism but still needs realistic grouping and temporal boundaries.

What makes a useful baseline stronger than predicting the majority class?

It reflects the decision context: the current production rule, a simple interpretable model, or a seasonal forecast. Compare on the same data, costs, and constraints so added complexity has a measurable benefit.

Training error is low but validation error is high. What does a learning curve help diagnose?

It shows whether the gap shrinks with more training data and whether either curve has plateaued. More data may help variance, while leakage, distribution mismatch, or a poor target requires different fixes.

Why can regularization improve held-out performance while worsening training fit?

It constrains the fitted function, trading some training accuracy for reduced sensitivity to sample noise. Choose its strength with valid validation rather than assuming the lowest training loss is best.

Which models commonly need attention to feature scaling?

Distance-based methods and regularized linear models are sensitive to relative scales; gradient-based optimization can also benefit. Tree split decisions are generally invariant to monotonic rescaling, though preprocessing can still affect missing values and numerical behavior.

When is a linear model a credible first candidate?

When additive relationships or a transformed linear approximation are plausible and low latency or interpretability matters. Examine residuals, interactions, and regularization; simple architecture does not guarantee a correct model.

What does logistic regression model directly?

The log-odds of a binary outcome as a linear function of predictors. The sigmoid maps this to a probability, but calibration depends on model fit, sampling, and regularization rather than the name of the algorithm.

Why can unregularized logistic coefficients diverge on separable data?

The likelihood can keep improving as coefficient magnitudes grow without a finite optimum. Regularization or appropriate bias-reduction methods can stabilize estimation; first check for leakage or data artifacts causing separation.

Why can a deep decision tree be unstable?

Small changes in data can alter early splits and downstream structure. Constrain depth or leaf size and validate; averaging trees reduces some variance but does not fix a leaked feature or unrepresentative split.

How do random forests reduce correlation among trees?

They train on resampled data and consider random subsets of features at splits. Averaging less-correlated trees reduces variance; it does not make trees independent or guarantee extrapolation outside observed ranges.

What does gradient boosting add at each stage?

A new weak learner approximates a direction that reduces the chosen loss, often through negative gradients. Learning rate, number of trees, depth, and sampling interact, so tune them as a system rather than independently maximizing each.

Why use a held-out set for early stopping in boosting?

It limits training when further stages stop improving the chosen validation objective. Keep the final test separate and remember that monitoring validation repeatedly is part of model selection.

Why is 99% accuracy uninformative when positives occur in 1% of cases?

Always predicting negative already reaches 99%. Inspect precision, recall, decision cost, ranking, and calibration at a realistic prevalence and operating threshold.

A model predicts 100 positives, of which 30 are real, and there are 60 real positives total. What are precision and recall?

Precision is 30/100 = 30%; recall is 30/60 = 50%. The first describes the reliability of positive alerts, the second their coverage of actual positives.

Why can precision fall after deployment even if sensitivity and specificity stay fixed?

A lower positive base rate makes false positives a larger share of alerts. Re-evaluate expected precision and costs at the deployment prevalence instead of copying precision from a balanced test sample.

What does ROC AUC measure for a binary score?

The probability a randomly chosen positive receives a higher score than a randomly chosen negative, with ties handled by convention. It summarizes ranking, not calibration or usefulness at the operating threshold.

Why distinguish average precision from trapezoidal PR AUC?

Average precision weights precision values by recall increments, while trapezoidal integration interpolates between plotted points. They can differ; report the exact metric implementation rather than labeling both simply PR AUC.

Why does log loss penalize confident errors strongly?

It uses the negative log probability assigned to the observed class. Assigning nearly zero probability to the true outcome gives a large loss; implementation clipping prevents infinities but does not make confident mistakes harmless.

What does the Brier score measure?

The mean squared difference between predicted probabilities and binary outcomes. It evaluates probabilistic forecasts, reflecting calibration and resolution together; a lower Brier score alone does not isolate calibration improvement.

What does a reliability diagram compare?

Average predicted probability with observed event frequency within probability bins or smoothing regions. Include sample counts and uncertainty; sparse bins can look miscalibrated by chance and coarse bins can hide errors.

Why must a probability calibrator use predictions not fitted on the same labels?

Training-set predictions may be overconfident or overly accurate, giving an unrealistic mapping. Use held-out or cross-validated predictions and keep a separate final evaluation of the full calibrated system.

Why is 0.5 not a universal classification threshold?

The appropriate threshold depends on false-positive and false-negative costs, prevalence, capacity, and probability calibration. Select it against the actual decision objective on permitted validation data.

With calibrated probabilities and costs C_FP and C_FN, what simple threshold minimizes expected misclassification cost?

Predict positive when p > C_FP / (C_FP + C_FN), assuming only those two error costs and zero cost for correct decisions. Capacity constraints or other utilities require a different decision rule.

A review team can inspect 200 cases daily. What evaluation matches that constraint?

Measure precision and recovered value at the top 200, along with missed harm and stability across days. A fixed probability threshold can produce more alerts than the team can handle when traffic changes.

How can class weighting change probability interpretation?

It changes the effective loss and often the population the fitted scores represent. Improved minority recall does not imply calibrated deployment probabilities; evaluate calibration and any prior correction under explicit assumptions.

Where should oversampling or synthetic minority examples be created?

Inside the training fold only. Resampling before splitting can place duplicated or related synthetic points in validation, inflating scores; retain the natural evaluation prevalence unless a specific weighted estimand is intended.

How do MAE and RMSE emphasize different regression errors?

MAE scales linearly with absolute error; RMSE penalizes large errors quadratically before taking the root. Choose based on the consequence of misses, and inspect tail errors rather than treating either metric as universally better.

Why can MAPE be unsuitable for zero or near-zero outcomes?

It divides by the observed magnitude, making zero values undefined and tiny values disproportionately influential. Consider scale-appropriate absolute, squared, or scaled errors with a clearly stated business interpretation.

What does quantile regression estimate?

A conditional outcome quantile rather than the conditional mean, typically by minimizing pinball loss. A 90th-percentile prediction targets an upper conditional boundary; check empirical coverage by segment and distribution shift.

What is a seasonal naive forecast?

It predicts using the most recent value from the same seasonal position, such as the same weekday last week. It is a strong benchmark when seasonality dominates and should be evaluated on the same horizons as the candidate.

How does rolling-origin evaluation mimic forecasting deployment?

Repeatedly train using only information available at an origin and score future horizons, then move the origin forward. Report performance by horizon and account for overlapping forecast errors when quantifying uncertainty.

Why can impurity-based tree importance favor some features?

Features with many possible splits can gain importance through more opportunities to fit noise. Correlated features also share or substitute signal; compare with held-out permutation analysis and domain checks.

What does permutation importance measure, and what complicates it?

It measures performance degradation when a feature's values are shuffled under a specified evaluation setup. Correlated features can mask importance or create unrealistic combinations; it is model reliance, not a causal effect.

Why should SHAP values not be interpreted as intervention effects?

They allocate a model prediction relative to a background distribution and attribution convention. Changing a feature in the real world can alter other variables; attribution does not identify those causal consequences.

What makes a useful error analysis slice?

A meaningful deployment group or failure condition with adequate support, such as new users, missing inputs, or long-tail categories. Report counts and uncertainty and distinguish exploratory slice discovery from confirmed evaluation.

How do epistemic and aleatoric uncertainty differ?

Epistemic uncertainty concerns limited knowledge of the fitted relationship; aleatoric uncertainty concerns outcome variability conditional on modeled inputs. The distinction depends on the model and available features, and one scalar confidence score rarely separates them cleanly.

What does standard split conformal prediction guarantee under exchangeability?

Finite-sample marginal coverage at the selected level for new examples exchangeable with calibration data, subject to the method's construction. It does not generally guarantee coverage for every subgroup or survive arbitrary temporal shift.

When can allowing a model to abstain be useful?

When uncertain or unsupported cases can be routed to a safer fallback or human review. Evaluate coverage, error among accepted cases, review capacity, and failures of the confidence signal itself.

A model's offline score is strong but production predictions differ on the same records. What do you compare?

Feature values and availability, preprocessing artifacts, category mappings, model revision, inference precision, and decision thresholds. Replay a small versioned example through both paths to isolate the first divergence.

What evidence supports replacing a deployed predictive model?

A realistic held-out comparison, task-relevant quality and calibration, important slice checks, serving compatibility, cost and latency, and a monitored rollout with recovery. A single leaderboard metric does not cover the deployment decision.