ML pipelines

What do you settle before you choose tools for the pipeline?

The decision the model supports, how fresh the answer must be, the latency and cost limits, and what a wrong or missing answer does to the user.

What makes a handoff between stages safe?

A named artifact, a version, and a check the next stage can refuse. A training job should be able to reject a feature table that fails its checks.

Why name an owner for each stage?

Data, features, and the model change on different clocks. Someone has to answer when a stage is late, wrong, or too expensive.

What do you list as sources before you design ingestion?

The systems that produce the raw events or tables, how often they update, who owns them, and which fields the model is allowed to use.

When do you ingest on a schedule, and when as events arrive?

A schedule fits data that can be hours old. A stream fits a decision that goes stale if it waits for the next batch. The product's freshness requirement picks one.

What is the landing copy of raw data for?

An unchanged record of what arrived. A later bug in cleaning can be fixed by replaying that copy, without asking the source to send the data again.

What belongs in the cleaning step after landing?

Parsing, type checks, deduplication, and quarantine for rows that break the contract. Training reads the cleaned output.

What is the trade-off with events that arrive after the window closed?

Waiting longer makes the table more complete and the data older. You set a cutoff, keep what misses it, and decide whether a later run may revise that window.

What does an ingestion contract fix with the source?

Column names, types, units, allowed values, and how late data may be. A renamed field ships as a new version the pipeline accepts on purpose.

Why must replaying a source file leave the table unchanged?

Jobs retry. A load that appends the same rows again doubles counts and shifts every feature built on those counts.

How do you choose how fresh ingested data must be?

From the decision. A nightly price model can train on yesterday. A check that runs during checkout needs features from the last few minutes.

What is a feature in this pipeline?

A value computed from raw data that the model sees at training and again at serving. The definition includes the window, the entity key, and the time it was known.

Where can a feature be computed?

In a batch job ahead of time, in the request path, or in both from one definition. The choice turns on freshness, cost, and whether training and serving share the code.

Why use one feature definition for training and serving?

Two copies drift apart. A model trained on a 7-day count will mis-score if serving computes a 1-day count and stores it under the same name.

What happens if a training row uses a feature from after its label time?

The model learns from information it will not have when it scores a new row. The training metric looks strong, and live quality is weaker.

What is label leakage?

A feature that contains the answer, or a close stand-in written only after the outcome is known. The model is memorizing a column it will not see live.

When is a shared feature store worth more than a one-off feature job?

When several models need the same values, or training and serving must match exactly. One model with slow features can compute them inside its own training job.

What is the trade-off in a feature's time window?

A longer window is steadier and slower to notice a change. A shorter window reacts faster, is noisier, and costs more to refresh.

A feature is often missing when a request arrives. What do you decide?

A default the model was trained with, a coarser feature that is usually present, or a path that skips the model. A default the training rows never saw is a new input.

What keys does a feature lookup need?

The entity it describes and the time it was known. Serving looks up by entity. Training looks up by entity and by a time no later than the example.

What do you test about a feature before training depends on it?

Known inputs produce the expected value, missing inputs follow the documented rule, and the same entity and time always return the same value.

What does the training stage take in and put out?

It takes a versioned feature table and labels. It puts out a model artifact, metrics on a held-out set, and the record of code, data, and settings.

How do you split examples that happen over time?

Train on earlier examples and evaluate on later ones. A random split lets the model see the future of the same entity and overstates how it will do live.

What is the trade-off between training on a schedule and training when a signal fires?

A schedule is predictable and can spend compute when nothing changed. A signal spends compute when the world moved, and it needs a signal you trust.

When do you train from scratch, and when do you continue from the current weights?

From scratch when the data or the features changed shape. Continue when the same task has new examples and a faster, cheaper update is enough.

What does a candidate have to clear before anyone serves it?

It beats the live model on the agreed metric, it holds up on the slices you named, and its latency and size fit the serving budget.

Why is the average score a weak reason to promote a model?

The average can rise while a slice you serve every day gets worse. The gate names those slices and the drop you will refuse.

What do you compare a new model with?

The model in production, and a simple rule or the previous training run. A gain that vanishes against the rule came from the features.

What do you keep so a training run can be explained months later?

The data version, the feature definitions, the code revision, the settings, the metrics, and the artifact id. Serving loads that artifact.

What do you spend to train more often?

Compute, label wait, and review time. More frequent models track change sooner and leave less time to notice a bad one before it is served.

Where do labels come from, and why do they arrive late?

From a later outcome, a person, or a stronger system. The outcome often arrives after the decision, so the training set trails the requests you are scoring now.

When do you score inside the user request, and when in a job?

Inside the request when the user is waiting on that answer. In a job when the entities are known ahead of time and the score can be stored for lookup.

How does serving spend its time budget?

Fetch the features, run the model, and apply any rules after it. Each step gets a share of the latency the product allows, including the slow tail.

The feature lookup is slow or down. What does the request do?

Use the fallback the model was trained to accept, return a cached score, or skip the model for the product's safe default. Waiting out the timeout stalls the user.

What is the safe answer when the model cannot be used?

The behavior the product already trusts, such as the previous rule, a popular item, or no automated action. That path has to be acceptable on its own.

How do you expose a new model to users?

Start with a copy of live traffic that users do not see, then a small share of real users, and widen only while the signals you agreed on still hold.

What is pinned when a model version is served?

The artifact, the feature definitions and their versions, the code that calls them, and the settings. Rolling back serves the previous pinned set.

When is scoring ahead of time cheaper than scoring each request?

When the set of entities is known and changes slowly. You pay for a batch job and a lookup, and you wait until the next job to score an entity that just appeared.

What decides how many serving copies you run?

Peak requests, how many each copy can finish inside the latency budget, and spare capacity for a copy dying or a traffic spike.

What do you watch once the model is live?

Whether the service answers in time, whether inputs still resemble training, and whether outcomes are still good once labels arrive.

How do you tell that live inputs have moved away from training?

Compare each important feature's distribution, and its rate of missing values, with the data the served model was trained on.

What can you watch about quality before labels arrive?

The spread of scores, the rate of each decision, and early product signals such as accepts. Those move before the true outcomes are recorded.

When can you measure the model's real quality?

When the outcome it predicted has been recorded and joined back to the scored request, using the features that were actually served.

What deserves a page, and what stays on a dashboard?

Page on a broken service, a feature job that stopped, or a quality drop past the rollback line. A slow drift stays on a chart someone reviews on a schedule.

Which signal should start another training run?

A real drop in outcome quality, or a scheduled refresh you already trust. A small shift in one input is a reason to look before it is a reason to train.

Which signal should take the new model back out?

A breach of the latency or error target, or a quality drop past the line written down before launch. You return to the previous pinned version.

How do live outcomes become the next training set?

Scored requests are stored with the features that were served, joined to outcomes when those arrive, and that table is an input to the next training run.

What is the smallest pipeline that still closes the loop?

A scheduled load, one shared feature job, a training run with a held-out check, a way to serve and roll back one model, and a check that inputs and outcomes are still arriving.

What do you say you are leaving for later?

The stages you skipped on purpose, such as streaming features or automatic retraining, and the signal that would make you add them. A pipeline you can operate beats a complete diagram you cannot run.