Training pipelines

What is a DAG in pipeline orchestration?

A directed acyclic graph of tasks, where edges say which tasks must finish before others start.

What does an orchestrator such as Airflow, Dagster, or Kubeflow do?

Schedules tasks, runs them in dependency order, retries failures, and records each run's status and logs.

The scheduler paints the whole DAG red after one task fails. Where do you look first?

At the first task that failed and its own log. Downstream tasks are usually red only because they never ran.

What is an idempotent pipeline task?

One that gives the same result if run again with the same inputs, for example by overwriting its output partition instead of appending.

Why design tasks so a run can restart from the failed step?

Rerunning hours of earlier work wastes compute and can pick up different data if sources have changed.

Which pipeline failures should be retried automatically?

Transient ones, such as network timeouts or preempted machines. Code bugs and bad data will just fail again.

What can trigger a retraining run?

A schedule, new data arriving, a drift alert, or a manual request, each recorded with the run.

What is a data snapshot?

A fixed, versioned copy or view of the training data at a point in time, which does not change after it is taken.

Two tables are read from snapshots taken at different times. What is the risk?

They describe different moments, so joins can mix states, such as orders that reference shoppers not yet in the other table.

What is a point-in-time correct join?

For each training row, using only feature values that existed at that row's timestamp, never later ones.

What is a data contract?

An agreement between data producer and consumer on schema, types, units, allowed values, and freshness, checked automatically.

A column keeps its name while its unit changes from cents to dollars. What failed?

The contract. A change in meaning needs a new column or version, because unchanged names hide the break from every consumer.

What is a backward-compatible schema change?

One that old consumers can still read, such as adding an optional column. Renaming or retyping a column is not.

What does a data validation step check before training?

Schema, row counts, null rates, value ranges, label balance, and duplicates, failing the run when they break expectations.

What is a feature store?

A system that defines features once and serves them both offline for training and online for low-latency inference.

What are the offline and online stores?

Offline holds full feature history for building training sets. Online holds the latest values for serving.

Why version feature definitions?

A model depends on the exact computation used in training. Changing the logic under the same version silently changes the model's inputs.

You revert a bad feature commit, but the online store still holds the bad values. What else is needed?

A backfill or recompute of the affected values. Reverting code does not fix data that was already written.

What is a backfill?

Recomputing a feature or table for past dates, usually after a fix or a new definition.

Why should training and serving share feature code?

Two separate implementations drift apart and cause training-serving skew.

Training standardizes with the snapshot's mean, but serving uses a constant from an old notebook. What is wrong?

Serving does not match training. The fitted statistics must be saved with the model and loaded at serving.

Why is the preprocessor part of the model artifact?

Scalers, encoders, and vocabularies fitted on training data are part of the model. Weights without them cannot reproduce its outputs.

Serving runs model v18 weights with the v19 preprocessor. What is the problem?

The inputs are transformed differently than v18 was trained on, so its outputs are unreliable even if nothing errors.

A vocab file next to the weights is overwritten on every train. What happens on rollback?

The old weights load with the new vocabulary. Every artifact needed for a version must be stored with that version, never overwritten.

Why should model artifacts be immutable?

So a version always means the same files, and rollbacks and audits are trustworthy.

What is content-addressed storage for artifacts?

Naming each file by a hash of its contents, so identical files are stored once and any change gets a new name.

What is a model registry?

A catalog of model versions with their artifacts, metrics, lineage, and stage, such as staging or production.

Serving points at a model version in the registry. What does moving that pointer change?

Only which version is served. Anything not bundled with the version, such as a shared preprocessor in the image, stays as it was.

What is model lineage?

The record of exactly which data snapshot, code commit, config, and environment produced a model version.

What must be pinned to reproduce a training run?

Code commit, data snapshot, config and hyperparameters, random seeds, and the container image with locked dependencies.

Why pin a container image by digest rather than tag?

A tag such as latest can be moved to a new image. A digest always refers to the same bytes.

What does a dependency lockfile do?

Records exact versions of every library, so rebuilding the environment later gives the same packages.

Why can two runs with the same seed still differ slightly?

Nondeterministic GPU kernels, parallel data loading, and floating-point order can vary. Strict determinism needs specific settings.

What does an experiment tracker record?

Each run's parameters, metrics over time, artifacts, code version, and environment, so runs can be compared.

Training runs out of memory after a batch-size change, on the same machine and data. What is the likely cause?

The batch-size change itself. Activation memory grows with batch size, so compare the configs of the failing and last green runs.

How do you find what changed between a green run and a failing one?

Compare their code commits, configs, data snapshots, images, and machines from the recorded lineage.

How do pipelines cope with preemptible machines?

Save training checkpoints regularly and resume from the latest one when a new machine starts.

What is an evaluation gate?

A pipeline step that blocks promotion unless the new model meets set thresholds and does not regress against the current model.

What is a golden evaluation set?

A fixed, versioned set of examples used to compare every candidate model the same way.

Why must the golden set stay fixed across promotions?

If it is rebuilt each night, score changes mix model changes with data changes, and comparisons lose meaning.

Why evaluate on slices in the gate?

An overall gain can hide a regression in an important segment, which the gate should catch.

What is the risk of shipping every green training run automatically?

Green only means the pipeline ran and passed its checks. Problems the checks miss go straight to production.

What usually happens between registering a model and full production?

Gate checks, shadow or canary traffic, monitoring, and sometimes human approval, before the pointer moves for all traffic.

What does a fast model rollback need?

The previous version's full artifact bundle still stored and loadable, and a one-step pointer change in serving.

What tests belong in CI for a training pipeline?

Unit tests for feature code, schema tests, a small end-to-end run on sample data, and checks that artifacts load in serving.

What is step caching in pipelines?

Reusing a step's output when its code and inputs are unchanged, skipping recomputation.

What must a step's cache key include?

The code version, all input data versions, and parameters. Missing any of them returns stale outputs.

What is a pipeline SLA?

A deadline for when outputs must be ready, such as the nightly model by 06:00, with alerts if it will be missed.

Why give each pipeline and data source an owner?

So failures and contract breaks reach someone who can fix them quickly.