Card 1 of 20
Front. Logical Interval. Why should a daily task use its assigned data interval instead of today's wall-clock date?
These cards are original study material. They are not questions from a named employer or a live hiring loop.
All 20 cards in this deck
1.Logical Interval
Why should a daily task use its assigned data interval instead of today's wall-clock date?
Retries and backfills must process the intended historical slice. Wall-clock selection changes the input when execution is delayed or repeated.
2.Input Version
Why can rereading the latest upstream table make a retry produce different output?
The table may have changed since the first attempt. Pin a snapshot, partition version, or manifest when the run must be reproducible.
3.Partial Files
A task writes nine of ten output files and crashes. How should consumers avoid reading a partial dataset?
Write under a run-specific staging location and publish a validated manifest or atomic table snapshot only after all required files exist.
4.Rename Assumption
Why should a pipeline not assume renaming an object-storage directory is an atomic publish?
Object stores may implement such operations as multiple object copies and deletes. Use the storage system's actual atomic metadata or commit mechanism.
5.Retry Identity
Should each retry create a new business operation identifier for an export request?
No, when the retry represents the same operation. Keep a stable identity so the destination can recognize duplicate attempts and recover the original outcome.
6.Commit Unknown
A load job times out after sending a warehouse commit. What should recovery check first?
Whether the run's output already committed, using its durable run identifier or transaction status. Blindly appending again can duplicate the load.
7.Remote Handoff
Why should a downstream task receive an object URI rather than a path on the upstream worker's local disk?
Tasks may run on different machines and local files may disappear on retry. Publish durable data and pass a small reference plus version or checksum.
8.Parse Work
Why should network lookups not run at the top level of an Airflow DAG file?
The scheduler repeatedly parses DAG definitions. Expensive or unreliable imports and lookups can delay scheduling unrelated work; defer execution to tasks or use static generated configuration.
9.Retry Class
Which failures should not be retried repeatedly without changing input or code?
Deterministic schema violations, invalid credentials that require intervention, and malformed records. Classify errors so retries address transient failures rather than consume capacity indefinitely.
10.Backfill Isolation
How can a historical backfill avoid starving today's scheduled load?
Use separate capacity budgets, concurrency limits, and priority. Track both live freshness and backfill progress while protecting the shared source and sink.
11.Write Conflict
A backfill and live job both replace the same partition. What coordination is necessary?
A defined owner or version-checked publication protocol. Last-writer-wins can erase valid newer data even when each job succeeds individually.
12.Dependency Version
Why is 'upstream task succeeded' weaker than 'the required upstream dataset is ready'?
The successful run may cover a different interval or publish an older version. Dependencies should identify the specific data slice and completeness contract.
13.Empty Success
When is an empty output a legitimate success rather than a missing-data incident?
When the source is known complete for the interval and zero records is allowed by the contract. Record that evidence; an empty listing alone does not establish completeness.
14.Rerun Downstream
After correcting an intermediate table, how do you decide which downstream results to rebuild?
Trace dependencies at the affected partition or version level, including aggregates spanning the correction interval. Rebuilding only the failed task leaves stale descendants.
15.Code Version
Why should a run record the transformation code version as well as input versions?
The same data can produce different results under changed logic. Reproducibility and rollback require both, plus relevant configuration and dependencies.
16.Checkpoint Grain
What tradeoff determines whether a batch checkpoints per file or per record?
Recovery work versus metadata and commit overhead. The chosen boundary must also support idempotent publication and detection of partially applied units.
17.Retry Storm
A shared API recovers after an outage. Why can thousands of simultaneously retried tasks break it again?
Retries synchronize demand and exceed recovery capacity. Add jitter, concurrency limits, and a global retry budget instead of only increasing attempt counts.
18.Rollback Data
Why can reverting pipeline code fail to undo a bad production load?
Incorrect rows already persisted. Restore or reconstruct the affected dataset and republish it, accounting for downstream consumers and any new valid data.
19.Recovery Proof
What evidence demonstrates that a repaired partition is complete?
Source-to-target reconciliation, expected key coverage, contract checks, and verified downstream publication. A green task status only proves the task's configured success condition.
20.Runbook Specific
What makes a pipeline recovery runbook executable by someone unfamiliar with the job?
Exact dataset and interval selection, safe rerun commands, ownership rules, expected checks, rollback steps, and escalation criteria. Test it with an actual controlled failure.