Data system design

What is a lambda architecture?

A batch path that recomputes history and a stream path that covers the recent gap. You keep two implementations of the same metric.

What is a kappa architecture?

One stream path, and you reprocess history by replaying the log. There is no second batch codebase to drift.

What is a data lake?

Files in object storage, kept in open formats, that many jobs can read. Without a table format and owners, it becomes a pile of unreadable files.

What is a serving layer?

The store a product reads for a low-latency lookup, filled by the pipeline. It is not the warehouse the analyst scans.

What is lineage?

Which sources and jobs produced a table. It is how you find everything a broken upstream touches.

What is a data quality check?

A test on freshness, volume, nulls, or an accepted range, run before the table is marked ready for readers.

What does a producer contract protect?

The downstream jobs. A renamed field fails the contract at the producer, instead of zeroing a metric tomorrow.

What belongs in a backfill plan?

The range, the expected row change, and how readers are kept on the old snapshot until the new one is checked.

How do you make a daily job safe to rerun?

The partition is overwritten, or the write is a merge on a key. An append-only rerun duplicates the day.

What does an orchestrator do?

It runs jobs in dependency order, retries a failure, and records which run produced which output.

What is a sensor in a pipeline?

A wait for something upstream to exist, such as a partition. A sensor with no timeout waits forever and looks like a running job.

What is a pipeline SLA?

The time by which a table must be ready. The alert fires on the lateness, not only on a task that threw an exception.

How do you keep a streaming table readable?

Compact files on a cadence, and do not commit a file per event. Readers pay for every file they open.

What is schema-on-read?

The files have no enforced schema. Each reader interprets the bytes. One job's assumption can disagree with another's.

What is schema-on-write?

The table rejects a write that does not match. Producers find out immediately. Readers can trust the columns.

How do you design access for sensitive columns?

Put them in a separate location or policy, and give the wide analyst table a token or nothing. Copying them into every mart spreads the risk.

What is time travel on a table?

Reading a snapshot from earlier. It is how you compare a bad load with the previous good one, and how you restore.

Where does change data capture sit?

Close to the source database, writing an append-only log. The warehouse applies that log. The source is not queried for diffs.

How do you get a once-only effect?

Store the source key with the row, and make the write a merge. Reprocessing then replaces the row instead of adding another.

How do you handle events from many clocks?

Carry event time and arrival time. Define the window on event time, and decide how late an event may be and still count.

Why keep a staging table?

So the raw payload, the cleaned row, and the mart can change at different speeds. Skipping straight to the mart makes a replay expensive.

What is fan-in?

Several sources landing in one table. Each source needs a key namespace, or their ids will collide.

What is fan-out from a cleaned table?

Several marts read one cleaned table. A fix in that table reaches every mart on the next build. Logic copied into each mart has to be fixed once for every mart.

What drives warehouse cost in a design?

Bytes scanned, how often, and how long the engine is on. Partitioning and a schedule cut more spend than a larger engine.

What is the freshness and cost tradeoff?

A tighter freshness reruns more often and scans more. Name the decision the fresher number changes, or keep the cheaper schedule.

Where do bad rows go in the design?

A side table with the reason, while the published table contains only rows that passed. Dropping them on the floor hides the source bug.

What does publishing a table mean?

Swapping the ready snapshot into the name readers use, after the checks pass. Readers never see the half-built copy.

What is a blue-green table publish?

Build the new version beside the current one, check it, then point the name at it. The old version remains for a rollback.

How do you model a pipeline dependency?

A downstream job starts from the upstream snapshot it actually read, not from a clock that hopes the upstream is done.

A fact arrives before its dimension. What do you do?

Load the fact with the key, and allow the dimension lookup to miss until the dimension row arrives, or hold the fact.

How do deletes propagate?

The change feed includes the delete, and every downstream merge applies it. A pipeline that only inserts never forgets a row.

What does a deletion request demand of the design?

A way to find that person's rows in every table, including raw and backups, within the time you promised.

Why keep the raw log immutable?

A logic bug can be fixed and the mart rebuilt from the log. If you transformed in place, the original is gone.

What do you replay in a pipeline test?

A small log with a duplicate, an update, a delete, and a late event, asserted against the table you expect.

What do you alert on besides task failure?

Freshness, row-count change, and null rate. A job that succeeds on zero rows is a failure the task status will not show.

Who owns a table in the design?

The team that can change its definition and answer when it is wrong. A table with no owner will be used anyway.

What belongs in a table's catalog entry?

The grain, the freshness, the owner, and the definition of each metric column. A name alone will be misused.

How do you change a column readers use?

Add the new column, move readers, then remove the old one. Dropping it in the same change breaks today's dashboards.

When is a sample the wrong serving path?

When the user needs a correct total. A sample is for exploration. A published metric is computed on the full grain.

What is an aggregate table?

A precomputed rollup at a coarser grain, so a dashboard does not scan the raw fact. It has to be rebuilt when the fact changes.

What is definition drift?

The same metric name computed differently in two pipelines. A single definition, stored once, is how you stop it.

Why expose a watermark?

So a reader knows up to what event time the table is complete, and does not treat a partial trailing window as final.

How do you place a pipeline across regions?

Keep the raw data in the region that requires it, and be explicit about which aggregates are allowed to leave.

Where does a schema registry sit?

Between producers and the raw log, so an incompatible event is rejected before it is the warehouse's problem.

What happens when the warehouse cannot absorb the stream?

The ingest lags, on purpose, instead of dropping events or crashing. The freshness SLO tells you when that lag is an incident.

Can you change a partition column later?

It is a rewrite of the table. Choose it for the filter you actually run. Changing it is a migration, not a setting.

How do you join a tiny dimension to a huge fact?

Broadcast the dimension, or filter the fact first. Shuffling the fact to match a hundred-row dimension wastes the cluster.

What is a retention rule?

How long each layer is kept. Raw logs, snapshots, and marts do not need the same lifetime, and backups count.

Why stamp a run id on produced files?

So a failed or wrong run can be identified and removed without a full restore.