Card 1 of 50
Front. Batch. What is batch ingestion?
These cards are original study material. They are not questions from a named employer or a live hiring loop.
All 50 cards in this deck
1.Batch
What is batch ingestion?
A job that reads a bounded set of files or rows on a schedule and loads them. It is simple, and it is late by the length of the schedule.
2.Stream
What is stream ingestion?
Events are processed as they arrive, continuously. The table moves forward with the stream instead of waiting for a nightly file.
3.Cdc
What is change data capture?
Reading the database log of inserts, updates, and deletes, and emitting those changes, instead of polling the table.
4.At-least
What does at-least-once ingestion imply?
A row may arrive twice. The load has to be idempotent, or the table will double-count.
5.Idempotent
What makes a load idempotent?
Loading the same event again does not change the result, usually because the write is keyed by a natural id.
6.Checkpoint
What is a checkpoint?
The position in the stream the job has safely finished. On restart it resumes there, not from the beginning.
7.Offset
What is a consumer offset?
The last message a consumer has committed. Messages after it are the ones still to process.
8.Watermark
What is a watermark?
A moving marker of event time the system believes is complete. Events later than the watermark can still arrive, and they are late.
9.Late
What is late data?
An event that arrives after the window it belongs to was already computed. You either recompute that window or drop the event.
10.Event-time
What is event time?
When the thing happened, according to the event. It is not when your pipeline happened to receive it.
11.Process-time
What is processing time?
When your job saw the event. Windows on processing time are easy and wrong for anything that arrives out of order.
12.Window
What is a window in a stream?
A bounded slice of event time, such as five minutes, over which you compute an aggregate.
13.Session
What is a session window?
A window that closes after a gap with no events from that key, rather than on a fixed clock.
14.Partition
Why partition an ingested table?
So a query or a load touches only the slices it needs, usually by day or by a key, instead of rewriting everything.
15.Schema
What is schema evolution?
A change to the fields over time. Adding an optional field is usually safe. Renaming or changing a type breaks readers.
16.Contract
What is a data contract?
The fields, types, and meaning a producer promises. A consumer builds on that promise, and a breaking change is a migration.
17.Dlq
What is a dead-letter path for events?
Events that fail validation go somewhere a person can inspect, and the good events keep moving.
18.Backfill
What is a backfill?
Reprocessing a past range after a bug or a new column, without disturbing the offsets of the live stream.
19.Replay
What does it mean to replay a topic?
Reading it again from an earlier offset. Downstream tables must tolerate seeing the same events a second time.
20.Ordering
When is stream order guaranteed?
Usually within one partition key. Across keys, arrival order is not the order events happened.
21.Key
What is a good partition key for a stream?
One that spreads volume and keeps events that must stay ordered on the same partition.
22.Skew
What is key skew?
One key has far more events than the others, so one worker does most of the work while the rest wait.
23.Exactly
What do people mean by exactly-once in a pipeline?
The effect of an event lands once, usually by committing the result and the offset together. The event may still be read twice.
24.Txn-sink
Why commit the offset with the write?
If you write and then crash before the offset, the restart writes the row again. If you offset first, a crash loses the row.
25.Snapshot
What is a snapshot load?
A full copy of a source at one moment, used to start a pipeline before the change stream takes over.
26.Upsert
What is an upsert?
Insert the row if the key is new, otherwise replace it. It is how a change stream applies an update.
27.Delete
How does a delete arrive in a change stream?
As its own event. If you ignore it, the warehouse keeps a row the source has removed.
28.Dedupe
Where do you dedupe?
Before the aggregate, using the event id. Deduping after a sum cannot tell a real second event from a retry.
29.Schema-reg
What is a schema registry?
A store of the field definitions producers and consumers agree on, so an incompatible payload is rejected at publish time.
30.Compat
What is backward compatible schema change?
New code can still read events the old code wrote. An optional new field passes that test. Removing or retyping a required field fails it, and old payloads will not load.
31.File-size
Why do tiny files hurt a table?
Each file has overhead to open and to plan. A stream that flushes a file per second creates a table that is slow to read.
32.Compact
What is compaction?
Merging small files into larger ones, and applying pending updates and deletes, so reads scan less.
33.Raw
What belongs in a raw layer?
The payload as received, with arrival time, before you rename or drop fields. You can reprocess when the logic was wrong.
34.Quarantine
What is a quarantine table?
Rows that failed a check, kept with the reason, so the clean table stays trustworthy and the bad rows are not thrown away.
35.Freshness
What is data freshness?
How old the newest row is, compared with now or with the source. It is the lag a dashboard user actually feels.
36.Slo-fresh
What is a freshness SLO?
A target for how late the table may be, such as under 15 minutes, with an alert when the lag crosses it.
37.Watermark-idle
What happens if one source goes idle?
An event-time watermark that waits for that source never advances. An idle timeout lets the watermark move.
38.Fanout
What is fan-out of an ingested event?
One event feeding several downstream jobs. Each consumer keeps its own offset.
39.Pii
Where do you drop or hash a sensitive field?
As early as the contract allows, so the raw layer you keep is already something you are willing to store.
40.Clock
Why not trust a producer's local clock?
Clocks drift. Prefer the source's event time when it is disciplined, and keep arrival time so you can see the skew.
41.Batch-window
What is a batch window's failure mode?
A missed run leaves a hole. The next run has to cover the gap, or that hour is missing until someone notices.
42.Incremental
What is an incremental load?
Reading only the rows that changed since the last successful run, instead of the whole source.
43.High-water
What is a high-water mark?
The latest source timestamp you have fully loaded. The next run starts after it.
44.Gap
How do you notice a hole in a daily table?
A check that every expected partition exists and is above a minimum size. An empty day that loads as zero files looks like success.
45.Schema-drift
What is schema drift?
The source adds, removes, or retypes a field without a coordinated change. A strict load fails. A loose load silently misreads.
46.Poison
What is a poison event?
One payload that crashes the worker every time. It has to be skipped or quarantined, or the partition stalls.
47.Backpressure
What is backpressure on an ingest job?
The sink cannot keep up, so the source read slows. Without it, the job buffers until it runs out of memory.
48.Throughput
What limits ingest throughput?
The slowest of the source read, the transform, and the sink write, plus skew on a hot key.
49.Lineage-in
Why record the source file or offset on each row?
So a bad load can be found and removed by the batch that created it, instead of by guessing which rows.
50.Test-event
How do you test an ingest path?
With a fixture event that includes a duplicate, a late event, and a broken payload, not only the happy shape.