Card 1 of 20
Front. Watermark Estimate. A watermark passes noon. Does that prove no event with an earlier timestamp will arrive?
These cards are original study material. They are not questions from a named employer or a live hiring loop.
All 20 cards in this deck
1.Watermark Estimate
A watermark passes noon. Does that prove no event with an earlier timestamp will arrive?
No. It represents the source or runner's progress assumption. Define what happens to late records rather than treating the watermark as proof of completeness.
2.Idle Partition
Why can one idle source partition prevent event-time windows from closing?
A downstream watermark can be constrained by the slowest active input. Configure supported idleness detection carefully, including how resumed late events will be handled.
3.Trigger Output
Why can one event-time window produce several outputs?
Early and late triggers may emit intermediate panes or revisions. Downstream consumers must know whether each output is a delta or a replacement aggregate.
4.Late Policy
What business decision is hidden inside an allowed-lateness setting?
How long results remain revisable and what happens to later evidence. Balance completeness, state cost, and correction needs; retain a repair path for rejected late events.
5.Checkpoint Scope
What does restoring consistent operator state from a checkpoint fail to guarantee by itself?
Exactly-once effects in an external sink. The sink must participate in a compatible commit protocol or make repeated writes idempotent.
6.Checkpoint Duration
Why can growing checkpoint duration threaten recovery even when the job processes records fast?
The latest completed recovery point becomes older, increasing replay and risking timeout or retention gaps. Inspect state size, storage throughput, and barrier alignment.
7.State Rescale
What must be preserved when rescaling a keyed stateful job?
The mapping from logical keys to their state, using the engine's supported redistribution and compatible operator identities. Starting empty workers without state migration loses history.
8.State TTL
Why is expiring join state after one hour a correctness decision, not only a memory setting?
A matching event arriving later can no longer join. Derive retention from event-time and lateness assumptions, and define unmatched-event behavior.
9.Unbounded Join
Why can joining two unbounded streams on customer_id grow state indefinitely?
Without a time bound or eviction semantics, any old record might match a future record. Bound the relationship or use a model that retains only the required versioned state.
10.Hot Key
Why does doubling parallelism not fix a single customer key that dominates processing?
All events for that key may still go to one state owner. Use a valid partial aggregation or decomposition if the operation permits it; arbitrary salting can break order and state semantics.
11.Associative Combine
When can pre-aggregating before a shuffle reduce traffic safely?
When partial results can be combined with the operation's required semantics. Sum and count combine naturally; an average needs sum and count, and order-dependent operations require more care.
12.Lookup History
Why can enriching a replayed event with the current customer record change historical results?
The reference data has changed since the original event. Use event-time versioned enrichment or record the reference version when reproducibility matters.
13.Backpressure Location
Lag grows at the source. Why might adding source readers not help?
The bottleneck may be a downstream sink, join, or hot partition. Follow backpressure and stage throughput to the constrained operator before changing parallelism.
14.Replay Capacity
Why should replay throughput be budgeted separately from live ingestion?
Recovery competes for the same state, storage, and sink capacity. Reserve headroom or isolate replay so catching up does not make current data increasingly stale.
15.Retention Gap
A job resumes from a checkpoint whose offsets have expired from the source. What is required?
A recovery source such as archived raw data or a new consistent snapshot, plus reconciliation. Skipping to the earliest available offset silently loses the gap.
16.Dedupe State
Why does deduplicating every event forever require an explicit storage plan?
The set of seen identities grows without bound. Bound retention according to replay guarantees or use a durable business constraint where duplicate effects must never recur.
17.Source Sequence
Why is source sequence preferable to arrival time when applying CDC updates to one record?
Network and retry order can differ from source mutation order. Reject stale versions using authoritative sequencing rather than letting the last arrival overwrite newer state.
18.Snapshot Handoff
What must a snapshot-plus-CDC ingestion establish at its handoff?
A consistent snapshot boundary and change-log position with no missing changes. Any overlap must be deduplicated by source identity or version.
19.Correction Output
A late refund changes yesterday's revenue. What must the downstream table support?
A revision, retraction, or idempotent replacement at the aggregate key. An append-only consumer that sums every emitted total will overcount.
20.Recovery Test
What should a stateful stream test compare after an injected failure and restore?
Final logical state and sink results against an uninterrupted reference run, including duplicates, late events, and reordered arrivals. Processing the same number of messages is not sufficient.