Data quality and governance

Why can equal source and destination row counts coexist with missing data?

Missing rows can be offset by duplicates or unrelated rows. Reconcile stable keys and relevant aggregates, not only total counts.

Why must a completeness check name the expected population and time interval?

A count has no correctness meaning without its denominator. Define which sources, entities, and cutoff are expected, including legitimate exclusions and late arrivals.

A table contains one event from the last minute. Does that prove the dataset is fresh and complete?

No. A recent maximum timestamp can hide stalled partitions or missing producers. Measure progress and completeness by source or required partition.

Why should a quality rule distinguish unknown, not applicable, and missing due to failure?

Those states require different interpretation and repair. Converting all of them to zero or an empty string hides uncertainty and can distort measures.

A fact table rejects repeated customer IDs. When is that uniqueness test wrong?

When the declared grain includes more fields, such as customer plus order. Test the actual composite grain; a convenient column is not necessarily the entity key.

Why can a fixed daily volume threshold create false alarms every weekend?

Expected volume varies by calendar and business cycles. Use an appropriate baseline and retain absolute completeness checks so adaptive thresholds do not normalize a real outage.

When should a failed quality check block publication rather than only alert?

When publishing would violate a defined consumer-critical invariant, such as duplicate payment keys. Specify severity and fallback per rule instead of making every anomaly fatal or ignoring all of them.

Why should dropping invalid rows into quarantine be visible in downstream metrics?

The retained dataset may become systematically biased. Publish rejected counts and reasons, track repair, and decide whether partial results remain fit for the intended use.

What should a quality alert identify so it can lead to repair?

Affected dataset and interval, failed rule, magnitude, consumer impact, responsible owner, and a safe diagnostic path. An unowned alert is unlikely to restore data.

Why can automatically collected table lineage miss a critical dependency?

Dynamic SQL, exports, spreadsheets, and application-side transformations may not be captured. Verify important consumption paths instead of assuming the catalog graph is complete.

Why is hiding a sensitive column in a dashboard insufficient access control?

Users or services may query the underlying table or exports directly. Enforce access at the data-serving boundary and test with the actual consumer identity.

What should a tenant-isolation test query besides the user's normal dashboard?

Direct table or view access, joins, exports, cached results, and service-account paths. A UI filter does not establish row-level isolation.

Does replacing a person's name with a stable hash make a dataset anonymous?

Not necessarily. Linkability and other attributes may permit reidentification. Treat hashing as a transformation with a threat model, not an automatic removal of sensitivity.

Why must a deletion workflow inventory snapshots, replicas, exports, and derived datasets?

Deleting the source row does not remove every copy. Track propagation and retained recovery data under an explicit policy, with evidence of completion.

How can restoring an old backup reintroduce records intentionally removed later?

The backup predates the deletion. Apply the relevant deletion history during recovery before exposing the restored dataset, according to the established retention policy.

Why should temporary debugging access have an expiry and an owner?

Incident access can otherwise become permanent privilege. Use scoped access, record its purpose, and verify removal when the investigation ends.

How should a producer remove a column used by unknown consumers?

Identify usage and owners, announce a deprecation, provide a migration, and verify remaining use before removal. Lack of catalog entries is not proof that no consumer exists.

Two teams report different active-user counts from the same source. What definitions should be compared first?

Eligibility, event types, identity resolution, time zone, date window, deduplication, and late-data handling. Different valid definitions can explain disagreement without a broken pipeline.

What belongs in evidence for a high-trust published dataset?

Input versions, transformation version, executed rule results, rejected-record accounting, reconciliation, and publication identity. A manually checked spreadsheet or a green scheduler icon is not a reproducible quality record.