Data Quality Tests and Freshness SLOs
Data-quality checks need a declared contract: what should exist, which values are valid, which keys must be unique, and when consumers need the result. A freshness SLO adds an objective and observation window for a meaningful indicator. A recent maximum timestamp can coexist with stale, missing or incorrect data.
Begin with the consumer's decision and the expected population. Then choose checks that can detect failures that matter to that consumer. A table can pass a schema check while missing half its intended records, or arrive on time with the wrong values.
TL;DR
- Quality checks need a declared contract: what should exist, which values are valid, which keys are unique, and when consumers need the data.
- A freshness SLO adds an objective and an observation window for a meaningful indicator.
- A recent maximum timestamp can coexist with stale, missing, or incorrect data.
- Start from the consumer's decision and the expected population, then choose checks that catch the failures that matter to it.
- A successful pipeline run is orchestration status, not evidence of correct output.
Quick Example
This Python example uses two fictional rows and a fixed timezone-aware clock. It checks one duplicate, one negative value and a fifty-percent fresh-row fraction under a declared one-hour rule. It contacts no source system or monitoring service.
The maximum event timestamp is recent even though one row is stale. Neither that maximum nor the fresh-row fraction tells us how many expected rows are absent. The example's negative amount is invalid under this synthetic contract only; another domain could legitimately allow negative values. No actual SLO window, incident or pipeline was evaluated.
Core Concepts
Completeness requires an expectation
Define what should have arrived: a manifest, known partition range, source count, required fields or another defensible control. A nonempty dataset is not necessarily complete. If the expected population is unknown, record that limitation rather than reporting a precise completeness percentage without a denominator.
Distinguish missing rows from missing fields and missing partitions. Their detection and recovery paths differ. A completed job can still omit a source partition, while a populated partition can contain null values in a field the consumer requires.
Validity and uniqueness need domain scope
State the allowed types, ranges, relationships and meanings. A syntactically valid date can still be impossible for the business event, and a valid identifier can refer to an absent parent. Keep business-rule validation separate from format acceptance.
Define the uniqueness key and period. The same local identifier may be valid across tenants or repeated across intentional revisions. An overly broad duplicate rule can reject correct data; an overly narrow rule can miss repeated business events. Test the actual identity contract.
This matrix is an editorial test plan. The two-row fixture exercises only its stated duplicate, range and age calculations.
Freshness needs the right time and population
Google's SRE workbook distinguishes pipeline freshness, coverage and correctness indicators, with examples tied to what users consume. That separation is useful: recently updated data can be incomplete or wrong, and a successful job count can hide which result a consumer actually reads. Pipeline SLI guidance
Define whether time means source event time, ingestion time, transformation completion or published availability. These measure different delays. A newly ingested historical event can be recent by arrival time while old by event time, and neither interpretation is universally correct for every consumer.
Choose a denominator and window that fit the use. You might assess eligible records, partitions or consumer reads, each with stated exclusions. The article's half-fresh sample is one instantaneous calculation and not a measured monthly SLO or error budget.
Quality failures need a decision policy
Decide whether a failed check blocks publication, quarantines a partition, serves a clearly labeled older result, or escalates for review. Base that choice on consequences. Silently substituting stale data may be acceptable for one dashboard and harmful for another workflow.
Separate an unavailable check from a passing check. If the source control count could not be fetched, completeness remains unknown. If a parser failed, dropping those rows before computing an error rate can make the surviving data look healthier while concealing the failure.
Best Practices
Keep the data contract, check definitions, ownership and recovery path together. Version material rule changes so a rise in reported failures can be interpreted against changed expectations. Avoid describing a threshold adjustment as improved underlying data quality without evidence.
Use small fixtures that isolate each rule, including missingness, duplicates, timezone boundaries, future timestamps and empty populations. Then test representative real pipeline behavior in an authorized environment. A rule that works on an in-memory list still needs integration with publication, monitoring and consumer access.
Preserve counts at important boundaries. Reconcile source, accepted, rejected, transformed and published populations according to the pipeline's semantics. Legitimate deduplication or aggregation can change counts, so the expected relationship should reflect the transformation rather than demand equality everywhere.
Route incidents to an owner who can act. Include the affected partition or output identity, failing rule, observation window and consumer impact without dumping private rows into an alert. Define when to stop publication, when to recover and how to verify that corrected output reached the consumer.
Review alerts for usefulness. A threshold that pages continuously without changing an action is a sign to revisit the objective or operating process. A quiet dashboard can also hide missing instrumentation. Verify signal coverage as well as reported success.
Reverse a freshness conclusion when only the newest row is observed and the consumer needs a complete partition. Revisit a validity rule when domain owners show that its assumptions are wrong. A quality system should make those disagreements explicit rather than encode one convenient rule forever.
Common Mistakes
Using max(timestamp) as freshness
Checking uniqueness without key scope
Excluding rejected rows from every denominator
Treating a green pipeline as correct output
FAQ
Does a recent latest timestamp prove freshness?
Only for that latest observation under its time definition. It can hide stale or missing records. Choose a population and indicator that reflect the consumer's requirement.
Is the sample's fifty percent an SLO result?
No. It is an instantaneous fictional row fraction. A real SLO needs a defined indicator, target, window, exclusions and observed production evidence.
What should happen when a check cannot run?
Record unknown or failed observation explicitly and follow the publication/recovery policy for that missing evidence. Do not silently convert unavailable validation into a pass.