Scheduled and Batch Workloads: Reliability That Counts

Reliability of Scheduled and Batch Workloads

Reliability of scheduled and batch workloads starts with the output a consumer needs by a deadline. “The process exited successfully” is useful execution evidence, but the product may still be missing a partition, using yesterday's input, or publishing after the decision it was meant to support.

Define the expected run independently of the worker. Then check that complete, valid output for its intended data interval became available on time. A job that never starts cannot report its own failure.

TL;DR

Identify runs by the intended data interval, separately from individual attempts.
Define completion, completeness, validity, freshness, and publication criteria.
Detect missing runs from an independent expectation and deadline.
Treat partial output and late recovery separately from on-time success.
Make retries and backfills safe for data and external side effects.
Test monitoring and recovery failures, including loss of the monitor itself.

Quick Example

This complete Python 3.12 example evaluates fictional completion manifests. Times are timezone-aware UTC values. It does not create a schedule, publish data, read a live system, or send alerts.

The result intentionally preserves a missed deadline after recovery. It evaluates one expected interval and trusts the supplied manifest. Production monitoring must verify manifest identity, timestamps, completeness, and publication from an independent source; this toy function does not authenticate evidence or prevent forged success.

Core Concepts

Completion has several dimensions

Start with the consumer's task. A daily report may need all required partitions, validated totals, and publication by 06:00. A snapshot pipeline may instead need a bounded age for the newest complete output. Google's data-processing reliability chapter discusses freshness objectives expressed through processing delay, oldest data, or job completion. Choose the form that reflects the actual consequence of delay.

A run is not an attempt

Use a stable run identity such as the dataset plus intended interval. Attach separate attempt IDs to retries. Otherwise two attempts for one day can look like two successful daily outputs while another day is missing. Record code/configuration version and input boundaries so an operator can reproduce what was attempted.

Publication is a deliberate boundary

For a proposed workflow, write into a staging location, validate the complete result, then expose a committed output version or manifest. Consumers should have an agreed rule for selecting a usable version. Merely creating one output file cannot demonstrate that all partitions are ready or that readers avoid a half-written result.

Build the Failure Matrix Before the Alert

This is an editorial design aid, not a universal alert configuration. A financial close, a search-index refresh, and an internal experimental report have different impact and escalation needs. Agree the deadline and acceptable fallback with the output's consumer before deciding which failure wakes someone.

Include upstream readiness in the investigation. A slow job may have started late because required input was unavailable. Preserve scheduled time, actual start, input-ready time, publication time, and the intended data cutoff separately. One duration metric cannot explain all five boundaries.

Scheduling Controls Do Not Establish Output Correctness

Kubernetes CronJobs illustrate the distinction: startingDeadlineSeconds limits delayed starts, while concurrency policy controls overlap for that CronJob. The CronJob documentation describes approximate scheduling, possible duplicate or absent Job creation, and the need for idempotent jobs. A start deadline is not the report's completion deadline.

Choose and record the intended timezone, daylight-saving policy, catch-up behavior, and treatment of skipped intervals. Test them in the actual scheduler. If overlap is forbidden, decide whether skipped runs are acceptable and how the missed interval will be recovered. If overlap is allowed, verify resource limits and conflicting writes.

The Python example uses a fixed deadline, not a calendar engine. It cannot validate timezone transitions, scheduler outage recovery, Kubernetes admission, or production clocks. Keep those acceptance tests separate from the business rule it demonstrates.

Recovery Must Preserve Meaning

For an isolated recovery drill, simulate a missing partition, interruption before publication, and a duplicate attempt. Verify that consumers see either the prior complete version or the newly validated one according to the agreed contract. Check that a retry does not send duplicate notifications or repeat an irreversible downstream action.

Backfills need an explicit interval range, input/version choice, resource budget, and publication policy. A corrected historical result may change downstream totals; decide whether consumers receive a correction event or must reread a versioned output. Do not silently replace history and assume every downstream cache notices.

Record both objective attainment and recovery time. A late valid output can restore service while still consuming the timeliness error budget. That distinction helps teams improve the missed deadline without ignoring useful recovery work.

Best Practices

Derive monitoring from an expected-output register

Maintain the dataset, interval, due time, required partitions, and consumer owner outside the worker's transient process state. This makes “nothing happened” observable and avoids depending on a success counter to infer absent runs.

Validate a representative output

Use partition inventories, checksums where appropriate, row-count bounds, and business invariants selected for the dataset. A nonempty file can still be incorrect. Reconcile a known interval with an independent source before treating the validation policy as meaningful.

Test alert delivery and monitoring coverage

Introduce a controlled failure and verify that the intended operator receives actionable context. Include run identity, missing condition, deadline, recent attempts, and a bounded recovery route. Also detect a stale monitor so silence has a defined interpretation.

Common Mistakes

Bad: Count exit code zero as successful delivery for every batch.

Correct: Require the consumer's completeness, validity, freshness, and publication conditions.

Bad: Rename every retry as a new scheduled run.

Correct: Keep the interval's identity stable and record separate attempts.

Bad: Erase a missed deadline when a backfill eventually succeeds.

Correct: Record the original miss and the later recovery as separate observations.

FAQ

Is freshness the same as job duration?

No. A short job can publish stale input; a longer job may still meet its consumer's deadline. Track the intended data interval and publication time alongside execution duration.

How do you detect a job that never starts?

Compare an independently recorded expected run with completed usable output after the deadline. Test that the monitor can still run when the worker or its scheduler is unavailable.

Should every missing run page an operator?

Choose escalation from user impact, deadline, fallback, and recovery urgency. Some misses need immediate intervention; others can enter a visible work queue with an owner and due time.

What does the example establish?

The declared manifest rules distinguish seven local scenarios, including pre-deadline pending. It does not establish scheduler reliability, atomic publication, trustworthy manifests, alert delivery, or a successful recovery drill.