Device Telemetry, Timestamps & Offline Buffering

Device Telemetry, Time, and Offline Buffering

A sensor that loses its network connection for six hours keeps measuring. When it reconnects, the backend needs every reading — in order, without duplicates, and with the right timestamps. That sounds simple until you remember that the device may have rebooted twice, may not have known the time when it woke up, has a few kilobytes of RAM and limited flash, and may upload the same batch three times because acknowledgements were lost.

Reliable device telemetry is designed for those conditions: every sample carries enough context to be ordered and timed later, buffers are bounded with explicit policies for what to drop, uploads are acknowledged and idempotent, and losses are counted and reported rather than silent.

TL;DR

Record monotonic time (since boot) for every sample, plus wall-clock time when known; never assume the clock is set.
Add a boot ID and a persistent sequence number so the backend can order, deduplicate, and detect gaps.
Backfill wall-clock timestamps after time synchronization using the monotonic offset.
Buffer in a bounded ring buffer in RAM and, when needed, a flash-backed queue with wear leveling.
Decide drop policies in advance — drop oldest, downsample, aggregate, or prioritize — and count drops.
Upload in batches with acknowledgements; ingest idempotently on the server, keyed by device, boot, and sequence.

Quick Example

A sample format and a bounded ring buffer that keeps the newest data and counts what it drops (C):

On the server, the same triple identifies every sample exactly once:

Core Concepts

Two Kinds of Time

Devices often boot without knowing the time: no battery-backed RTC, no network yet, GNSS not locked. Recording only wall-clock time either produces samples dated 1970 or blocks measurement. Record monotonic time always, wall-clock time when known, and the boot ID, so timestamps can be reconstructed. Store and transmit UTC; convert to local time only for display.

Getting the Time

Wall-clock time can jump on synchronization; never use it for intervals or timeouts. TLS certificate validation also needs a roughly correct clock, so the first sync often happens before secure connections are possible — plan for it.

Sequence Numbers and Boot IDs

A per-device sequence number persisted in non-volatile memory (or rebuilt from the boot ID and a per-boot counter) lets the backend order samples, detect gaps (lost data), and deduplicate retransmissions. Persist the counter in batches to limit flash writes, skipping ahead after reboot so numbers are never reused.

Buffering and Storage

Flash has limited erase cycles; write in pages, batch writes, use wear leveling (via a flash file system such as littlefs or a log-structured design), and handle partially written records after power loss with checksums.

Drop Policies

When storage fills, something must give. Decide in advance:

Drop oldest to keep recent data (common for monitoring).
Drop newest to preserve a complete early record (common for event logs).
Downsample or aggregate (store minute averages instead of raw samples) as the buffer fills.
Prioritize classes: alarms and events are kept over routine readings.

Always count dropped and downsampled samples and report the counts, so the backend knows a gap is real rather than an outage of the data pipeline.

Uploading Reliably

Batch samples to reduce radio wake-ups and protocol overhead; see Low-Power Design and Battery Budgets.
Acknowledge batches end to end — an MQTT QoS 1 PUBACK only confirms receipt by the broker, so use application-level acknowledgements when the backend must confirm storage. See MQTT.
Delete local data only after acknowledgement; expect duplicates after timeouts and design ingestion to be idempotent. See Idempotency.
Upload oldest first when ordering matters, or newest first when freshness matters, but mark backfilled data so dashboards and alerts treat it correctly.
Version the payload schema and include the firmware version; fleets run many versions at once. See Firmware OTA Updates.
Use compact encodings (CBOR, protobuf) on constrained links.

Best Practices

Make Every Sample Self-Describing

Include device ID, boot ID, sequence number, monotonic time, wall-clock time if known, and schema version.

Bound Every Buffer

Fix capacities in RAM and flash, choose drop policies deliberately, and test what happens when they fill.

Deduplicate on the Server

Use a natural key such as device plus sequence number so retries and replays are harmless.

Report Data Health

Send dropped counts, buffer fill levels, time-sync status, and clock offset with telemetry and monitor them.

Common Mistakes

Timestamps From an Unset Clock

Unbounded Queues

Treating Broker Acknowledgement as Storage

FAQ

Why record monotonic time if the device has a clock?

Because the wall clock may be unset, wrong, or corrected with jumps. Monotonic time is always valid within a boot and lets you reconstruct wall-clock timestamps once the offset is known.

How do I avoid duplicate readings after reconnecting?

Give each reading a unique key — for example device ID plus a persistent sequence number — and make ingestion idempotent so repeated uploads are ignored.

How much data should a device buffer offline?

Enough to cover expected outages, sized from sample rates, record sizes, and available memory and flash, with an explicit policy for what happens when the buffer fills.

How do I protect flash memory from wearing out?

Write in batches, use wear-leveling file systems or log-structured storage, avoid rewriting the same location frequently, and estimate write volume against the flash's rated endurance.

What should a device report about its own data quality?

Dropped and downsampled sample counts, buffer fill levels, time synchronization status and last sync time, and firmware and schema versions.