Apache Avro
Apache Avro is a schema-based serialization system used in data pipelines and messaging. The important design boundary is between the schema that wrote a value and the schema the consuming application expects. Preserve the writer schema and test the actual reader/writer pair before changing a pipeline.
Schema compatibility is directional. A new consumer reading old events and an old consumer reading new events are different cases. A compatibility badge is useful only when its direction, version history, and implementation match the rollout you intend.
TL;DR
- Preserve the writer schema needed to decode binary data correctly.
- Distinguish raw binary records, object container files, and transport-specific envelopes.
- Test old-to-new and new-to-old reader/writer combinations separately.
- Use defaults for missing reader fields without treating them as business truth.
- Check meaning, replay, and downstream transformations alongside structural compatibility.
Quick Example
This complete Python example uses Apache avro 1.12.0 and in-memory bytes. It writes an old record and reads it using a newer schema with an added default. A separate negative case tries to read a new long field with the old int reader.
This is real local Avro serialization and resolution, not a schema-registry or Kafka integration test. The value 7 fitting in an integer does not make a declared long writer compatible with an int reader. The schemas, not just a convenient sample value, define that boundary.
Core Concepts
Schema and encoding
Avro binary records omit field names and individual type labels; decoding depends on the writer schema. Object container files include schema metadata, while raw record bytes need a separate agreed mechanism. A Kafka envelope or registry identifier is part of that transport contract, not a property every Avro byte sequence automatically carries. Avro serialization specification
Resolution and defaults
Reader/writer resolution matches compatible types and record fields. A reader field absent from the writer can use its declared default; without an applicable default, resolution fails. A default does not make a field optional for every writer or establish that the default accurately describes historical business data.
Runtime implementation
Use the library and version deployed by each producer and consumer. Apache’s Python guide describes its schema and file workflow, but cross-language and transport behavior still need representative tests. Pin dependencies and retain the schema and fixture data with the result. Python getting started
Build a directional evolution matrix
Define compatibility terms through concrete pairs. For this article, “new reads old” means a new reader consumes data written with the old schema; “old reads new” means an old reader consumes new-writer data. Record whether your registry checks only the previous schema or the relevant full history.
The table is a test-planning guide, not a universal compatibility verdict. Exercise the exact library pairs, union branches, null values, logical types, and aliases that your data uses. A successful test with one common value can miss a branch that appears only in historical data.
Version the semantic contract as well as the schema. Replacing milliseconds with seconds inside the same numeric type may remain structurally readable while changing every result. Name units and time assumptions in the contract and add assertions that detect the change.
Roll out with replay and failure handling
Start with producers, consumers, stored files, connectors, and reprocessing jobs. Identify which versions can coexist and which older records can reappear. Design the sequence so each supported reader/writer pair works for the overlap period.
Keep a controlled route for malformed or incompatible records with schema identity, origin, and restricted diagnostic evidence. Do not silently drop them or convert them into a success-shaped empty record. Decide whether the pipeline should stop, quarantine, or continue according to the consequence of missing data.
Rehearse historical replay into an isolated target, then compare business counts and aggregates as well as deserialization success. A consumer can decode a record correctly and still produce a duplicate or incorrect output. Preserve idempotency and checkpoint assumptions when retrying or backfilling.
Best Practices
Retain writer-schema identity
Store enough information to retrieve the exact writer schema for every retained data segment or message. Test retrieval during replay and recovery, not just ordinary ingestion.
Test both structure and meaning
Use negative cases for unsupported schema pairs and assertions for units, defaults, keys, and allowed categories. A decoder accepting bytes is one layer of the contract.
Tie compatibility to rollout history
Keep the supported version matrix and retirement decision with the release. Removing an old consumer does not remove old data or a scheduled historical job.
Common Mistakes
Bad: Assuming a reader default proves historical data contained that value.
Correct: Preserve provenance and distinguish substituted defaults where the business decision needs it.
Bad: Testing only new readers with old writers.
Correct: Also test old readers against new writers when those versions can coexist.
Bad: Treating raw binary bytes as self-describing.
Correct: Preserve the writer schema through the documented container or transport contract.
FAQ
Is adding a default always sufficient?
No. It addresses particular missing-reader-field cases. Check type resolution, direction, implementation, and business meaning.
Does this example test a schema registry?
No. It exercises the Apache Python library locally with explicit schemas and invented data.
What should a replay test verify?
Successful resolution plus expected record counts, keys, semantics, duplicate handling, and downstream outcomes.
Related Topics
- Data Contracts and Schema Evolution — Define the pipeline contract.
- Event Schema Evolution — Plan producer/consumer overlap.
- Kafka Producers — Understand message transport choices.