Protocol Buffers: Schemas, Wire Format & Evolution

Q: How do I make a field optional?

In proto3, mark it with the optional keyword to track presence, or use a message type. Without optional, scalar fields cannot distinguish an explicit zero from an unset field.

Protocol Buffers

Protocol Buffers (protobuf) are Google's language-neutral format for structured data. You describe messages in a .proto file, a compiler generates classes for Go, Java, Python, TypeScript, C++, Rust, and many other languages, and those classes serialize to a compact binary encoding that is smaller and faster to parse than JSON. Protobuf is the default message format for gRPC, a common choice for events in Kafka, and the storage format inside many large systems.

The binary encoding is only half the value. The other half is the schema: a single, reviewable definition of every message, shared by every service, with well-understood rules for changing it without breaking producers and consumers that deploy at different times.

TL;DR

Messages are defined in .proto files; field numbers, not names, identify fields on the wire.
Never reuse or change a field number; mark removed numbers and names as reserved.
In proto3, scalar fields default to zero values that are not serialized; use optional (or wrapper types) when you must distinguish "unset" from zero.
Use repeated, map, oneof, nested messages, and well-known types such as Timestamp and Duration.
Generate code with protoc or Buf; use buf lint and buf breaking in CI to catch incompatible changes.
Adding fields is safe; removing, renumbering, or changing types is not.

Quick Example

Using the generated Python class:

Core Concepts

Field Numbers and the Wire Format

Each field is encoded as a tag (field number plus wire type) followed by its value. Names never appear in the binary format, which is why the encoding is compact and why renaming a field does not break binary compatibility — but changing its number does. Numbers 1–15 take one byte to encode, so assign them to the most frequent fields.

Negative numbers in int32 and int64 always use ten bytes; sint32 and sint64 use zigzag encoding and are far more compact for values that are often negative.

Field Types and Structures

Scalars: integers, floats, bool, string (UTF-8), bytes.
repeated: lists; numeric repeated fields are packed by default in proto3.
map<K, V>: key-value maps (keys are integers or strings; order is not preserved).
oneof: at most one of several fields is set — a tagged union.
Enums: the first value must be zero; name it ..._UNSPECIFIED so the default is never a real state.
Nested and imported messages compose larger structures.

Presence and Defaults

In proto3, a scalar field set to its default — 0, "", false — is not serialized, and reading an unset field returns the default. That means discount_cents = 0 and "no discount specified" look identical. Declare the field optional (or use a message type, which always has presence) when the difference matters. Newer editions (Edition 2023 and later) replace the proto2/proto3 split with per-feature settings, including explicit presence by default.

Well-Known Types

JSON Mapping

Protobuf has a canonical JSON form: field names become lowerCamelCase, 64-bit integers become strings, enums become their names, and Timestamp becomes an RFC 3339 string. It is useful for debugging, logs, and HTTP gateways — but renaming a field does break JSON compatibility, even though the binary format does not care.

Evolving Schemas Safely

Packages with versions (shop.orders.v1) allow a clean break when one is truly needed: create v2 alongside v1 and migrate consumers. See Event Schema Evolution and API Versioning.

Tooling

protoc: the reference compiler, with plugins per language.
Buf: a modern toolchain with dependency management, linting, formatting, breaking-change detection, and remote code generation through the Buf Schema Registry.
Language runtimes: official libraries for most languages; alternatives such as protobuf-es for TypeScript and prost for Rust produce more idiomatic code.
Schema registries: Kafka deployments often register protobuf schemas in a registry that enforces compatibility on publish.

Best Practices

Treat .proto Files as Public Contracts

Keep them in a shared repository or registry, review changes like API changes, and run buf breaking against the main branch in CI.

Make the Zero Value Meaningless

Use ..._UNSPECIFIED = 0 for enums and validate that required fields are set, so a missing field never silently means "pending" or "free".

Use Explicit Units and Types

Name fields with units (timeout_ms, price_cents), use Timestamp and Duration for time, and avoid floating point for money.

Validate Semantics Separately

Protobuf checks types, not meaning. Use a validation layer — such as Protovalidate annotations — or application code for ranges, formats, and required fields.

Common Mistakes

Reusing a Deleted Field Number

Relying on Zero Values

Enum Zero as a Real State

FAQ

What are Protocol Buffers?

A language-neutral, schema-based format for serializing structured data. Messages are defined in .proto files, code is generated for each language, and data is encoded in a compact binary format.

Is protobuf faster than JSON?

Generally yes: messages are smaller and parsing is faster, because the format is binary and field names are replaced by numbers. The difference matters most for high-volume service-to-service traffic and storage; for browser APIs, JSON's readability often wins.

Should I use proto2, proto3, or editions?

Use proto3 or editions for new work. Editions are the newer model that unifies proto2 and proto3 behavior through feature flags; tooling support continues to grow, and proto3 remains widely used and well supported.

How do I make a field optional?

In proto3, mark it with the optional keyword to track presence, or use a message type. Without optional, scalar fields cannot distinguish an explicit zero from an unset field.

Can I change a field's name?

The binary format does not use names, so renames are safe for binary data. They break the JSON mapping and generated code that refers to the field, so treat renames as breaking whenever JSON or source compatibility matters.