Schema evolution: event formats & compatibility

Which side ships first?

The pull request is four lines. The charging service of an electric-vehicle network publishes a SessionEnded event each time a car unplugs, with the energy delivered in energy_wh, and the change adds a connector field so the maintenance team can see which connector types fail most. The first review comment is a question: does this go out before the billing service upgrades, or after? Billing reads the same events to charge drivers and deploys on its own schedule.

There is no moment when every program agrees on the event's shape. Producers and consumers deploy separately, often days apart. The topic keeps a week of events, so a consumer that restarts reads events written before the change (retention and replay are explained in Pub/sub: topics, subscriptions & replay). And a loader copies every event into a data lake, where files from 2022 sit next to this morning's. Schema evolution is changing an event's shape so that every reader that may meet it, today or in a replay a year from now, can still read it.

What the bytes know about themselves

Whether connector is harmless depends on the encoding. Plain JSON carries every field name in every message and checks nothing before sending; a consumer that needs a missing field finds out when its code fails, and JSON Schema catches a bad message only where someone validates against it. Two binary formats make the shape part of the contract:

• Avro sends values with no field names, readable only with the schema that wrote them, the writer schema. The consumer brings its own reader schema, and Avro resolves the two by field name: a field only the writer has is skipped, a field only the reader has takes the reader's declared default, and with no default the read fails.
• Protobuf tags each value with a field number instead of a name. A reader skips numbers it does not know (proto3 keeps them as unknown fields), and in proto3 a scalar field it expects but does not find reads as its type's default, zero or an empty string, indistinguishable from a real zero unless the field has explicit presence, as an optional or oneof field does.

So the same change can be safe in one format and breaking in another. Renaming energy_wh breaks a JSON reader that looks for the old name. In Avro, a reader on the new schema can read old events by declaring the old name as an alias, but a reader still on the old schema finds no energy_wh in new ones. Protobuf's binary encoding never carries the name, so only the code and any JSON form of the message notice.

Backward, forward, full

The review question has a name. Compatibility is always between two versions of a schema, and the direction that holds decides the deploy order. In Avro terms:

• Backward compatible: code using the new schema can read data written with the old one. Upgrade the consumers first; they keep reading the old events still in the topic, and the producer switches once they are all done. Adding a field with a default is backward compatible, and so is removing a field.
• Forward compatible: code using the old schema can read data written with the new one. Upgrade the producer first; old consumers skip what they do not know. Adding a field is forward compatible; removing one is forward compatible only when old readers have a default for it.
• Full compatibility is both, between consecutive versions, and lets the two sides deploy in either order. In practice that mostly means adding and removing fields that have defaults.

In Avro, connector with a default of unknown is fully compatible: new billing code reads last week's events by filling in the default, and old billing code skips a field its reader schema lacks. The charging service can ship first. In proto3, which has no declared defaults, a field added or removed keeps both sides decoding as long as its number is never reused, and a missing scalar without explicit presence reads as its type's default. Proto2 differs: it allows declared defaults, and its required fields cannot be safely added or removed.

What breaks, and how it shows

In Avro, add connector with no default instead, and billing, once upgraded, fails on every older event, because there is nothing to fill the field with. Change a field's type, or remove one that readers depend on, and readers on one version misread or reject what the other wrote; the exceptions are narrow, such as Avro's promotion of int to long and Protobuf's interchangeable int32 and int64, where a value too big for the narrower type is truncated. Protobuf has a failure of its own: a field deleted as number 4 and a new one later given 4. Old events still carry the old value under 4, and new code decodes it as the new field, at best a parse error, often a plausible wrong value. Listing retired numbers as reserved makes the compiler refuse the reuse.

The worst change passes every check. Someone decides energy_wh should carry kilowatt-hours. Same name, same integer type, a valid schema: the upgraded producer sends 32 for a 32,000 Wh session, and billing, still reading the field as Wh, charges for 32 Wh. Compatibility checks compare structure, never meaning; a new unit gets a new field name.

A registry that says no

A schema registry is a service that stores every version of each schema and checks new ones against the old. In the Confluent Schema Registry, versions live under a subject, by default named after the topic (charge-sessions-value), and each subject has a compatibility mode. The charging service's serializer registers its schema, gets back a numeric id, and puts that id at the front of every message. Billing reads the id, fetches that writer schema from the registry, caches it, and resolves it against its own reader schema.

Registration is where the check runs. A version that breaks the subject's mode is refused, and a producer that registers on the fly, as the serializers do by default, fails at its first send instead of publishing in a shape the mode forbids. Many teams register from CI instead, so the refusal arrives before the deploy. Confluent's default mode is BACKWARD, which fits a topic whose consumers upgrade first.

The oldest shape comes back

BACKWARD compares a new version with the latest one only. Version 2 added connector with a default. Version 3 drops the default, since every event since version 2 has the field, and the check passes: a version 3 reader can read version 2 data. Then analytics rebuilds two years of charging history from the lake, and a version 3 reader meets 2022 events with no connector and nothing to fill it with. The job fails on the first old file.

The transitive modes (BACKWARD_TRANSITIVE, FORWARD_TRANSITIVE, FULL_TRANSITIVE) check a new version against every earlier one, and BACKWARD_TRANSITIVE would have refused version 3. They suit any subject whose old data is still read: a topic with long retention, a lake, a replay. How such a rebuild is run is in Batch processing: partitions, shuffles & backfills.

When the shape has to break

Billing later wants each session's energy split by tariff period, a list where energy_wh was one number. Often the cheapest way there is no break at all: add tariff_segments beside energy_wh and fill both. When the old field must really go, the break goes into a new contract:

1. The charging team defines a new event type, SessionEndedV2, on a new topic with its own subject.
2. It publishes every session to both topics.
3. Each consumer moves to the new topic in its own release, choosing its switch-over point so it neither skips sessions nor counts any twice.
4. Once every consuming team has confirmed its move and the old topic's consumer groups are retired, publishing to it stops.

Dual publishing is a temporary state, not a design: the two streams disagree if the service writes one and crashes before the other. A consumer that cannot move yet can be fed by a Message Translator that converts between the two shapes in flight. Where old events are the system's own history and cannot be republished, Event Sourcing usually upcasts them as they are read.

A table that keeps changing

In change data capture the event shape follows a source table, so a column added in PostgreSQL usually becomes a new field in the change events that follow (Change data capture traces those events; Schema migrations without downtime handles the table side). The CDC Replication Pipeline challenge requires a schema registry for exactly this. Place it beside Debezium and Kafka, label the subject's compatibility mode, and walk three changes through your design: a nullable column added, a column renamed, a column dropped while week-old events still carry it. For each, decide whether the registry accepts the version and which side deploys first.