Schema Evolution
Changing the structure of data over time while old and new versions of the code, and old and new data, keep working together.
Also known as: Schema Migration, Schema Versioning, Data Format Evolution, Schema Compatibility
Category: Software Development
Tags: data, databases, software-architecture, compatibility, distributed-systems, software-engineering, data-management
Explanation
Applications change, so the shape of their data changes too: new fields, removed fields, different types. Schema evolution is the practice of making those changes without breaking the programs that read and write the data. It matters because you almost never switch everything at once:
- During a rolling deployment, old and new versions of a service run side by side
- Old clients stay in use for a long time
- Stored data lives longer than the code that wrote it. Data written five years ago is still in the database, in its original shape
**Two directions**:
- **Backward compatibility**: New code can read data written by old code
- **Forward compatibility**: Old code can read data written by new code
Rolling upgrades need both.
**Rules that keep you safe**:
- **Add fields as optional, with a default**: Old data doesn't have them. Old code ignores them
- **Only remove optional fields**: And never reuse the identifier of a removed field
- **Don't change the meaning of a field**: Add a new one instead
- **Be careful with type changes**: Widening can work. Narrowing loses data
- **Preserve unknown fields**: Code that reads a record, updates it and writes it back must not drop fields it doesn't know
- **New enum values are a risk**: Old readers have to cope with values they've never seen
**How formats help**:
- **Protocol Buffers and Thrift**: Fields have numeric tags. Names can change, tags can't. Unknown tags are skipped
- **Avro**: The reader's schema and the writer's schema are compared and resolved when the data is read
- **JSON**: Flexible, with no built-in rules. The discipline has to come from conventions and validation
- **Schema registries**: They check every new schema version against the previous ones and reject incompatible changes before deployment
**For relational databases**:
Migrations change the schema in place. Zero-downtime changes use the expand and contract approach (see Parallel Change): add the new structure, move the data and the code, then remove the old structure.
The Tolerant Reader pattern is the consumer-side half of schema evolution: read what you need and ignore the rest.
Related Concepts
← Back to all concepts