Data Contracts for Database Events

A schema nobody designed

When a service publishes an event through an API, someone decides which fields exist and what they mean. In change data capture the producer is a database table. The fields of an event are the table's columns at the moment of the change, so every ALTER TABLE is also an edit to a published schema, made by whoever ran the migration, often without a consumer in mind. The schema is a side effect of DDL, not a deliberate interface.

Confluent's data-contracts documentation defines a data contract as a formal agreement between an upstream and a downstream component on the structure and semantics of data in motion, and says a schema is only one element of it: structure, integrity constraints, metadata, rules, and a way to change over time. This page applies that idea where the upstream component is a database you may not control, and stays on the parts that are specific to CDC: what the envelope promises, what each kind of DDL does to it, how to enforce it, and who owns it. It builds on Schema Evolution (registry modes, subject naming, rollout) and Event Envelope (the fields one by one) and does not repeat them.

What a contract does not do. It describes shape and meaning. It does not change delivery. Debezium documents at-least-once delivery, so a compatible schema can still be delivered twice, late, or replayed. The sink still upserts on the primary key and applies a change only if its source log position is newer. Nothing on this page makes a hop exactly-once; see Is CDC exactly-once, per hop.

As of: 2026-10-09, against the Debezium 3.7 documentation (the “stable” pages), the Confluent Platform Schema Registry documentation, and the Avro 1.12.0 specification. Setting names and defaults move between releases; where a setting below matters to you, check your version's page.

The contract surface: what the envelope promises

A Debezium change event has a key and a value. The value carries before, after, source, op and timestamps. Your consumers depend on some of that by design and on the rest by accident. Write down which is which. The table uses the PostgreSQL connector page; other connectors differ in the position fields, so read yours.

Two ways the contract changes without a migration

• Connector settings. The PostgreSQL page maps temporal types through time.precision.mode and decimals through decimal.handling.mode, so changing either changes the field type consumers see. field.name.adjustment.mode and schema.name.adjustment.mode (both none by default) can replace characters that Avro names do not allow, so a column called first-name may reach consumers under a different field name depending on them.
• Connector upgrades. Debezium's page warns that with include.unknown.datatypes=true the binary representation of an unsupported type may change between releases, and that once a type is supported it is sent as a logical type, which “would require adjustments by consumers”. Treat the connector version as part of the contract, and run the contract test (below) before an upgrade reaches production.

Which DDL breaks which consumer

The registry cannot see your DDL; it sees the schema the converter derives from it. Debezium's nullable columns become optional Connect fields, and Confluent's Avro converter writes an optional field as a union with null and a default of null (AvroData.java, read on 2026-10-09; check the converter version you run). That makes “nullable column” mean “optional field” to the registry, and the rest follows from Confluent's compatibility table. The table below uses Avro. Protobuf and JSON Schema follow different rules (Confluent's table shows widening, for one, accepted differently in Protobuf), so check your format.

Sources and what was checked. Rows 1, 3, 4, 7 and 8 are rows of Confluent's Avro table (“add optional field”, “remove optional field”, “remove required field”, “widen a scalar type”, “narrow a scalar type”). The others are derived: rows 2, 5 and 6 by combining rows, the type-change row from the Avro schema-resolution rules (numeric promotion goes int to long to float to double, and string and bytes promote to each other, so a PostgreSQL text to bytea change passes every mode; there are no other promotions, so int to text is rejected), and the nullability rows from the table's union-variant rows, because Confluent says Avro expresses nullability as a union with null. This repository's tests check every cell above against a small model of the Avro resolution rules, and the model reproduces all 24 cells of Confluent's Avro table. That is a check of the page against the documentation, not a run of a real Schema Registry. The registry's own check is the authority for your subject.

The rename is the dangerous one

A nullable column renamed is accepted in every mode, because to the registry it is a removed optional field and an added optional field, both of which are always allowed. The connector keeps running, the registry stays green, and every consumer that reads email now reads nothing while email_address fills with data nobody asked for. A sink that stores what it reads keeps its old column, which stops changing, and ignores the new one. No exception is raised. Only a contract test on the consumer's side (below) or a reconciliation check on the data notices.

Avro has aliases for exactly this, but the specification describes them as a reader-side mapping: a reader's schema names the old name as an alias. The Debezium pages I read do not describe the connector emitting aliases. Treat a rename as a drop plus an add and migrate with the sequence in the versioning section.

Inference, not a measurement: that a rename reaches consumers this way follows from the event being built from the table's current columns and from the absence of a rename field in the envelope on the connector pages. It was not run against a live database for this page. SQL Server is different, below.

Registry modes, and which one fits a CDC topic

Schema Evolution explains the modes. Here is the part that matters for a database producer, from Confluent's documentation.

• Why BACKWARD is the default. Confluent says it is so you can rewind consumers to the beginning of the topic: a reader on the new schema can read old data. CDC topics get replayed (a new sink, a re-snapshot, an offset reset), so this fits. Confluent also notes that Protobuf's best practice is BACKWARD_TRANSITIVE.
• Non-transitive lets drift through. Suppose a table gets ADD COLUMN tier text NOT NULL DEFAULT 'standard' and later ALTER COLUMN tier DROP DEFAULT. Version 2 can be read by a version 3 reader, because version 2 data contains tier, so BACKWARD accepts version 3. But version 1 data has no tier and version 3 has no default to supply, so a replay from the start of the topic fails on the oldest records. BACKWARD_TRANSITIVE rejects version 3 at registration. This is the model's result for the schemas as described; it assumes the connector propagates the dropped default, which the PostgreSQL page says it tries to do.
• BACKWARD puts consumers first. That order is easy to say and hard to get when the DDL is run by a team that does not know your consumers. It is the reason for the process section below. FULL removes the ordering constraint at the price of rejecting more changes (see the table above).
• Per table. With the default subject naming each table's topic has its own subject, so compatibility is enforced and set per table. Set the mode on purpose per subject (PUT /config/<subject>), not by inheriting the global one.
• Where it bites. With auto.register.schemas at its default of true, the serializer tries to register a schema it has not seen as it serializes a record that uses it. The API reference lists 409 Conflict (“Incompatible schema”) for a register call, so an incompatible schema is refused, the converter throws, and the connector task fails (next sections).

Confluent's data-contracts feature goes further: tags, ownership metadata, domain rules and migration rules that transform data between versions. The documentation says rules need Confluent Platform Enterprise (7.4 or later) or Confluent Cloud with the Advanced Stream Governance package, and that Kafka Connect on Confluent Cloud does not run rules. Nothing on this page depends on it.

How Debezium learns about DDL, by engine

How a connector finds out that a table changed differs by database, and so does what breaks.

• MySQL: the schema history topic. The MySQL page says the binlog contains DDL, the connector parses it, and it records every DDL statement with its binlog position in a separate schema history topic (schema.history.internal.kafka.topic, schema.history.internal.kafka.bootstrap.servers) so it can rebuild each table's structure at the point where it restarts. The page says that topic is for the connector's internal use only, that it must have a single partition, and that you should not partition it. If it is lost or corrupted, snapshot.mode=recovery rebuilds it from the source tables, but the page warns: “Do not use this mode to perform a snapshot if schema changes were committed to the database after the last connector shutdown.” If DDL ran while the connector was down, the recovered history would describe the new tables for older binlog events; the page does not give an alternative in that warning, so check your version's page for the supported route before you rely on recovery after DDL. Consumers that want to be told about DDL should read the separate schema change topic, controlled by include.schema.changes (default true). The page also says the format of that topic is “in an incubating state and is subject to change without notice”, so it is a signal, not a contract to parse.
• PostgreSQL: no DDL in the stream. The page says logical decoding does not support DDL changes, so the connector cannot report them to consumers. It refreshes its schema from the database when it detects a change during streaming, and says pgoutput is more eager about it than the other plug-ins (a default change triggers a refresh with pgoutput but not with others). There is no schema history topic to read or to lose, and also no DDL event to react to: the first sign of a change is a different shape of event.
• SQL Server: capture instances. The SQL Server page says that, because of how SQL Server CDC works, a Debezium user cannot update the capture tables and has to coordinate with a database operator, who creates a new capture instance after the schema change (offline or online). It also says SQL Server does not allow some schema changes on a table with CDC enabled, naming a column rename and a column type change, and that after a change between NULL and NOT NULL the emitted records do not show the change until a new capture instance exists. For that engine the DDL runbook is part of the contract.

Shape the contract at the source: column lists

column.include.list captures only the columns you name, and column.exclude.list captures every column except the ones you name; set one, not both. Key columns are always included in the key. An include list turns the contract into an allow-list: a column added to the table later does not appear in events until you add it to the list, which is a useful property for a column that might hold personal data (see Security & PII). An exclude list does the opposite, and a new column flows to consumers on its own. On PostgreSQL, skip.messages.without.change only applies when the table's replica identity is FULL.

Shaping the contract on the way out: SMTs and the outbox

A single message transform sits between the database's shape and the consumer's. Two are worth knowing, and they pull in opposite directions.

Flattening with ExtractNewRecordState

The event flattening SMT replaces the envelope with the after row. That is a simpler contract and a smaller one: op, before and source are gone unless you ask for them. Debezium's page shows add.fields=op,table,lsn,source.ts_ms producing __op, __table, __lsn and __source_ts_ms. If your sink orders by source position, put the position into the flattened record on purpose. On PostgreSQL the position is absent for incremental-snapshot reads (see the source position row above), so __lsn is absent or null for those and the sink needs a rule for them:

transforms=unwrap
transforms.unwrap.type=io.debezium.transforms.ExtractNewRecordState
transforms.unwrap.add.fields=op,lsn
transforms.unwrap.delete.tombstone.handling.mode=rewrite

With rewrite the SMT keeps the delete as a record with __deleted: true (and removes the tombstone), which keeps a delete marker a position-guarded sink can use. The option's default is a version trap: in the 3.7 page the options table gives delete.tombstone.handling.mode a default of tombstone, while the “Behavior” paragraph of the same page still says the SMT drops delete records by default. Do not rely on a default; set it, and check your version. drop removes both the delete event and the tombstone, and a target that never receives a delete never removes the row. Metadata for a delete is added only with rewrite. The Debezium page recommends applying the SMT selectively with a predicate, because heartbeat, schema-change and transaction messages have a different structure.

The outbox event router

The outbox pattern turns the problem round: the contract is a table you designed for the purpose, not your domain tables. Debezium's router assumes an outbox table with id, aggregatetype, aggregateid, type and payload columns (the page lists options for taking them from other columns). The router puts aggregateid in the message key, the id in a header (the page suggests using it to remove duplicates), and routes by aggregatetype. Its documentation expects outbox rows to be inserts only: an update is handled according to table.op.invalid.behavior (warn by default; the other settings are error and fatal), and deletes are filtered out. Your domain tables can then change without the event changing, and the payload and type become the versioned contract. Two caveats from the page: the routed message in its example carries a key, an id header and the payload, not the source block, and table.fields.additional.placement adds columns of the outbox table, not the log position. So a sink that must order by log position has no position to read unless your application writes a per-aggregate sequence into the outbox row. The delivery side of the outbox is the subject of Exactly-Once Semantics.

Evolve the topic or version it

Evolve in place when the change is compatible with every consumer you are willing to break, which for a long-retention topic means the transitive modes. For a change the registry rejects, Confluent says to upgrade all producers and consumers at once or, more likely, to create a new topic and move applications to it, rather than hold two incompatible versions in one topic. For a database producer the practical version of that is expand and contract:

1. Expand. Add the new column as nullable (accepted in every mode). Consumers ignore it.
2. Backfill. Populate it from the old column. Every updated row is a change event, so a big backfill is a burst of traffic on the topic; do it in batches.
3. Move consumers to read the new field, one at a time, each checking its own contract test.
4. Wait out the replay window. Old events stay in the topic for the retention period. A consumer that rewinds will read records with the old shape.
5. Contract. Drop the old column only when nothing reads it, including any consumer that could replay history.

A rename is steps 1 to 5 with a copy in the middle. If the change cannot be expanded (a type change to an unrelated type), the new topic is the honest answer, and a re-snapshot on the new topic is part of the cost.

A DDL gate you can run

Testing a CDC pipeline shows a contract test that checks each recorded event against the envelope. The gate below is the other half: it runs when someone proposes a DDL change, before it ships. It compares the field list of the after struct before and after the change (the shape of schema.fields in a Debezium JSON-with-schema record) with the columns one consumer reads and assumes. It is a consumer-driven check: it does not ask whether the schema is valid in general, it asks whether this consumer still works. Everything in it is deliberately small.

// ddl-gate.mjs  (Node 20+, no dependencies)  run: node ddl-gate.mjs
// Compares the "after" struct of a Debezium JSON-with-schema record before and
// after a proposed DDL change with the columns one consumer actually reads.
// A field is { field, type, optional, default? }, as in the Connect schema.
const before = [
  { field: "id", type: "int32", optional: false },
  { field: "email", type: "string", optional: false },
  { field: "name", type: "string", optional: true },
  { field: "tier", type: "string", optional: false, default: "standard" },
];

// What this consumer reads, and what it assumes about each column.
const reads = {
  id: { type: "int32", nullable: false },
  email: { type: "string", nullable: false },
  name: { type: "string", nullable: true },
  tier: { type: "string", nullable: false },
};

function gate(beforeFields, afterFields, consumerReads) {
  const byName = (fields) => new Map(fields.map((f) => [f.field, f]));
  const was = byName(beforeFields);
  const now = byName(afterFields);
  const added = [...now.values()].filter((f) => !was.has(f.field));
  const findings = [];
  for (const [column, want] of Object.entries(consumerReads)) {
    const f = now.get(column);
    if (!f) {
      // A rename reaches the consumer as one field gone and another present.
      const twin = added.find(
        (a) => a.type === was.get(column).type && !(a.field in consumerReads),
      );
      const hint = twin ? `; "${twin.field}" is new with the same type (rename?)` : "";
      findings.push(["BREAK", column, `removed${hint}`]);
      continue;
    }
    if (f.type !== want.type) {
      findings.push(["BREAK", column, `type ${want.type} -> ${f.type}`]);
    } else if (f.optional && !want.nullable) {
      findings.push(["BREAK", column, "can now be null; the consumer assumes it cannot"]);
    }
    const old = was.get(column);
    if (old.default !== f.default) {
      findings.push(["WARN", column, `default ${old.default} -> ${f.default}`]);
    }
  }
  for (const f of added) {
    if (!(f.field in consumerReads)) findings.push(["OK", f.field, "added, not read"]);
  }
  return findings;
}

const rewrite = (fields, change) => change(structuredClone(fields));
const scenarios = {
  "add nullable phone": (f) => [...f, { field: "phone", type: "string", optional: true }],
  "drop name": (f) => f.filter((c) => c.field !== "name"),
  "rename email": (f) => f.map((c) => (c.field === "email" ? { ...c, field: "email_address" } : c)),
  "widen id to bigint": (f) => f.map((c) => (c.field === "id" ? { ...c, type: "int64" } : c)),
  "email becomes nullable": (f) => f.map((c) => (c.field === "email" ? { ...c, optional: true } : c)),
  "change tier default": (f) => f.map((c) => (c.field === "tier" ? { ...c, default: "basic" } : c)),
};

let broken = 0;
for (const [name, change] of Object.entries(scenarios)) {
  const findings = gate(before, rewrite(before, change), reads);
  broken += findings.filter(([level]) => level === "BREAK").length;
  console.log(`${name}:`);
  for (const [level, column, message] of findings) console.log(`  ${level.padEnd(5)} ${column}: ${message}`);
}
process.exitCode = broken > 0 ? 1 : 0;

Run it with node ddl-gate.mjs. The six scenarios are the script's own, a sample of the rows in the DDL table above. Expected output, and an exit code of 1 because at least one scenario breaks:

add nullable phone:
  OK    phone: added, not read
drop name:
  BREAK name: removed
rename email:
  BREAK email: removed; "email_address" is new with the same type (rename?)
  OK    email_address: added, not read
widen id to bigint:
  BREAK id: type int32 -> int64
email becomes nullable:
  BREAK email: can now be null; the consumer assumes it cannot
change tier default:
  WARN  tier: default standard -> basic

This repository's test runs the script exactly as printed and compares the output. The rename hint is a heuristic (a removed field and an added field of the same type), and it is a hint. The script never talks to a database or a registry.

Getting the “after DDL” schema

The hard part is the second field list. Two honest options. Exact: apply the migration to a scratch database with a connector attached, record the first event, and take the schema from it. Cheap: derive it from the migration by hand, as the script's scenarios do, and accept that the connector's type mapping could differ from your reading. A registry check complements either:

# Written from the Schema Registry API reference; not run against a registry for this page.
# candidate.json is {"schema": "<the proposed Avro schema, as an escaped string>"}
curl -s -X POST \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data @candidate.json \
  "http://localhost:8081/compatibility/subjects/dbserver1.public.customers-value/versions?verbose=true"
# {"is_compatible": false, "messages": [...]}   (messages only with verbose=true)

The command is written from the API reference and was not run against a registry. The reference describes this endpoint as performing the same check as a register call for that subject, using the subject's configured level (or the global level if the subject has none): against the latest version for the non-transitive modes, and against all earlier versions for the transitive ones. To make the registry the gate rather than the converter, the Confluent converter documentation describes auto.register.schemas=false with use.latest.version=true: the serializer then uses the latest registered version instead of registering one, and while latest.compatibility.strict stays at its default (true) it raises an error if that version is not backward compatible with the schema of the object being serialized. That was not tried with a Debezium connector here; check how your converter version behaves before you depend on it.

Who owns it: the process around DDL

Confluent's page says the upstream component enforces the contract and the downstream component can assume the data conforms. For a database producer, upstream is the team that runs the migrations. The points below are practice, not a documented rule, and are where a contract usually breaks.

What goes wrong, and what catches it

Every row above is about shape. Duplicates and late records are a separate failure, covered by the sink: the same change delivered twice, or an older change arriving after a newer one, is harmless to a sink that upserts on the primary key and applies a change only if its source log position is newer. A schema contract does not replace that, and a stricter registry mode does not remove it.

See a schema change

The Change Feed Playground has a Schema Evolution scenario with a schema walkthrough that adds or drops a column while events stream. From its source, it has add and drop buttons only: no rename and no registry. It shows that a column appears or disappears in the events; it does not show a compatibility check or a failed connector, and it was not run for this page. The Schema Evolution page links the same scenario.

Further resources

• Schema Evolution: registry modes, subject naming, rollouts, consumer defenses.
• Event Envelope: schema contracts and validation, and Strategy for where contracts sit in an operating model.
• Testing a CDC pipeline: contract tests, and Is CDC exactly-once, per hop.
• Glossary: schema registry, schema evolution, SMT, outbox, tombstone, dead letter queue, at-least-once, convergence test.
• Confluent: Schema evolution and compatibility: types, the Avro and Protobuf table, transitive, upgrade order.
• Confluent: Data contracts: what a contract contains, rules, requirements.
• Confluent: Schema Registry API reference: the compatibility and config endpoints, the 409 on register.
• Confluent: Schema Registry and Kafka Connect: auto.register.schemas, use.latest.version.
• Confluent: Kafka Connect error handling and dead letter queues.
• Kafka Connect user guide: error reporting in Connect.
• Avro 1.12.0 specification: schema resolution, aliases.
• Debezium PostgreSQL connector, MySQL and SQL Server: events, schema history, schema changes.
• Debezium event flattening and outbox event router.
• Debezium: Avro serialization.