CDC Beyond Relational Databases

What transfers, and what does not

Every method on this site rests on one idea: the database already writes an ordered, durable record of every change, so read that instead of polling the table. That idea is not relational. MongoDB, DynamoDB and Cassandra all keep an equivalent log, and all three expose it.

What does not transfer is the set of guarantees you stopped noticing. Relational CDC hands you a single ordered log per database, a stable primary key, and a before-image on request. Drop any one of those and a pipeline that was correct becomes subtly wrong — usually without an error.

MongoDB: change streams over the oplog

Change streams are the closest analogue to relational CDC, and the easiest migration. They read the replica set’s oplog and hand you an ordered stream with a resume token on every event — the same role an LSN plays elsewhere. Store the token with your sink offsets and a restart continues exactly where it stopped.

// resume tokens are the offset; persist them with the sink write
const stream = db.collection('customers').watch([], {
  fullDocument: 'updateLookup',
  fullDocumentBeforeChange: 'whenAvailable',
  resumeAfter: savedToken
});

Two details catch people. First, an update event carries only the changed fields unless you ask for fullDocument, and updateLookup re-reads the document at lookup time — so it can return a newer state than the one the event described. If you need the exact post-change state, that difference matters.

Second, the oplog is capped by size, not time. A write burst can age out the token you were about to resume from, and there is no warning before it happens — the resume simply fails and you are back to a snapshot. Size the oplog against your worst-case downtime, not your average.

DynamoDB Streams: a hard 24-hour horizon

DynamoDB Streams gives you before and after images directly — set the view type to NEW_AND_OLD_IMAGES and the envelope is essentially complete. Ordering is guaranteed per shard, and because shards map to partition keys, per-item ordering holds. That is the guarantee you actually need.

The constraint is retention: 24 hours, not configurable. There is no equivalent of extending log retention while you fix a sink. A consumer outage that outlasts a long weekend is not a backlog to work through — it is a gap, and the only repair is a fresh full scan of the table.

Cassandra: the one that breaks the model

Cassandra CDC is the case worth studying, because it violates the assumption everything else on this site is built on: there is no cluster-wide ordered log. CDC is enabled per table and writes to a per-node commitlog directory. Each node records the mutations it personally accepted.

With replication factor 3, that has consequences:

• The same write appears on multiple nodes. Duplicates are structural, not the result of a retry.
• No cross-node ordering exists. There is no sequence number to sort by across the cluster, so “order by log position” has no cluster-wide meaning.
• Events are mutations, not row states. A write records the columns it touched. There is no before-image, and no complete after-image unless the write happened to set every column.

So the usual sink recipe — upsert on the primary key, order by log position — does not port. What ports instead is Cassandra’s own conflict rule: last write wins by cell timestamp. Every mutation carries per-cell write times, and merging on those reproduces the source’s semantics, deduplicating the replicas as a side effect. It is the one place on this site where a timestamp is the correct ordering key — because here it is a logical clock the database assigns, not a wall-clock guess like ts_ms.

Before you port a pipeline

• Identify what plays the role of the LSN — resume token, shard sequence number, cell timestamp — and confirm your sink orders on it.
• Find the retention limit and express it in time you can be down, then alert well inside it.
• Confirm whether you get a before-image, and whether your consumers actually need one.
• Check that the source has a stable key. Idempotent upserts need one just as much here as they do on Postgres.
• Assume at-least-once. None of these transports changes that, and none of them removes the need for an idempotent sink.