Interactive Introduction to CDC
An interactive dashboard covering core concepts, methods, architectures, and the tooling ecosystem.
Series Overview
Change Data Capture marks a fundamental evolution in data integration, moving away from the latent, resource-intensive world of batch processing and into the dynamic paradigm of real-time streaming. By capturing individual data changes as they occur, CDC provides a mechanism to keep disparate systems synchronized with minimal impact and sub-second latency. This technology is a strategic enabler, unlocking real-time analytics and forming the backbone of resilient, modern data architectures.
Start with the fundamentals. Understand what Change Data Capture is and why it’s a cornerstone of modern data architecture through real-world use cases.
CDC Pipeline Architecture
How Change Events Flow
Click any node to highlight its connections. Hover for details.
graph LR
SRC[Source Database] -->|Transaction Log| LOG[WAL / Binlog / Redo]
LOG -->|Read| CONN[CDC Connector]
CONN -->|Emit Events| SMT[Transforms]
SMT -->|Publish| STREAM[Streaming Platform]
STREAM -->|Consume| SINK[Target System]
style SRC fill:#e1f5ff
style CONN fill:#fff3cd
style STREAM fill:#d4edda
style SINK fill:#f8d7da
Interactive: click a node to highlight its path through the pipeline.
Event Envelope & Delivery Guarantees
Keys vs payload, before/after images, tombstones; ALO vs EOS scope and per-key ordering.
Materialization 101 (Upsert/Delete)
Practical MERGE patterns for upserts & deletes; compaction vs history tables; late-arrivals 101.
Snapshotting: The First Sync
Learn how CDC pipelines perform the initial, consistent snapshot of a database before streaming live changes.
Exactly-Once Semantics
Visual walkthrough of ALO vs EOS + transactional outbox.
Multi-Tenancy
Isolation patterns, topic math, and rough egress estimates.
Partitioning
Partition keys, skew, late-arrivals, and audit loops.
Schema Evolution
Handle schema changes gracefully with forward/backward compatibility and schema registries.
Ops: Offsets & Replays
Offset stores, safe rewind, idempotency, and resync drills when things go sideways.
Observability Basics
Golden signals (lag, throughput, error rate), alerting, and minimal dashboards to keep.
CDC Beyond Relational Databases
MongoDB change streams, DynamoDB Streams, and Cassandra CDC — and where each one breaks the WAL/binlog mental model.
Security, PII & Access Control
Mask columns before they reach the broker, size the privileges CDC actually needs, and plan for a log that outlives the row.
Reconciliation & Offset Surgery
Repair out-of-sync sinks and safely reset offsets. SQL diff patterns, checksum verification, and Kafka Connect REST API offset operations.
Real-World Use Cases
Explore practical applications of CDC, from real-time analytics to cache invalidation.
The Strategic Value of CDC
Understand the business case and philosophical shift behind adopting an event-driven data culture.
The CDC Ecosystem
A curated overview of the most popular open-source and commercial tools in the landscape (Debezium, Fivetran, etc).
Real-World Case Study
Follow a mid-sized e-commerce company's journey from batch ETL to real-time CDC, including architecture decisions, implementation challenges, and business outcomes.
Hands-On Lab: Kafka + Debezium + Sinks
Stand up Kafka, Connect, Postgres source & sink with guided copy-paste commands. Includes upsert patterns and schema evolution.
Quickstarts
Pick your source database and follow a 10–20 minute setup with checks and commands.
Acceptance Tests
Run shell scripts that confirm your lab stack is up, the connector is healthy, and events keep flowing after restarts.
Failure Scenario Drills
Hands-on drills to build troubleshooting fluency: backpressure, DLQ handling, schema drift, and offset replays.
Cloud CDC Labs
End-to-end CDC implementations with cloud-native platforms: AWS DMS, Snowflake, and Matillion.
Connector Config Builder
Generate Debezium configs for Postgres, MySQL, or Oracle in minutes.
DLQ Triage Assistant
Guided commands and playbooks for decoding and re-driving Kafka DLQ events.
Debezium Event Decoder
Paste Kafka events to get before/after diffs and MERGE-ready SQL templates.
Nuances & Errata
Corrections, caveats, and sharp edges across CDC: effectively-once vs exactly-once, snapshots & replays, tombstones/compaction, schema evolution, and ops guardrails.