Postgres Replication Slots & WAL Growth

What this runbook covers

You run Postgres with a logical-decoding CDC connector (this page uses Debezium's Postgres connector as the example). The disk on the primary is filling, or an alert says it might, and you need to know whether a replication slot is the cause, what each fix costs, and what the pipeline owes you afterwards.

The thread through all of it is the same as the rest of this site. A slot is a position held on the source. The consumer moves that position forward only by confirming it has received changes, so delivery is at-least-once. Anything that gives up or loses the position (a drop, an invalidation) means the only correct way back is a re-snapshot into a sink that is idempotent: keyed by primary key and ordered by log position. There is no end-to-end exactly-once at any point in this runbook.

Triage: is a slot why the disk is filling?

Connect to the primary (not a replica) and look at every slot, logical and physical. pg_current_wal_lsn() cannot run during recovery, so on a standby use pg_last_wal_replay_lsn() in its place.

-- Every slot, biggest WAL pin first. Needs PostgreSQL 13+ (wal_status, safe_wal_size).
-- A lost slot pins nothing, so its restart_lag is shown as NULL and it sorts last.
SELECT slot_name,
       slot_type,
       plugin,
       database,
       active,
       active_pid,
       wal_status,
       pg_size_pretty(safe_wal_size) AS safe_wal_size,
       pg_size_pretty(CASE WHEN wal_status <> 'lost'
                           THEN pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) END) AS restart_lag,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), confirmed_flush_lsn)) AS unconfirmed,
       restart_lsn,
       confirmed_flush_lsn
FROM pg_replication_slots
ORDER BY CASE WHEN wal_status <> 'lost'
              THEN pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) END DESC NULLS LAST;

Careful with lost slots: a lost slot pins nothing, whatever restart_lag would show. With invalidation_reason = 'wal_removed' its restart_lsn is NULL (observed, see the last section), but with other reasons (for example idle_timeout on PostgreSQL 18) it can stay set. A NULL lag therefore means “this slot has already failed”, not “nothing is retained by design”. Read wal_status (and, on 17+, invalidation_reason) before the lag columns.

Next, confirm the disk is really WAL, and look for the other thing a slot pins. Reading pg_ls_waldir() needs a superuser or a role with pg_monitor privileges.

-- Size of the WAL directory
SELECT pg_size_pretty(sum(size)) AS pg_wal_size FROM pg_ls_waldir();

-- The same slots also hold back VACUUM on catalog rows
SELECT slot_name,
       age(xmin)         AS xmin_age,
       age(catalog_xmin) AS catalog_xmin_age
FROM pg_replication_slots;

-- PostgreSQL 17+ only: when did it go idle, and was it invalidated?
SELECT slot_name, inactive_since, invalidation_reason, failover, synced
FROM pg_replication_slots;

If pg_wal_size is far larger than max_wal_size and a slot's restart_lag accounts for most of it, a slot is the cause. If no slot explains it, look at WAL archiving and at physical slots for standbys before blaming CDC (the first query lists those too).

How a logical slot holds WAL

A logical slot represents a stream of changes that can be replayed to a client in the order they were made. It lives on one database, persists independently of whichever connection is using it, and is crash-safe. Crucially it knows nothing about its consumer. Only one receiver can consume from it at a time, but the server does not care whether that receiver is alive.

1. The consumer connects and starts streaming. The server begins at the greater of the position the client asks for and the slot's confirmed_flush_lsn, so a client cannot read behind its own confirmed position.
2. As it processes changes, the consumer sends status updates that report the WAL position it has flushed. That is how confirmed_flush_lsn moves forward. restart_lsn follows, possibly a little behind it, and that is what lets the server recycle older WAL. (From PostgreSQL internals rather than a docs sentence I can cite: in practice a long-running open transaction can hold restart_lsn back, because decoding must be able to restart from the start of every transaction not yet decoded.)
3. If the consumer stops confirming, nothing else moves. Both positions stay where they were, and every new WAL file written from restart_lsn onward is kept.

Two details matter for the rest of this page. First, the slot's current position is persisted only at checkpoints, so after a server crash the slot can return to an earlier position and recent changes are sent again; the PostgreSQL docs say logical decoding clients are responsible for handling the same message more than once. Second, the confirmation is about receipt by the consumer. It says nothing about whether the sink applied the change.

Why an unconsumed or inactive slot fills the disk

The PostgreSQL docs put it bluntly: replication slots persist across crashes and prevent removal of required resources even when there is no connection using them. The WAL is not recycled, so the disk fills at the rate the database writes. The usual routes into that state:

• The connector is stopped, paused, crash-looping or deleted but its slot remains. With Debezium's defaults the slot survives a graceful stop (slot.drop.on.stop is false), so a connector you shut down on Friday is still pinning WAL on Monday.
• The connector is running but slower than the write rate. The slot is active, unconfirmed keeps growing, and restart_lsn trails.
• Most of the database's writes are not on captured tables. The connector reads the log as usual but rarely emits a change record, so no offset is committed and it never gets the chance to send the latest LSN back. The database keeps WAL for events the connector has already read. Debezium documents this case and the heartbeat fix below.
• The captured database is the quiet one in a busy cluster. Slots are per database, but WAL is shared by all databases in the cluster. If only another database is busy, the connector is never invoked, cannot confirm a position, and WAL grows until the captured database emits an event.
• Managed idle environments. Debezium's docs note that on AWS RDS for PostgreSQL, writes to RDS's own system tables that are invisible to clients can leave a nominally idle database in the same situation. Regular events solve it. Check your provider's behaviour.
• The slot is abandoned. A test connector, a retired pipeline, a trial of a managed ELT product. Nobody remembers it exists.

WAL is not the only cost. The same caution in the PostgreSQL docs says a slot also prevents VACUUM from removing the catalog rows it still needs, and that in extreme cases the database can shut down to prevent transaction ID wraparound. That is why the triage includes age(catalog_xmin).

Capping retention: max_slot_wal_keep_size

max_slot_wal_keep_size is the maximum size of WAL that replication slots are allowed to retain in pg_wal at checkpoint time. The default is -1, which means unlimited: slots may retain any amount of WAL. A value without units is read as megabytes. The setting exists in PostgreSQL 13 and later and the docs for 13, 16, 17 and 18 describe it the same way.

With a limit set, if a slot's restart_lsn falls behind the current position by more than the limit, the server stops protecting that WAL. The slot moves through unreserved to lost, and the consumer can no longer continue from it because the WAL it needs has been removed. The docs word it as the consumer possibly “no longer [being] able to continue replication due to removal of required WAL files”. In PostgreSQL 17 and later invalidation_reason says wal_removed.

What a limit trades

A limit trades disk for a lost slot. With no limit, a stuck consumer can fill the volume and stop the primary from writing, which takes the application down with it. With a limit, the primary survives, and the cost moves to the pipeline: the slot is invalidated and the only correct recovery is a re-snapshot. Both are real costs. A limit decides which one you pay.

So size it as policy, not as a number copied from a blog post: it should be large enough to cover the longest consumer outage you intend to survive at your peak WAL rate, and small enough that the volume still has headroom for normal WAL, checkpoints and archiving. Both inputs (write rate, disk size) are yours. Treat safe_wal_size as the countdown to lost and alert well before it reaches zero.

Newer safety nets, by version

The docs say max_slot_wal_keep_size can only be set in postgresql.conf or on the server command line. On PostgreSQL 18.3, pg_settings.context for it is sighup, which means a reload applies it; check the same column on your version and on a managed service, where the setting may live in a parameter group.

Debezium specifics

-- Heartbeat table for a quiet database (the connector user needs rights on it)
CREATE TABLE IF NOT EXISTS cdc_heartbeat (id int PRIMARY KEY, beat timestamptz);
ALTER PUBLICATION dbz_publication ADD TABLE cdc_heartbeat;  -- only for a publication that is not FOR ALL TABLES
-- With publication.autocreate.mode=filtered, also list public.cdc_heartbeat in table.include.list

{
  "heartbeat.interval.ms": "10000",
  "heartbeat.action.query": "INSERT INTO cdc_heartbeat (id, beat) VALUES (1, now()) ON CONFLICT (id) DO UPDATE SET beat = EXCLUDED.beat"
}

The interval above is an example, not a recommendation: pick it from how long you can tolerate WAL being pinned on a quiet database. The queries and snippets on this page were checked for syntax where noted in the last section; the connector JSON was not run against a live Debezium.

What happens to confirmed_flush_lsn while the connector is down

Nothing. It stays at the last position the connector confirmed, and restart_lsn stays with it, so WAL accumulates for the whole outage. Heartbeats do not help here, because there is no connector to send them. When the connector restarts it asks for its stored offset, and the server streams from the greater of that and the slot's confirmed_flush_lsn. Changes the connector had already published to Kafka but not yet confirmed come around again. Debezium states this plainly: it provides at-least-once delivery and a recovering connector might emit duplicates.

Failover and standbys

A slot lives on one server. What happens to it when the primary fails depends on the PostgreSQL version.

Failover slots are not a setting you flip on a slot. For synchronisation to work on PostgreSQL 17 and later, the PostgreSQL docs require:

• sync_replication_slots = on on the standby (it is off by default), a physical slot between primary and standby named in primary_slot_name, hot_standby_feedback enabled, and a valid dbname in primary_conninfo.
• Strongly recommended: list that physical slot in synchronized_standby_slots on the primary, so a subscriber cannot consume changes faster than the standby. The price is latency: logical senders wait for the standby to confirm, and the primary will not shut down completely until the listed standbys confirm.

Caveats worth knowing before you trust it:

• The standby persists a synchronised slot only if the WAL and catalog rows it needs are still available on the standby. Otherwise it logs that synchronisation could lead to data loss and does not persist the slot. Temporary synchronised slots cannot be used for logical decoding.
• Verify readiness before you rely on it. The PostgreSQL docs give this check on the standby (synced and not temporary and not invalidated):

  SELECT slot_name,
         (synced AND NOT temporary AND invalidation_reason IS NULL) AS failover_ready
  FROM pg_replication_slots
  WHERE synced;

• A slot with no active consumer will not advance, which stalls synchronisation; the docs suggest advancing it manually on the primary.
• If the old primary comes back during promotion, subscribers can keep talking to it. The docs recommend disabling subscriptions before promotion and re-enabling them once the connection string points at the new primary.
• Synchronisation is periodic. My reading, not a documented guarantee: the position on the promoted standby can be behind what the consumer had confirmed, so expect some replay after a failover. That is the same at-least-once rule, not a new failure mode.
• Debezium: do not drop or re-create a failover slot, and pause the connector until you have verified the slot is intact. Some managed services (Debezium names AWS RDS and Google Cloud SQL) replicate the slot with disk replication instead, so check what yours does.

Decision table: what is wrong, and what to do

Take two samples of the triage query, far enough apart that pg_current_wal_lsn() has clearly advanced, and read them against this table. Dropping a slot is the last row you should reach for, not the first.

Do not restart the connector against a fresh slot while keeping the old offset. Debezium's own upgrade procedure explains why: a new slot can only return changes from after it was created, so when the connector asks for the stored offset it can skip the existing change events and resume from the newest position, silently, with no records for what it skipped. Debezium documents a manual procedure for a rebuilt slot (rename the connector, remove old offsets, recreate the slot and publication, choose a snapshot mode); follow the version of that procedure that matches your Debezium.

Dropping an abandoned slot safely

1. Identify the owner. Match slot_name, plugin and database against your connector configs (slot.name), your managed ELT products and your runbooks. If you cannot find an owner, ask before dropping.
2. Stop the consumer first. Dropping an active slot raises an error by default; the replication-protocol command has a WAIT option that waits instead. Check active = false.
3. Write down what you are giving up. Save the row from the triage query (restart_lsn, confirmed_flush_lsn, wal_status). It is the evidence of how far the pipeline had got.
4. Drop it.

   SELECT pg_drop_replication_slot('my_abandoned_slot');

   The function drops a physical or logical slot and is the same as the replication command DROP_REPLICATION_SLOT. Re-run the triage query to confirm it is gone, then watch the WAL directory size over the following checkpoints rather than expecting an instant change.
5. Clean up the consumer side. If the pipeline is retired, delete the connector and its offsets, drop its publication and heartbeat table, and remove it from monitoring. If the pipeline continues, you are in the invalidated-slot row: the old offset is not safe to reuse, and a re-snapshot is the way back.

Dropping is a decision about the pipeline's history, not a rule you apply at a given size. “Drop the slot when it exceeds N GB” is how a live pipeline ends up in the invalidated-slot row.

After a restart: replay, duplicates and the idempotent sink

The slot confirms what the connector received. Debezium publishes to Kafka and Kafka Connect commits offsets on its own schedule; Debezium's docs tie the two together, noting that when no offsets are committed the connector has no opportunity to send the latest LSN to the database. So there is always a window of changes that were delivered downstream but not yet confirmed to the slot. A restart, a crash (the slot may return to its last checkpointed position) or a failover sends them again. Debezium documents this as at-least-once delivery.

The fix lives in the sink, not in Postgres:

• Key the target on the primary key.
• Carry the change's log position (for Postgres, Debezium's source.lsn) into the target, and apply a change only when its position is newer than the one stored for that key. A replay then does nothing, and an older change arriving after a newer one cannot overwrite it. Do not order by an event timestamp.
• Keep deletes as delete markers with their own log position, so a replayed insert cannot resurrect a deleted row.

The worked patterns are on Partitioning: version guards and watermarks, Materialization 101 and Snapshotting: idempotent apply. They matter doubly after slot loss: a re-snapshot is the one time rows from two different eras meet in the sink, and a position guard is what lets the snapshot and the stream converge. The related glossary entries are idempotent write and effectively-once.

Try a re-insert in the playground

The Change Feed Playground has a Re-insert after Update template. It is not a replay: an earlier version of a row is written again as an ordinary later source write, and the sink applies it in order. No old event is delivered a second time, so it cannot show a stale change losing to a newer one; for that, see the guarded-apply examples above. Pick it from the template list to see how each capture method reports that write. The playground is a simulation with made-up data; it has no Postgres slots.

Alerting: policy, not magic numbers

Any threshold quoted without the database's write rate and the volume's free space is a guess. Write the alerts as rules, then derive the numbers from your own measurements. To measure the write rate, take the WAL position twice and difference it:

-- psql: remember the position now ...
SELECT pg_current_wal_lsn() AS t0 \gset
-- ... and later, bytes of WAL written since then
SELECT pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), :'t0')) AS written_since_t0;

Wire these into the dashboards and routes described on Observability Basics, and link each alert to the matching row of the decision table above.

What was checked, and what was not

• Read in the primary docs on 2026-10-09: the PostgreSQL pages for pg_replication_slots (14, 16, 17, 18), replication configuration (13, 16, 17, 18), logical decoding concepts (17), the replication protocol (17), the administration functions (17), logical replication failover (17) and the PostgreSQL 17 release notes, plus the Debezium PostgreSQL connector reference (stable).
• Run: the monitoring queries were executed on PGlite 0.5.8, which embeds PostgreSQL 18.3, with wal_level = logical and a real pgoutput slot that nothing consumed. They parse, plan and return the columns shown. On that instance, with max_slot_wal_keep_size = 32MB, writing about 50 MB of WAL took the slot from reserved (safe_wal_size 41 MB) to unreserved (safe_wal_size had gone negative, -10048 kB), and a CHECKPOINT then moved it to lost with NULL restart_lsn and safe_wal_size and invalidation_reason = 'wal_removed'. Peeking at the lost slot failed with “can no longer access replication slot”, and pg_drop_replication_slot removed it. With idle_replication_slot_timeout = 1s an unused slot became lost with invalidation_reason = 'idle_timeout' after a CHECKPOINT, and its restart_lsn stayed set; the triage query shows a NULL restart_lag for it. The heartbeat table DDL, the action query and ALTER PUBLICATION ... ADD TABLE also ran.
• Not run: an actively streaming consumer. Decoding pending changes in PGlite (pg_logical_slot_peek_binary_changes with pgoutput) returned nothing and the instance then stopped returning rows, so active and active_pid were only checked as columns and never seen as true; PostgreSQL 14 to 17 behaviour (docs only), failover slots and synchronisation, the Debezium JSON, whether Debezium's filtered mode really removes a manually added heartbeat table, and any managed service (RDS, Cloud SQL) or the Fivetran product. Settings that depend on your Debezium release (lsn.flush.mode, slot.failover) are marked “check your version” for that reason.

Further resources

On this site

• Ops: Offsets & Replays for stored offsets and safe rewinds.
• Snapshotting for the re-snapshot after a lost slot.
• Backfill and Re-Snapshot Safely for choosing a reload and sweeping the deletes a snapshot cannot carry.
• Reconciliation & Offset Surgery for repairing the sink afterwards.
• Connector Config Builder for generating a Postgres connector config with a distinct slot and publication.
• Postgres quickstart and Troubleshooting.

Primary sources (read 2026-10-09)

• PostgreSQL: pg_replication_slots (columns, wal_status values, safe_wal_size; 14 and 17 differ in the newer columns).
• PostgreSQL: replication configuration (max_slot_wal_keep_size, wal_keep_size, idle_replication_slot_timeout, synchronized_standby_slots, sync_replication_slots).
• PostgreSQL: logical decoding concepts (replication slots, slot synchronisation and the disk-usage caution).
• PostgreSQL: logical replication failover (the failover_ready check).
• PostgreSQL: streaming replication protocol (START_REPLICATION, DROP_REPLICATION_SLOT, standby status updates).
• PostgreSQL: system administration functions (pg_drop_replication_slot, pg_wal_lsn_diff, pg_ls_waldir).
• Debezium connector for PostgreSQL (WAL disk space consumption, heartbeats, slot.* and snapshot settings, failure handling, topologies).