Shaunak Tulshibagwale
12/17/2025, 10:00 PMjava.lang.RuntimeException: org.postgresql.util.PSQLException: ERROR: cannot read from logical replication slot "airbyte_slot" Detail: This slot has been invalidated because it exceeded the maximum reserved size.
When I check psql, this is the output I'm getting:
postgres=> SHOW max_slot_wal_keep_size;
max_slot_wal_keep_size
------------------------
1GB
(1 row)
I hesitate to increase the max_slot_wal_keep_size as it can negatively affect our I/O usage and was wondering if there could be something re configured on the airbyte side to get this through. We've previously been able to extract our postgres data into other tools too. Any help is appreciated! Thanks 🙂kapa.ai
12/17/2025, 10:00 PMairbyte_slot because the slot exceeded max_slot_wal_keep_size (1GB in your case). From Airbyte’s perspective, this usually happens when: - WAL accumulates faster than Airbyte consumes it, or - There are long periods with very few/no changes, so the LSN doesn’t advance and WAL can’t be recycled safely. [[Postgres troubleshooting](https://docs.airbyte.com/integrations/sources/postgres/postgres-troubleshooting#troubleshooting); [WAL disk consumption](https://docs.airbyte.com/integrations/sources/postgres/postgres-troubleshooting#advanced-wal-disk-consumption-and-heartbeat-action-query)] Because you don’t want to raise max_slot_wal_keep_size much, the knobs you can turn on the Airbyte side are: 1. Increase sync frequency Run the Postgres→ClickHouse connection more often so Airbyte reads WAL before it hits the 1GB cap. This is explicitly recommended when WAL is being purged between jobs. [[Postgres troubleshooting](https://docs.airbyte.com/integrations/sources/postgres/postgres-troubleshooting#troubleshooting)] 2. Ensure the replication slot is actively advancing even with low change volume With low-traffic databases, WAL can pile up without the slot’s confirmed_flush_lsn moving. Airbyte’s Postgres connector supports a Debezium “heartbeat query” to force periodic changes and advance the LSN. The docs describe the pattern: - Create a small airbyte_heartbeat table. - Add it to the publication. - Configure the Debezium heartbeat query in the Postgres source so Airbyte periodically inserts into that table. [[WAL disk consumption](https://docs.airbyte.com/integrations/sources/postgres/postgres-troubleshooting#advanced-wal-disk-consumption-and-heartbeat-action-query)] This keeps the slot “fresh” without you having to increase WAL retention too much. 3. Add a high‑volume table to the publication (read‑only workaround) If you don’t want Airbyte to have write access, you can add one or more higher‑churn tables to the publication so that WAL moves forward naturally, even if you don’t actually sync those tables in Airbyte. This is called out as the “simple fix (read‑only)” in the docs. [[WAL disk consumption](https://docs.airbyte.com/integrations/sources/postgres/postgres-troubleshooting#advanced-wal-disk-consumption-and-heartbeat-action-query)] 4. If the slot is already invalidated, reset and re‑sync Once Postgres has invalidated airbyte_slot, you’ll need to: - Drop and recreate the replication slot and publication, and - In Airbyte, reset the connection so it can do a fresh snapshot and then resume CDC. [[Postgres CDC tutorial](https://airbyte.com/tutorials/postgresql-change-data-capture-elt#step-2); [CDC troubleshooting](https://docs.airbyte.com/integrations/sources/postgres/postgres-troubleshooting#troubleshooting)] Given your constraints, a practical approach is: 1. Reset the connection and recreate the slot/publication. 2. Increase sync frequency (e.g., hourly or more often, depending on write volume). 3. Either: - Configure the Debezium heartbeat table + query, or - Add a naturally high‑churn table to the publication. This combination minimizes how much you need to raise max_slot_wal_keep_size while avoiding future “slot exceeded maximum reserved size” errors.Shaunak Tulshibagwale
12/17/2025, 11:08 PMkapa.ai
12/17/2025, 11:08 PMkapa.ai
12/17/2025, 11:08 PMmax_slot_wal_keep_size, the problem is less about frequency and more about how much WAL is generated during that long‑running initial snapshot. From the docs and forum threads, there are a few concrete levers you can use on the Airbyte/Postgres side (without only relying on raising max_slot_wal_keep_size): 1. Shorten the initial snapshot window (fewer tables / staged loads) The initial CDC sync does a full snapshot of all selected tables and then starts streaming WAL. For very large datasets, that snapshot can take long enough that WAL for the slot grows past your 1 GB cap. [[Postgres troubleshooting](https://docs.airbyte.com/integrations/sources/postgres/postgres-troubleshooting#connector-limitations)] To reduce pressure: - Start with a smaller subset of tables (or the most critical ones). - Once those are synced and CDC is running, add more tables in additional connections or phases. 2. Make sure WAL isn’t being purged faster than Airbyte can read it If Postgres purges WAL needed by the slot during that long first run, Airbyte will fail and the slot can be invalidated. The docs recommend: - Increasing WAL retention (e.g., wal_keep_size / WAL retention period) so it’s longer than the time the initial sync takes, and - Running syncs frequently once the snapshot is done. [[CDC best practices](https://docs.airbyte.com/platform/understanding-airbyte/cdc-best-practices#database-level-cdc-configuration); [Postgres troubleshooting](https://docs.airbyte.com/integrations/sources/postgres/postgres-troubleshooting#troubleshooting)] I understand you’re hesitant to increase max_slot_wal_keep_size, but for the initial load you may need a temporary bump (e.g., a few GB) so the first snapshot can complete, then you can lower it again once CDC is stable. 3. Use a heartbeat / high‑volume table so the slot advances cleanly With low or bursty write volume, Debezium/Airbyte can end up in a state where WAL accumulates but the slot’s confirmed_flush_lsn doesn’t advance, which keeps disk reserved and can hit your slot limit. [[WAL disk consumption](https://docs.airbyte.com/integrations/sources/postgres/postgres-troubleshooting#advanced-wal-disk-consumption-and-heartbeat-action-query); [Replication slot safe_wal_size](https://discuss.airbyte.io/t/3263)] Two options from the docs: - Write‑access option (recommended) - Create an airbyte_heartbeat table, add it to the publication, and configure the Debezium heartbeat query in the Postgres source so Airbyte periodically inserts into it. [[WAL disk consumption](https://docs.airbyte.com/integrations/sources/postgres/postgres-troubleshooting#advanced-wal-disk-consumption-and-heartbeat-action-query)] - Read‑only option - Add a naturally high‑churn table to the publication (even if you don’t sync it in Airbyte) so WAL moves forward and can be recycled. [[WAL disk consumption](https://docs.airbyte.com/integrations/sources/postgres/postgres-troubleshooting#advanced-wal-disk-consumption-and-heartbeat-action-query)] 4. If the slot is already invalidated, you must recreate it and reset the connection Once Postgres says the slot is invalid because it exceeded the reserved size, it can’t be reused. You’ll need to: - Drop and recreate the replication slot (and publication if needed), and - In Airbyte, Reset the connection so it performs a fresh snapshot and then CDC. [[Cloud SQL CDC setup](https://docs.airbyte.com/integrations/sources/postgres/cloud-sql-postgres#advanced-setup-using-cdc); [Postgres troubleshooting](https://docs.airbyte.com/integrations/sources/postgres/postgres-troubleshooting#troubleshooting)] Given that your first daily sync doesn’t terminate: 1. Temporarily increase max_slot_wal_keep_size and/or WAL retention enough to cover the initial snapshot duration. 2. Reset the connection and recreate the slot/publication. 3. Limit the initial sync to a smaller set of tables (or run multiple connections) so the snapshot can finish. 4. Configure a heartbeat (or add a high‑volume table to the publication) so the slot’s LSN advances and WAL can be recycled. If you share approximate DB size and write rate, you can then tune how much you actually need to raise max_slot_wal_keep_size for that first run.