<@U04S8PLJWMT> the postgres db had "no space left"...
# ask-ai
f
@kapa.ai the postgres db had "no space left" and was triggered back to previous valid position. Now my connections couldn't advance. How can I try to resolve?
k
When Postgres runs out of disk, CDC can “jump back” to an older offset or lose access to the WAL segment it previously used. Then Airbyte sees that the saved offset is no longer valid and will either fail or keep re‑starting from scratch, so your connections never advance. From the docs and issues, there are essentially two classes of problems and corresponding actions: --- ### 1. WAL segments needed by Airbyte were purged If you see logs like: -
Saved offset is before replication slot's confirmed lsn. Please reset the connection...
[[GitHub issue](https://github.com/airbytehq/airbyte/issues/49802)] - or
Saved offset is before Replication slot's confirmed_flush_lsn, Airbyte will trigger sync from scratch
[[Stack syncs forum](https://discuss.airbyte.io/t/2675)] then Postgres has already removed WAL segments that Airbyte’s saved state points to. In that case: 1. Temporarily free up enough disk on Postgres so it can run. 2. Increase WAL retention / sync more frequently to avoid this in the future: - Increase
wal_keep_size
(at the cost of more disk). [[Under CDC incremental…](https://docs.airbyte.com/integrations/sources/postgres/postgres-troubleshooting#under-cdc-incremental-mode-there-are-still-full-refresh-syncs)] - Run CDC syncs more frequently so WAL doesn’t accumulate too long. 3. Reset the Airbyte connection and re‑sync from scratch so a new, valid snapshot and CDC position are established. The docs explicitly recommend a full refresh in this situation. [[Under CDC incremental…](https://docs.airbyte.com/integrations/sources/postgres/postgres-troubleshooting#under-cdc-incremental-mode-there-are-still-full-refresh-syncs)] If you’re using the “Invalid CDC position behavior” advanced setting, setting it to Re‑sync data will make Airbyte automatically start from scratch when it detects that the saved offset is before the replication slot’s
confirmed_flush_lsn
. [[GitHub issue](https://github.com/airbytehq/airbyte/issues/49802); [Stack syncs forum](https://discuss.airbyte.io/t/2675)] --- ### 2. WAL backlog /
confirmed_flush_lsn
not advancing Low‑volume databases (or heavily filtered publications) can get stuck with WAL accumulating but
confirmed_flush_lsn
not moving, especially if many events are filtered out. Debezium warns about this and Postgres can end up holding a large WAL backlog, which in turn can contribute to disk‑full conditions. [[Stack syncs forum](https://discuss.airbyte.io/t/2675); [Advanced WAL disk consumption](https://docs.airbyte.com/integrations/sources/postgres/postgres-troubleshooting#advanced-wal-disk-consumption-and-heartbeat-action-query)] To get things moving again: 1. Add a heartbeat or a high‑volume table to the publication so the LSN can advance: - Simple (read‑only) fix: add a high‑volume table to the publication (you don’t have to include it in the Airbyte catalog):
Copy code
sql      ALTER PUBLICATION <publicationName> ADD TABLE <high_volume_table>;
[[Advanced WAL disk consumption](https://docs.airbyte.com/integrations/sources/postgres/postgres-troubleshooting#advanced-wal-disk-consumption-and-heartbeat-action-query)] - Preferred (with write access): create and use a heartbeat table:
Copy code
sql      CREATE TABLE airbyte_heartbeat (        id SERIAL PRIMARY KEY,        timestamp TIMESTAMP NOT NULL DEFAULT current_timestamp,        text TEXT      );       ALTER PUBLICATION <publicationName> ADD TABLE airbyte_heartbeat;
Then set the Debezium heartbeat query in the Postgres source config to:
Copy code
sql      INSERT INTO airbyte_heartbeat (text) VALUES ('heartbeat');
Airbyte will periodically execute this query to advance the LSN and help consume WAL. [[Advanced WAL disk consumption](https://docs.airbyte.com/integrations/sources/postgres/postgres-troubleshooting#advanced-wal-disk-consumption-and-heartbeat-action-query)] 2. Clean up any unused replication slots if you’ve stopped using them, because they also retain WAL and consume disk. [[Connector limitations](https://docs.airbyte.com/integrations/sources/postgres/postgres-troubleshooting#connector-limitations)] --- ### 3. If Postgres itself still reports “no space left on device” If you’re seeing errors like
could not write to file "base/pgsql_tmp/..." : No space left on device
, that is Postgres’ temp / data disk being full. This is separate from Airbyte and must be fixed on the database side (increase storage, clean up WAL, tmp files, etc.). [[No space during sync](https://discuss.airbyte.io/t/1572); [Connector limitations](https://docs.airbyte.com/integrations/sources/postgres/postgres-troubleshooting#connector-limitations)] --- ### Practical recovery path 1. On Postgres: - Free disk until the DB runs again. - If WAL has grown huge, verify replication slots and
pg_wal
usage. - Optionally tune
wal_keep_size
and set up the heartbeat strategy. 2. On Airbyte: - For connections stuck at an old position, Reset the connection and run a full sync once. - Ensure the Postgres source connector and Airbyte versions you use don’t have known CDC state bugs (if you’re on an older version, consider upgrading). If you can share the exact CDC log line from your failing runs (around the “saved offset” or “replication slot” message), I can map it more precisely to one of the above scenarios and suggest a minimal set of steps.
f
@kapa.ai can you please help me with the error on this connection log?
k
Sorry @Fabrizio Spini, it looks like your question contains files, which I cannot process. Please reformulate your question without including any files.