@kapa.ai I'm trying to sync from Postgres to Redshift but the connection gets stuck at replication, more specifically when the replication orchestrator starts flushing. How can I fix this and what is the root cause?

Postgres to Redshift Sync Stuck During Replication/Flushing There are a few known root causes for this type of issue. Here's a breakdown: 1. Redshift Destination Hanging at Finalization A known issue causes syncs to hang indefinitely at lines like

"Executing finalization of tables."

or

"Finished syncing <source>"

. This was traced to a sidecar pod not being properly closed in Kubernetes deployments. [Redshift Hanging] Fix: Disable the sidecar. Once disabled, the hangs stopped. 2. CDC Replication Slot Not Being Consumed If you're using CDC mode, a known bug in Postgres source connector versions `0.4.27`+ combined with Airbyte versions

v0.38.3-alpha

through

v0.39.17-alpha

causes the incremental state to not be updated, meaning the replication slot accumulates WAL data but Airbyte never consumes it. This can eventually crash your Postgres database by filling up disk. [CDC Replication Slot] Fix: Upgrade the Postgres source connector to at least

0.4.31

. 3. PgJDBC Regression (Versions 3.6.19+) If you're on Postgres source connector 3.6.19 or later, there is a regression caused by PgJDBC 42.7.0+ which automatically advances

confirmed_flush_lsn

even when there are no new events for the publication tables. This causes CDC syncs to repeatedly fail or trigger full refreshes. [CDC LSN Issue; PgJDBC Issue] Immediate workaround: Downgrade to Postgres source connector 3.6.18. Alternative workaround: Create a heartbeat table and add it to your CDC publication to ensure the WAL always has events:

CREATE TABLE public.airbyte_heartbeat (
    id SERIAL PRIMARY KEY,
    timestamp timestamptz DEFAULT CURRENT_TIMESTAMP NOT NULL,
    text TEXT
);

Then periodically insert into it to keep the WAL active. 4. Redshift S3 COPY Strategy Error If you're using the S3 COPY strategy for Redshift, there was a bug causing failures during the write phase (with a

syntax error at or near "null"

error). [Redshift Connection Error] Fix: Upgrade the Redshift destination connector to 0.3.31 or later. As a temporary workaround, switching to the INSERT strategy also resolves the issue (though it is slower). Recommended Steps 1. Check which connector versions you're running (source Postgres and destination Redshift). 2. Upgrade both connectors to their latest stable versions. 3. If using CDC, verify your replication slot is being consumed and consider the heartbeat table workaround if on connector 3.6.19+. 4. If on Kubernetes, check for sidecar pod issues. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai This is the log message I keep receiving in loop: Worker finished flushing. Current queue size: 12966 REMAINING_BUFFERS_INFO Waiting for all streams to flush.

Based on the knowledge sources, I can see this exact log pattern appearing in a related issue, though the knowledge sources do not contain a definitive root cause or fix specifically for the Postgres → Redshift flushing loop scenario. "Waiting for all streams to flush" Loop The log pattern you're seeing:

Flush Worker -- Worker finished flushing. Current queue size: 12966
REMAINING_BUFFERS_INFO
Waiting for all streams to flush.

This appears in the context of the destination flush workers getting stuck — records accumulate in the buffer queue but the workers cannot fully drain it. [CDC Connector Stuck] Known Contributing Factors 1. Back pressure from the destination — The destination is taking too long to flush data, causing the source to keep feeding records faster than the destination can write them. This is especially common when many tables are involved. [CDC Connector Stuck] 2. Concurrent FlushWorker file index collision — Two flush workers can simultaneously pick up the same stream, upload to identical S3 keys, and cause silent data loss or stalling. [Redshift Silent Data Loss] Recommended Fixes 1. Increase the Redshift destination buffer parameter to

50

— this was confirmed to resolve the stuck flushing issue for at least one user. [CDC Connector Stuck] 2. Limit the number of tables per CDC connection to ~30 — having too many streams overwhelms the flush workers. [CDC Connector Stuck] 3. Ensure you are using the staging strategy on the Redshift destination rather than INSERT mode. [CDC Connector Stuck] 4. Upgrade both connectors (Postgres source and Redshift destination) to their latest versions. I'd recommend starting with adjusting the buffer parameter to

50

as it was the confirmed fix in a very similar reported case. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai Is there any quick fix besides the ones you sent?

Based on the knowledge sources, I'm sorry but there isn't enough information to suggest additional quick fixes specifically for the "Waiting for all streams to flush" loop beyond what was already mentioned. The knowledge sources do reference a related GlobalMemoryManager negative accounting bug that can cause infinite zero-byte flush loops, where the memory counter goes deeply negative and disables all buffer backpressure — resulting in streams stuck flushing indefinitely. [Memory Manager Bug] If this is the root cause in your case, the fix was submitted as a code-level patch (PR #74898), but the knowledge sources don't confirm which specific Airbyte or connector version includes this fix. I'd recommend: • Checking if your Airbyte version is OSS v2.0.1 or earlier, as that is the version where this bug was confirmed reproduced. • Upgrading to a newer Airbyte version if so, as the fix was submitted upstream. Beyond that, the knowledge sources do not contain enough information to suggest further quick fixes for your specific situation. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai I only have one stream running on this connection, it shouldn't error

That's a useful detail. Based on the knowledge sources, there is actually a documented case of this exact scenario — a sync hanging/looping on flush with only one stream and minimal traffic. From the GitHub issue on the CDC LSN bug: [CDC LSN Issue] > "The issue occurred in my test environment, which has minimal traffic and only one table involved." This points to the PgJDBC regression in Postgres source connector 3.6.19+, where

confirmed_flush_lsn

gets advanced incorrectly even with no new events, causing the CDC state to become invalid and the sync to get stuck. Quick Fix for Single Stream Downgrade your Postgres source connector to version 3.6.18, which is confirmed to not have this issue. [CDC LSN Issue] Alternatively, if you want to stay on a newer version, 3.7.2 was reported to fix this issue: [CDC LSN Issue] > "Airbyte released

3.7.2

which fixes this issue for me" So your two clearest options are: • Downgrade to

3.6.18

• Upgrade to

3.7.2

or later 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai What is the raw table in airbyte internal schema in the DB used for?

Airbyte Internal Raw Tables The

airbyte_internal

schema raw tables serve as a staging ground for Airbyte to optimize sync performance. Specifically, they are used for: 1. Buffering incoming data — As a first step in each sync, Airbyte copies records from the source into the raw table before processing them into the final table. [Incremental Sync Tutorial] 2. Supporting typed final tables — Raw records are processed and normalized from the raw table into the final destination table with proper column types. [Upgrading to Destinations V2] 3. Sync history and deduplication — They maintain sync history and assist with merging, deduplicating, and incremental syncs. [Forum Post] Raw Table Structure Each raw table typically contains these columns: [Postgres Destination Docs] Important Notes • Raw tables are not guaranteed to be stable — Airbyte may change their structure at any time. [Upgrading to Destinations V2] • For Postgres destination v3.0.0+ and Snowflake destination v4.0.0+, raw tables are deprecated in favor of Direct Load, which writes directly to final tables — so you can safely delete the

airbyte_internal

schema for those connectors. [Github Issue] • It is recommended to query the final tables rather than raw tables for downstream use. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements

@kapa.ai Can this flush issue be related to the tables in the destination?

Based on the knowledge sources, yes — there is a documented case that directly links destination table issues to the flush problem. Generation ID Mismatch in Destination Tables A known bug in destination-redshift 3.5.2 causes a silent data-loss and flush issue where: 1. Raw tables in

airbyte_internal

contain the data correctly 2. But the TypeAndDedupe (T+D) step produces 0 rows due to a generation ID mismatch between what was written to the raw table and what the T+D query expects 3. Final tables end up empty despite the sync reporting success [Truncate Refresh Bug] This is particularly relevant because: > _"The T+D step creates

_airbyte_tmp

final tables and runs

INSERT...SELECT

queries from the raw tables with a

WHERE _airbyte_generation_id = <current_generation_id>

filter. Because the records were written with a generation ID that does not match the current generation expected by the T+D query, 0 rows are inserted."_ How This Relates to Your Flush Issue If the destination tables have a generation ID mismatch, the flush workers could be stuck in a loop trying to process records that effectively produce no output — which aligns with your symptom of a single stream endlessly waiting to flush. However, I must note that the knowledge sources do not explicitly confirm this as the direct cause of the flush loop you're experiencing. I'd recommend: • Checking if your final tables in Redshift are empty despite the raw tables in

airbyte_internal

containing data • Verifying your Redshift destination connector version and upgrading if on

3.5.2

[Truncate Refresh Bug] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai I keep getting the following logs in repeat: 2026-05-25 104549 info INFO pool-4-thread-3 i.a.c.i.d.r.FileBuffer(deleteFile):75 Deleting tempFile data xxxxxxxxx.csv.gz 2026-05-25 104549 info INFO pool-4-thread-1 i.a.c.i.d.a.GlobalMemoryManager(free):78 Freeing 0 bytes.. 2026-05-25 104549 info INFO pool-4-thread-5 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):170 Flush Worker (c4cb4) -- Worker finished flushing. Current queue size: 12966 2026-05-25 104549 info INFO pool-4-thread-4 i.a.c.i.d.s.o.StagingStreamOperations(writeRecordsImpl):54 Buffer flush complete for stream field (0 bytes) to staging 2026-05-25 104549 info INFO pool-4-thread-4 i.a.c.i.d.s.o.StagingStreamOperations(writeRecordsImpl):60 Skipping writing to storage since there are no bytes to write 2026-05-25 104549 info INFO pool-4-thread-3 i.a.c.i.d.a.s.GlobalAsyncStateManager(flushStates):153 Flushing states 2026-05-25 104549 info INFO pool-4-thread-4 i.a.c.i.d.r.FileBuffer(deleteFile):75 Deleting tempFile data xxxxxxxxxxxxx.csv.gz 2026-05-25 104549 info INFO pool-4-thread-1 i.a.c.i.d.a.GlobalMemoryManager(free):78 Freeing 0 bytes.. 2026-05-25 104549 info INFO pool-4-thread-3 i.a.c.i.d.a.s.GlobalAsyncStateManager(flushStates):207 Flushing states complete 2026-05-25 104549 info INFO pool-4-thread-4 i.a.c.i.d.a.s.GlobalAsyncStateManager(flushStates):153 Flushing states 2026-05-25 104549 info INFO pool-4-thread-4 i.a.c.i.d.a.s.GlobalAsyncStateManager(flushStates):207 Flushing states complete 2026-05-25 104549 info INFO pool-4-thread-1 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):170 Flush Worker (98cdb) -- Worker finished flushing. Current queue size: 12966 2026-05-25 104549 info INFO pool-4-thread-3 i.a.c.i.d.a.GlobalMemoryManager(free):78 Freeing 0 bytes.. 2026-05-25 104549 info INFO pool-4-thread-4 i.a.c.i.d.a.GlobalMemoryManager(free):78 Freeing 0 bytes.. 2026-05-25 104549 info INFO pool-4-thread-4 i.a.c.i.d.a.GlobalMemoryManager(free):78 Freeing 0 bytes.. 2026-05-25 104549 info INFO pool-4-thread-3 i.a.c.i.d.a.GlobalMemoryManager(free):78 Freeing 0 bytes.. 2026-05-25 104549 info INFO pool-4-thread-4 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):170 Flush Worker (2bf4e) -- Worker finished flushing. Current queue size: 12966 2026-05-25 104549 info INFO pool-4-thread-3 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):170 Flush Worker (91027) -- Worker finished flushing. Current queue size: 12966 2026-05-25 104550 info INFO main i.a.c.i.d.a.FlushWorkers(close):226 REMAINING_BUFFERS_INFO Namespace: schema Stream: stream -- remaining records: 12966 2026-05-25 104550 info INFO main i.a.c.i.d.a.FlushWorkers(close):227 Waiting for all streams to flush.

Your logs confirm the infinite zero-byte flush loop caused by the

GlobalMemoryManager

negative accounting bug. The key indicators are:

GlobalMemoryManager(free):78 Freeing 0 bytes..
StagingStreamOperations(writeRecordsImpl):60 Skipping writing to storage since there are no bytes to write
Flush Worker -- Worker finished flushing. Current queue size: 12966  ← never decreases
Waiting for all streams to flush.

The queue size stays at 12966 records permanently while workers keep flushing 0 bytes — this is exactly the failure mode described in the bug report. [Memory Manager Bug] Root Cause

GlobalMemoryManager.free()

allowed

currentMemoryBytes

to go negative. Once negative, the backpressure gate in

requestMemory()

is permanently disabled, so: 1. Workers keep getting triggered to flush 2. But they dequeue 0 bytes each time 3. The 12966 records are stuck in the buffer indefinitely 4. The sync never completes [Memory Manager Fix PR] Fix The fix was submitted in PR #74898 and #74899, which: • Clamps

currentMemoryBytes

to 0 when it would go negative (CAS loop) • Guards

memoryManager.free(unusedBytes)

behind

if (unusedBytes > 0)

in

BufferDequeue.take()

Your immediate options are: 1. Upgrade Airbyte to a version that includes PR #74898/#74899 — this is the proper fix 2. Cancel and retry the sync as a short-term workaround, though it will likely recur until the fix is applied 3. Check your current Airbyte version — if on OSS v2.0.1 or earlier, upgrading is strongly recommended [Memory Manager Bug] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai But it was working before

That's consistent with how this bug behaves. The

GlobalMemoryManager

negative accounting issue is triggered by specific conditions during a sync, not present from the start. [Memory Manager Bug] Why It Works Sometimes and Fails Other Times The bug is triggered when: 1. Memory pressure occurs —

requestMemory()

returns

0

when memory is full, but the queue's internal counter is adjusted regardless 2. The drift accumulates —

BufferDequeue.take()

then frees more bytes than were actually allocated, driving

currentMemoryBytes

negative 3. Once negative, backpressure is permanently disabled for the rest of that sync This means the sync can work fine for a while and then suddenly tip into the broken state mid-sync once memory pressure is hit. [Memory Manager Bug] What Likely Changed The knowledge sources suggest this is more likely to be triggered by: • Larger data volumes than previous syncs — e.g. more records accumulated since the last successful sync • Memory pressure on the destination pod at the time of the sync So if your source data has grown, or the sync is running alongside other resource-intensive processes, it could explain why it worked before but is now consistently failing. [Memory Manager Fix PR] The only reliable fix remains upgrading to a version that includes the fix from PR #74898/#74899. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai The PRs weren't merged yet, what is the solution for this?

The knowledge sources do not contain enough information to suggest a definitive workaround specifically for the period while PRs #74898/#74899 are still unmerged. The only options I can confirm from the knowledge sources are: 1. Cancel and retry the sync — this is a short-term workaround, but the bug will likely recur once memory pressure is hit again during the same sync. [Memory Manager Bug] 2. Reduce the data volume per sync — since the bug is triggered by memory pressure, syncing smaller batches of data may help avoid hitting the condition that drives

currentMemoryBytes

negative. 3. Monitor the PR — both PRs are open and being tracked internally, so checking for a merge and upgrading as soon as a fixed version is released is the most reliable path forward: ◦ Community PR: #74898 ◦ Internal PR: #74899 I'd recommend: • Joining the Airbyte Community Slack to ask directly about the fix timeline • If you're an Airbyte Cloud customer, opening a support ticket referencing issue #74897 to get priority attention on this 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai When I run abctl install what is the default memory and cpu allocated?

Based on the knowledge sources, when

abctl local install

is run, the following default resource limits are hardcoded in `internal/helm/airbyte_values.go`:

global.jobs.resources.limits.cpu=3
global.jobs.resources.limits.memory=4Gi

[Resource Limits Bug] However, the knowledge sources do not explicitly mention the full set of default values for all components (e.g. worker, server, temporal, etc.) that

abctl

allocates by default. Important Caveat There is also an open bug where these configured values may not actually be respected by the

workload-launcher

when spawning replication pods. Instead, the observed pod resources were: • Source:

limits={memory=2Gi, cpu=2}, requests={memory=1Gi, cpu=1}

• Destination:

limits={memory=2Gi, cpu=2}, requests={memory=2Gi, cpu=1}

These appear to be internal defaults used by the

workload-launcher

itself, independent of the Helm values. [Resource Limits Bug] For the most accurate and complete default resource allocations, I'd recommend checking the official abctl documentation or the Airbyte Helm chart values directly. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.