<@U04S8PLJWMT> I'm trying to sync from Postgres to...
# replication-ask-ai
j
@kapa.ai I'm trying to sync from Postgres to Redshift but the pipeline does not work. It gets stuck and takes a long time but never loads the data in the final table. In the logs I see this: INFO pool-4-thread-4 i.a.c.i.d.a.s.GlobalAsyncStateManager(flushStates):153 Flushing states. destination INFO pool-4-thread-4 i.a.c.i.d.a.s.GlobalAsyncStateManager(flushStates):207 Flushing states complete. destination INFO pool-4-thread-4 i.a.c.i.d.a.GlobalMemoryManager(free):78 Freeing 371 bytes. destination INFO pool-4-thread-4 i.a.c.i.d.a.GlobalMemoryManager(free):78 Freeing 52428303 bytes. destination INFO pool-4-thread-4 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):170 Flush Worker (c9c21) -- Worker finished flushing. Current queue size: 8412. Is that a common message or expected? Should I wait more time? The pipeline only has 1 table and ran for 10 minutes before I stopped but didn't finish
k
Those
GlobalAsyncStateManager(flushStates)
/
GlobalMemoryManager(free)
/
Flush Worker ... Current queue size: ...
lines are internal async‑writer logs and, by themselves, are normal; they just show the destination flushing buffered data and freeing memory. Similar messages appear in another open issue where the problem is that the sync “hangs forever,” so the presence of those lines alone does not mean progress is being made. [[Jira–BQ hang](https://github.com/airbytehq/airbyte/issues/58082)] From the knowledge provided, there is no explicit guidance on: - What queue size is “too large” or abnormal. - How long you should wait for a single‑table sync before considering it stuck. - Specific troubleshooting steps for Postgres → Redshift with this exact symptom. So I cannot confidently say whether your job will eventually finish if you just wait longer. What the similar cases suggest is: - A sync that appears to complete reading from the source but never finishes finalization on the destination side is not expected behavior; it’s usually a hang or an external interference (e.g., sidecars/proxies) rather than just “slow.” [[Redshift hang](https://discuss.airbyte.io/t/4574)] - In one Redshift case, the root cause was an extra sidecar container attached to all pods; once removed, the hangs stopped. [[Redshift hang](https://discuss.airbyte.io/t/4574)] Given the limited information in the sources, the safest next steps (based on what is documented) are: 1. Try another destination (e.g., S3 or a different warehouse) to see if the issue is isolated to Redshift, as suggested in a similar “sync stuck” case. [[Postgres stuck](https://discuss.airbyte.io/t/2703)] 2. Check your deployment for sidecars or proxies (service mesh, DB proxy, etc.) that might keep connections open or interfere with job completion, as this was the confirmed cause in another Redshift hanging scenario. [[Redshift hang](https://discuss.airbyte.io/t/4574)] 3. Let one run continue longer while monitoring logs: if you see the source finish and only destination logs repeating without any final “completed” message for a long time, that aligns with the “hang” pattern seen in the Jira→BigQuery and Redshift threads, not with normal slow progress. [[Jira–BQ hang](https://github.com/airbytehq/airbyte/issues/58082); [Redshift hang](https://discuss.airbyte.io/t/4574)] Because the knowledge sources don’t document a known bug or a precise fix for “Postgres → Redshift with large Flush Worker queue size,” I can’t give a definitive root cause. If possible, I’d recommend: - Capturing the full job logs (source + destination) from a run you let go to timeout/hang. - Testing the same Postgres source against a different destination. - Checking for and temporarily disabling any sidecars/proxies around the worker/destination pods. If you share the full logs and your deployment details in the Airbyte forum or GitHub, they can be compared directly to the known hanging cases referenced above.
j
@kapa.ai It just keeps repeating those same messages and in between there are warnings like that: WARN pool-4-thread-5 i.a.c.i.d.s.c.CsvSerializedBuffer(flushWriter):97 Trying to flush but no printer is initialized.
k
A message from kapa.ai
The knowledge sources don’t mention the specific warning >
CsvSerializedBuffer(flushWriter):97 Trying to flush but no printer is initialized.
so I can’t say with certainty what it means internally or how to fix it. The logs you’re seeing (
GlobalAsyncStateManager(flushStates)
,
GlobalMemoryManager(free)
,
Flush Worker ... Current queue size: ...
) plus repeated warnings and no job completion match the general “sync hangs forever” pattern seen in other cases, not normal slow progress. [[Jira–BQ hang](https://github.com/airbytehq/airbyte/issues/58082)] From the similar Redshift hanging case, we know: - Jobs could read data from the source and then hang indefinitely on the destination side (e.g., “Executing finalization of tables.”). - The root cause there was an extra sidecar container attached to the pods; once removed, the hangs stopped. [[Redshift hang](https://discuss.airbyte.io/t/4574)] And from another “sync stuck” case, support explicitly suggested: - Testing with a different destination to see whether the issue is in the source or destination. [[Postgres stuck](https://discuss.airbyte.io/t/2703)] Given what is documented, practical next steps are: 1. Test the same Postgres source with another destination (e.g., S3) to see if only Redshift hangs. [[Postgres stuck](https://discuss.airbyte.io/t/2703)] 2. Inspect your deployment for sidecars / service mesh / proxies attached to worker or destination pods and try disabling them, since a sidecar was a confirmed cause of Redshift hangs. [[Redshift hang](https://discuss.airbyte.io/t/4574)] 3. Let one run continue and capture full logs: if the source finishes and only destination logs (including that CSV warning) repeat without any “completed” message, it’s consistent with the hanging behavior in the Jira→BigQuery and Redshift examples, not with expected behavior. [[Jira–BQ hang](https://github.com/airbytehq/airbyte/issues/58082); [Redshift hang](https://discuss.airbyte.io/t/4574)] Because the provided sources don’t document this exact
CsvSerializedBuffer
warning or a known bug for Postgres→Redshift with it, I can’t give a definitive root cause. The best next move is to run one full attempt to “hang,” export the complete job logs, and share them (plus your deployment details and any sidecars/proxies in use) in the Airbyte forum or GitHub so they can be compared directly to the known hanging cases above.