@kapa.ai can you help me identify the problem in my airbyte connection

I’m sorry, but the knowledge sources don’t contain enough specific information about your exact connection or error message to precisely identify the problem. To move forward, here’s what you can do inside Airbyte to narrow it down: 1. Open the failing connection’s logs ◦ Go to the connection, open the Job History tab, click the three dots on the failing sync, and choose View logs. [Browsing logs] ◦ Search in the log viewer for

Error

,

Exception

, or

Fail

to find the root cause. [Browsing logs] 2. Check the connection status page (Cloud/Enterprise) ◦ On the connection’s Status page, see whether the error is marked as a Configuration error (red) or a System error (yellow). [Connection status] ◦ If it’s a configuration error, go to the source/destination, re‑test the connection, and save before retrying the sync. [Connection status] 3. If schema discovery or setup is failing ◦ Double‑check host, port, credentials, and any connector‑specific settings. [Forum debug tip] ◦ Use the logs (step 1) to see if the connector is timing out, being rate‑limited, or returning a specific database error. If you paste the exact error message (or a small excerpt from the logs) from step 1 here, I can use that along with the docs above to give you more targeted guidance. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai the connection is trying to copy 8M rows of data from psql to redshift. it seems stuck and do not continue. trying to cancel and resync the connection. the latest logs are:

2026-03-06 16:30:20 destination INFO pool-4-thread-2 i.a.i.d.p.PostgresSqlOperations(insertRecordsInternal):137 COPY command completed sucessfully
2026-03-06 16:30:20 destination INFO pool-4-thread-2 i.a.c.i.d.a.s.GlobalAsyncStateManager(flushStates):153 Flushing states
2026-03-06 16:30:20 destination INFO pool-4-thread-2 i.a.c.i.d.a.s.GlobalAsyncStateManager(flushStates):207 Flushing states complete
2026-03-06 16:30:20 destination INFO pool-4-thread-2 i.a.c.i.d.a.GlobalMemoryManager(free):78 Freeing 1334 bytes..
2026-03-06 16:30:20 destination INFO pool-4-thread-2 i.a.c.i.d.a.GlobalMemoryManager(free):78 Freeing 26214150 bytes..
2026-03-06 16:30:20 destination INFO pool-4-thread-2 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):172 Flush Worker (d7d33) -- Worker finished flushing. Current queue size: 33182
2026-03-06 16:30:21 destination INFO pool-5-thread-1 i.a.c.i.d.a.DetectStreamToFlush(getNextStreamToFlush):119 flushing: trigger info: null - pos_transfer, time trigger: false , size trigger: true current threshold b: 25 MB, queue size b: 25.12 MB, penalty b: 0 bytes, after penalty b: 25.12 MB
2026-03-06 16:30:21 destination INFO pool-4-thread-1 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):137 Flush Worker (86d27) -- Worker picked up work.
2026-03-06 16:30:21 destination INFO pool-4-thread-1 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):143 Flush Worker (86d27) -- Attempting to read from queue namespace: null, stream: pos_transfer.
2026-03-06 16:30:21 destination INFO pool-4-thread-1 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):161 Flush Worker (86d27) -- Batch contains: 33025 records, 25 MB bytes.
2026-03-06 16:30:21 destination INFO pool-4-thread-1 i.a.i.d.p.PostgresSqlOperations(insertRecordsInternal):104 preparing records to insert. generationId=4, syncId=109863
2026-03-06 16:30:21 destination INFO pool-4-thread-1 i.a.i.d.p.PostgresSqlOperations(insertRecordsInternal$lambda$0):122 executing COPY command: COPY airbyte_internal.public_raw__stream_pos_transfer (_airbyte_raw_id, _airbyte_data, _airbyte_extracted_at, _airbyte_loaded_at, _airbyte_meta, _airbyte_generation_id) FROM stdin DELIMITER ',' CSV
2026-03-06 16:30:22 destination INFO pool-4-thread-1 i.a.i.d.p.PostgresSqlOperations(insertRecordsInternal):137 COPY command completed sucessfully
2026-03-06 16:30:22 destination INFO pool-4-thread-1 i.a.c.i.d.a.s.GlobalAsyncStateManager(flushStates):153 Flushing states
2026-03-06 16:30:22 destination INFO pool-4-thread-1 i.a.c.i.d.a.s.GlobalAsyncStateManager(flushStates):207 Flushing states complete
2026-03-06 16:30:22 destination INFO pool-4-thread-1 i.a.c.i.d.a.GlobalMemoryManager(free):78 Freeing 1333 bytes..
2026-03-06 16:30:22 destination INFO pool-4-thread-1 i.a.c.i.d.a.GlobalMemoryManager(free):78 Freeing 26214012 bytes..
2026-03-06 16:30:22 destination INFO pool-4-thread-1 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):172 Flush Worker (86d27) -- Worker finished flushing. Current queue size: 157
2026-03-06 16:31:17 destination INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 9.34 GB, allocated: 19.94 MB (19.93697738647461 MB), %% used: 0.002083714191730206 | Queue `pos_transfer`, num records: 157, num bytes: 121.59 KB, allocated bytes: 10.02 MB | State Manager memory usage: Allocated: 9 MB, Used: -87799 bytes, percentage Used -0.008443867023544327
2026-03-06 16:31:17 destination INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):129 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2026-03-06 16:32:17 destination INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 9.34 GB, allocated: 19.94 MB (19.93697738647461 MB), %% used: 0.002083714191730206 | Queue `pos_transfer`, num records: 157, num bytes: 121.59 KB, allocated bytes: 10.02 MB | State Manager memory usage: Allocated: 9 MB, Used: -87799 bytes, percentage Used -0.008443867023544327
2026-03-06 16:32:17 destination INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):129 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2026-03-06 16:33:17 destination INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 9.34 GB, allocated: 19.94 MB (19.93697738647461 MB), %% used: 0.002083714191730206 | Queue `pos_transfer`, num records: 157, num bytes: 121.59 KB, allocated bytes: 10.02 MB | State Manager memory usage: Allocated: 9 MB, Used: -87799 bytes, percentage Used -0.008443867023544327
2026-03-06 16:33:17 destination INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):129 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0

New message text here

@kapa.ai just pasted all logs. can you perform the checks you suggest me to do

Sorry @utku yaman, I encountered an error processing the file(s) you uploaded: File type is not supported

@kapa.ai just attached the logs, can you perform the checks you suggested me to do

Sorry @utku yaman, I encountered an error processing the file(s) you uploaded: Text file is too large (max 60K characters)

@kapa.ai latest log from the source

2026-03-06 16:30:17 source INFO main i.a.c.i.s.r.s.SourceStateIterator(computeNext):50 sending state message, with count per stream: {public_transfer=10000} 
2026-03-06 16:30:17 source INFO main i.a.i.s.p.c.CtidStateManager(generateStateMessageAtCheckpoint):79 Emitting ctid state for stream public_transfer, state is io.airbyte.integrations.source.postgres.internal.models.CtidStatus@53093491[version=2,stateType=ctid,ctid=(115022,24),incrementalState={"version":2,"state_type":"xmin","num_wraparound":0,"xmin_raw_value":69326954,"xmin_xid_value":69326954},relationFilenode=64201,additionalProperties={}]
2026-03-06 16:30:17 source INFO main i.a.c.i.s.r.s.SourceStateIterator(computeNext):50 sending state message, with count per stream: {public_transfer=10000} 
2026-03-06 16:30:18 source INFO main i.a.i.s.p.c.CtidStateManager(generateStateMessageAtCheckpoint):79 Emitting ctid state for stream public_transfer, state is io.airbyte.integrations.source.postgres.internal.models.CtidStatus@21719a0[version=2,stateType=ctid,ctid=(115323,14),incrementalState={"version":2,"state_type":"xmin","num_wraparound":0,"xmin_raw_value":69326954,"xmin_xid_value":69326954},relationFilenode=64201,additionalProperties={}]
2026-03-06 16:30:18 source INFO main i.a.c.i.s.r.s.SourceStateIterator(computeNext):50 sending state message, with count per stream: {public_transfer=10000} 
2026-03-06 16:30:18 source INFO main i.a.i.s.p.c.CtidStateManager(generateStateMessageAtCheckpoint):79 Emitting ctid state for stream public_transfer, state is io.airbyte.integrations.source.postgres.internal.models.CtidStatus@76b224cd[version=2,stateType=ctid,ctid=(115624,9),incrementalState={"version":2,"state_type":"xmin","num_wraparound":0,"xmin_raw_value":69326954,"xmin_xid_value":69326954},relationFilenode=64201,additionalProperties={}]
2026-03-06 16:30:18 source INFO main i.a.c.i.s.r.s.SourceStateIterator(computeNext):50 sending state message, with count per stream: {public_transfer=10000} 
2026-03-06 16:30:19 source INFO main i.a.i.s.p.c.CtidStateManager(generateStateMessageAtCheckpoint):79 Emitting ctid state for stream public_transfer, state is io.airbyte.integrations.source.postgres.internal.models.CtidStatus@77ee25f1[version=2,stateType=ctid,ctid=(115926,11),incrementalState={"version":2,"state_type":"xmin","num_wraparound":0,"xmin_raw_value":69326954,"xmin_xid_value":69326954},relationFilenode=64201,additionalProperties={}]
2026-03-06 16:30:19 source INFO main i.a.c.i.s.r.s.SourceStateIterator(computeNext):50 sending state message, with count per stream: {public_transfer=10000} 
2026-03-06 16:30:19 source INFO main i.a.i.s.p.c.CtidStateManager(generateStateMessageAtCheckpoint):79 Emitting ctid state for stream public_transfer, state is io.airbyte.integrations.source.postgres.internal.models.CtidStatus@6dd93a21[version=2,stateType=ctid,ctid=(116227,5),incrementalState={"version":2,"state_type":"xmin","num_wraparound":0,"xmin_raw_value":69326954,"xmin_xid_value":69326954},relationFilenode=64201,additionalProperties={}]
2026-03-06 16:30:19 source INFO main i.a.c.i.s.r.s.SourceStateIterator(computeNext):50 sending state message, with count per stream: {public_transfer=10000}

I’m sorry, but the knowledge sources don’t describe this exact combination of Postgres CTID incremental source → Redshift destination with async writer well enough to pinpoint a definitive root cause. I can only infer based on similar “sync hangs / stuck” patterns. From your two snippets together: • Source (Postgres CTID) ◦ Is actively emitting state messages every 10,000 records for `public_transfer`: ▪︎

sending state message, with count per stream: {public_transfer=10000}

▪︎

Emitting ctid state for stream public_transfer, state is ... ctid=(...)

◦ There are no errors; it looks like a normal CTID incremental read loop. • Destination (Redshift via PostgresSqlOperations) ◦ Is repeatedly executing

COPY ... FROM stdin

into the raw table and logging: ▪︎

COPY command completed sucessfully

◦ Async queue shows: ▪︎

Queue pos_transfer, num records: 157, num bytes: 121.59 KB

▪︎

Pool queue size: 0, Active threads: 0

◦ So all big batches have been flushed; only a tiny remainder is buffered and no workers are active. This pattern is very similar to other “sync hangs / stuck with no error” reports: • Jobs where logs show normal activity, then only periodic async queue/worker info while the UI still says “Running”. [Mixpanel/S3 hangs; HubSpot/S3 stuck] • Large Postgres loads that “never end” even though COPY/INSERTs complete, due to behavior outside the connector itself. [30GB Postgres load never ends] • Cases where the destination appears done but the job is blocked in orchestration/finalization, not in the actual data movement. [Redshift destination hanging] Given only what’s in the knowledge sources, the most we can say is: • Source side: appears healthy and progressing (emitting CTID state regularly). • Destination side: has successfully flushed multiple COPY batches and is mostly idle with a tiny queue. • Stuck behavior: likely in the job orchestration / finalization layer (e.g., waiting for some condition, or a long‑running operation not visible in these logs), not in the core read/write loops themselves. Concrete checks you can do (all are consistent with how similar issues were debugged in the docs/forum, but may not fully resolve it): 1. Look for long‑running queries on Redshift ◦ In similar “never ends” cases, the underlying warehouse query was still running even though Airbyte logs looked fine. [30GB Postgres load never ends] ◦ If you see a query from this sync running for a long time, try killing it and re‑running the sync. 2. Try isolating the stream ◦ Create a new connection with only

public_transfer

→ Redshift and run it. ◦ This is how support suggested isolating source vs destination issues in a stuck Postgres sync. [Postgres sync stuck] 3. Try a different destination ◦ Point the same Postgres source (same table) to another destination (e.g., S3) and run a test sync. [Postgres sync stuck] ◦ If that works, the problem is likely specific to Redshift/finalization. 4. Check for external factors that keep pods/jobs open ◦ In one Redshift case, a sidecar container kept pods from finishing, so syncs hung at “Executing finalization of tables” until the sidecar was removed. [Redshift destination hanging] Because the knowledge sources don’t document a specific bug for “CTID Postgres source + Redshift async writer stuck with small remaining queue”, I can’t give a precise fix. The best next step, aligned with documented troubleshooting, is: • Inspect Redshift running queries while the job is stuck. • Try an isolated connection (only this stream) and/or a different destination to see where the hang reproduces. If you can share whether an isolated

public_transfer

→ Redshift sync or

public_transfer

→ S3 sync completes, that would help narrow it further within what’s covered in the available docs. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.