Hi! :octavia-wave: We have been using Airbyte open...
# replication-troubleshooting
k
Hi! octavia wave We have been using Airbyte open source for several months now. Recently a new issue started plaguing our Postgres->Redshift CDC connector. Replication goes through, so does normalization, but the sync itself fails with a
Source cannot be stopped
error and then the next sync is a full one. Airbyte version: 40.28, Postgres version 1.0.36. Full stacktrace from logs in the thread.
stacktrace.txt
Another error snippet, that can be observed before connection to Postgres is closed
Copy code
at io.debezium.pipeline.ChangeEventSourceCoordinator.streamEvents(ChangeEventSourceCoordinator.java:174)
	at io.debezium.pipeline.ChangeEventSourceCoordinator.executeChangeEventSources(ChangeEventSourceCoordinator.java:141)
	at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:109)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.InterruptedException
	at java.base/java.lang.Object.wait(Native Method)
	at io.debezium.connector.base.ChangeEventQueue.doEnqueue(ChangeEventQueue.java:204)
	at io.debezium.connector.base.ChangeEventQueue.enqueue(ChangeEventQueue.java:169)
	at io.debezium.pipeline.EventDispatcher$StreamingChangeRecordReceiver.changeRecord(EventDispatcher.java:405)
	at io.debezium.pipeline.EventDispatcher$2.changeRecord(EventDispatcher.java:229)
	at io.debezium.relational.RelationalChangeRecordEmitter.emitUpdateRecord(RelationalChangeRecordEmitter.java:113)
	at io.debezium.relational.RelationalChangeRecordEmitter.emitChangeRecords(RelationalChangeRecordEmitter.java:53)
	at io.debezium.connector.postgresql.PostgresChangeRecordEmitter.emitChangeRecords(PostgresChangeRecordEmitter.java:96)
	at io.debezium.pipeline.EventDispatcher.dispatchDataChangeEvent(EventDispatcher.java:209)
	... 17 more
u
Hello Kristina, some questions to understand better the problem: do you have other connections with other sources? If not could you create one to test it? Can you upload the complete sync log file (it is easier to debug it), most cases the error message you pointed are a sequential one to the main issue. Also can you share a sync log from the period the sync was working?
👍 1
k
@Marcos Marx (Airbyte) after thorough investigation here is the verdict on this issue. TLDR: resolved, update for source connector required. We had a limit on job containers of 3GB. Incremental CDC syncs would fail after reaching over ~200k records per sync. We tried increasing the limit to 7GB - resolved issue partially, but it would still come up intermittently when syncs reached over ~500k or so records. Plus, our instance limitations of 16GB meant that we had to sacrifice other syncs to keep this one alive. Resolution: scaled limit back to 3GB, and updated Redshift destination to 4.0.4 which included periodic buffer flush feature, and updated Postgres source to 2.0.16 which included many fixes like Debezium retry policy and CDC sync failure fixes. Works great now! The Airbyte team did a great job fixing these issues! We still get the occasional historical full resync every couple of days, but not every time like before the update