Airbyte is an open-source data integration engine that helps you consolidate your data in your data warehouses, lakes and databases.

Airbyte

Does anybody experience that Airbyte closes the stream before the stream is finished? Details are included in the thread.

Here is the log:
```2022-12-05 23:11:59 INFO i.a.w.g.DefaultReplicationWorker(lambda$getReplicationRunnable$6):352 - Records read: 397000 (7 GB)
2022-12-05 23:12:19 source &gt; Killed
2022-12-05 23:12:20 destination &gt; 2022-12-05 23:12:20 INFO i.a.i.b.FailureTrackingAirbyteMessageConsumer(close):60 - Airbyte message consumer: succeeded.
2022-12-05 23:12:20 destination &gt; 2022-12-05 23:12:20 INFO i.a.i.d.b.BufferedStreamConsumer(close):201 - executing on success close procedure.
2022-12-05 23:12:20 destination &gt; 2022-12-05 23:12:20 INFO i.a.i.d.s.SnowflakeStagingSqlOperations(insertRecordsInternal):26 - actual size of batch for staging: 4379
2022-12-05 23:12:25 destination &gt; 2022-12-05 23:12:25 INFO i.a.i.d.s.SnowflakeInternalStagingConsumerFactory(lambda$onCloseFunction$3):165 - Finalizing tables in destination started for 1 streams
2022-12-05 23:12:25 destination &gt; 2022-12-05 23:12:25 INFO i.a.i.d.s.SnowflakeInternalStagingConsumerFactory(lambda$onCloseFunction$3):170 - Finalizing stream **. schema **, tmp table **, final table **
2022-12-05 23:12:19 INFO i.a.w.g.DefaultReplicationWorker(lambda$getReplicationRunnable$6):355 - Source has no more messages, closing connection.
2022-12-05 23:12:19 INFO i.a.w.p.KubePodProcess(close):737 - (pod: internal-analytics-test / dd-airbyte-read-10198-0-zyspz) - Closed all resources for pod
2022-12-05 23:13:19 INFO i.a.w.p.KubePodProcess(destroy):664 - (pod: internal-analytics-test / dd-airbyte-write-10198-0-eapyl) - Destroying Kube process.
2022-12-05 23:13:19 INFO i.a.w.p.KubePodProcess(close):737 - (pod: internal-analytics-test / dd-airbyte-write-10198-0-eapyl) - Closed all resources for pod
2022-12-05 23:13:19 WARN i.a.c.i.LineGobbler(voidCall):119 - airbyte-destination gobbler IOException: Socket closed. Typically happens when cancelling a job.
2022-12-05 23:13:19 INFO i.a.w.p.KubePodProcess(destroy):670 - (pod: internal-analytics-test / dd-airbyte-write-10198-0-eapyl) - Destroyed Kube process.
2022-12-05 23:13:19 ERROR i.a.w.g.DefaultReplicationWorker(run):188 - Sync worker failed.
java.util.concurrent.ExecutionException: io.airbyte.workers.general.DefaultReplicationWorker$SourceException: Source cannot be stopped!
	at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) ~[?:?]```

2 issues here:
1. This is not the full record. Don’t know why it says “Source has no more messages, closing connection.”.
2. “Sync worker failed.” with the reason “Source cannot be stopped!”. Is it because the sync is still going on? Then why does it try to kill the source?

I was running into something like this when I ran out of memory on my airbyte server. I ended up having to double the memory available and have been pretty solid since then.

Oh, thanks. I'll try to increase memory on my worker. The error message isn't clear though.

Agree, I only found it because I am running on an Azure VM and happened to go to the memory monitor when I was running my connection.