Airbyte is an open-source data integration engine that helps you consolidate your data in your data warehouses, lakes and databases.

Airbyte

Hi team - thanks in advance for your suggestions on this issue.

*Problem:* Airbyte has stopped logs for a given sync, and is in some sort of hung state.

*Environment:*
Airbyte self-hosted on GC, standard docker container

*Background:*
I have successfully synced a dozen &lt;100MB Postgres tables on this instance (to BigQuery). I attempted syncing a 350GB table. The process ran for many hours, then stopped logging. I have confirmed that the OS clock is accurate, so it seems like Airbyte stopped logging about 2 days ago, on Jan 17.

*Last few lines of logs, posting in thread below (I can attach entire log file if helpful)*

2023-01-17 20:00:19 [44msource[0m &gt; Reading stream concept_results. Records read: 436450000
2023-01-17 20:03:54 [32mINFO[m i.a.w.g.DefaultReplicationWorker(lambda$readFromSrcAndWriteToDstRunnable$6):321 - Records read: 436451000 (196 GB)
2023-01-17 20:06:14 [32mINFO[m i.a.w.g.DefaultReplicationWorker(lambda$readFromSrcAndWriteToDstRunnable$6):321 - Records read: 436452000 (196 GB)
2023-01-17 20:10:29 [32mINFO[m i.a.w.g.DefaultReplicationWorker(lambda$readFromSrcAndWriteToDstRunnable$6):321 - Records read: 436453000 (196 GB)
2023-01-17 20:12:51 [32mINFO[m i.a.w.g.DefaultReplicationWorker(lambda$readFromSrcAndWriteToDstRunnable$6):321 - Records read: 436454000 (196 GB)
2023-01-17 20:12:57 [32mINFO[m i.a.w.g.DefaultReplicationWorker(lambda$readFromSrcAndWriteToDstRunnable$6):321 - Records read: 436455000 (196 GB)
2023-01-17 20:12:59 [32mINFO[m i.a.w.g.DefaultReplicationWorker(lambda$readFromSrcAndWriteToDstRunnable$6):321 - Records read: 436456000 (196 GB)
2023-01-17 20:13:01 [32mINFO[m i.a.w.g.DefaultReplicationWorker(lambda$readFromSrcAndWriteToDstRunnable$6):321 - Records read: 436457000 (196 GB)
2023-01-17 20:49:16 [43mdestination[0m &gt; /airbyte/javabase.sh: line 26:     9 Killed                  /airbyte/bin/"$APPLICATION" "$@"
2023-01-17 20:49:16 [32mINFO[m i.a.w.g.DefaultReplicationWorker(lambda$readFromSrcAndWriteToDstRunnable$6):321 - Records read: 436458000 (196 GB)

<@U04EJMR8SDR> not sure about this, seems like stdin pipe is not connected any more

`$ top` indicates that airbyte processes are still running, supporting what you say

your log is referring here
<https://github.com/airbytehq/airbyte/blob/8c84ad2976125352240cbcdaa82e50227ffa2916/airbyte-integrations/bases/base-java/javabase.sh#L25>

Thanks - I'll look at that code once a meeting is finished.

Are you using an external S3 bucket for logging?

I am not. Everything lives on the VM, as far as I know.

Regarding the code snippet you shared - I'm not sure why this line would fail, when previous syncs were successful.

I checked disk usage on my airbyte VM, and it's healthy at 50% of total (via `$df -h`)

OK, I'd love to see the full logs if possible! Has the sync finished at this point? Was it successful?

Unfortunately, my Airbyte instance seems to have continued to degrade. The UI indicates two failed syncs. Toggling open the logs causes a "Oops, something went wrong" message to display. Additionally, the "Cancel Sync" button hangs. I'm planning to restart the server via `docker-compose down`.

Let me know what happens after you restart. Also - we have switched to using `docker compose`. You could also see logs in the server - are you able to get those?

The restart seems to have fixed the issue.