Hi all, great work on Airbyte so far. We're trying it out at Iteratively and set up a simple configuration with sync from PgSQL to BigQuery on an EC2 instance every 15 minutes. The job ran fine 9 times but on 10th try failed with:

2020-10-19 18:36:33 INFO (/tmp/workspace/15/4) WorkerRun(call):58 - Executing worker wrapper...
2020-10-19 18:36:33 ERROR (/tmp/workspace/15/4) SingerSyncWorker(run):86 - Sync worker failed.
java.lang.RuntimeException: java.nio.file.FileSystemException: /tmp/workspace/15/4/target_config.json: Too many open files
at io.airbyte.commons.io.IOs.writeFile(IOs.java:50) ~[airbyte-commons-0.1.0-alpha.jar:?]
at io.airbyte.workers.protocols.singer.DefaultSingerTarget.start(DefaultSingerTarget.java:69) ~[airbyte-workers-0.1.0-alpha.jar:?]
at io.airbyte.workers.protocols.singer.SingerSyncWorker.run(SingerSyncWorker.java:71) [airbyte-workers-0.1.0-alpha.jar:?]
...

Have you seen this before?

Hi Ondrej, thanks for reporting the issue. What instance type & OS are you using on EC2?

Also, if it's convenient, can you run

lsof | wc -l

and

find /tmp/workspace

on the instance to see how many open files there are/which files there are?

Hi Shrif, Jared, we followed the AWS instructions exactly. I attached to the

airbyte-server

container,

lsof

isn't available,

find

returns 173 files. On the EC2 host,

lsof

returns 999 files, almost 89k if run with

sudo

. Happy to provide more info

Hi Ondrej, I’ve created an issue with this to track our investigation of this issue: https://github.com/airbytehq/airbyte/issues/685 I’m currently focused on reliably reproducing this issue. Is it still happening on your end?