https://linen.dev logo
#troubleshooting
Title
# troubleshooting
s

Saman Arefi

03/15/2022, 1:58 PM
Hi everyone, hope you're all having a sensational day. Could I get some pointers regarding Airbyte's scalability? The docs recommend a
t2.large
instance and describe, in details, how Airbyte is mainly memory and disk bound. I've been testing stuff out now on an
t3.xlarge
and noticed the following: Loading one large-ish Oracle table (~9GB, 7M rows) takes me about 30min, which I think is pretty good. Now, loading two at the same time via the same connector (9GB, 7M rows, 13 GB, 7M rows) takes an hour in total, with both taking up roughly an hour each. What gives? Looking at htop, I seem to be running more into a CPU limit as well, so I'm not sure what's causing this. These are my two largest table, but in production I'd use Airbyte for another 30 or so tables, each between 10k and 1M rows as well, so this doesn't seem to scale well. Or am I doing something wrong?
a

Augustin Lafanechere (Airbyte)

03/15/2022, 3:06 PM
Hi @Saman Arefi, the read and write operations on streams in a single connection are currently run sequentially, in other word Airbyte start by syncing the first table, then the second one. A workaround could be to create a different connection for each table and tweak the
MAX_SYNC_WORKERS
env var to increase sync parallelism.
s

Saman Arefi

03/15/2022, 3:15 PM
Thanks for the speedy reply @Augustin Lafanechere (Airbyte)! ☺️ Would you mind further clarifying "connection" in this context? This is what my setup currently looks like. Two connections, both connected to the same Oracle DB, each loading a different table. Launching both at the same time resulted in what I described above.
a

Augustin Lafanechere (Airbyte)

03/18/2022, 3:28 PM
Ok so you already implemented the workaround I suggested 😄 If you have more than 5 connections increasing the
MAX_SYNC_WORKERS
value to >5 might help. I'd suggest also you try to upsize your instance to a t3.2xlarge to check if you get some performance boost.
s

Saman Arefi

03/18/2022, 4:24 PM
Yeah, I think I'll try that, cheers. I noticed that I currently also only have memory usage of up ~50% with 4 connections going at the same time, yielding me a throughput of only ~20GB/h - aren't both of those values quite low?
a

Augustin Lafanechere (Airbyte)

03/18/2022, 5:20 PM
Do you mean 50% memory usage of the container request or of the overall instance memory? Feel free to tweak your docker engine config to give more memory to containers or use the
JOB_MAIN_CONTAINER_MEMORY_REQUEST
env var.
2 Views