loading...

loading...

Maybe, this issue could be of interest to you: https://github.com/airbytehq/airbyte/issues/10260 We are not currently tackling throughput yet on that issue though

I read the convo @Chris Duong [Airbyte], however it is really specific to snowflake. In general I just wonder how to detect any bottlenecks and how to fix/improve the bottleneck. For e.g. if I check the logs, it seems that most of the time is spend on reading records from the source, however what are the ways to improve that? Is it IO bound or CPU/memory bound or in general how to determine such cases?

Yes, the issue/discussion is focused on snowflake, but we’ll now be slowly releasing the same set of changes to the other destinations after snowflake too. So once the memory usage is stabilised over most of the connectors, we can probably expect the next steps to further investigate and work on the connectors in general and optimize other aspects as you are pointing out (cpu, IO, etc) IF you want to contribute, you can also experiment on aspects we should be looking into and share your findings from your side, that’d be helpful! For example, I’m guessing we could do something about more parallelisms (when buffering records as they come from the source for example) since the CPU does not seem to be that much used at the moment… but as I said, I looked into memory not other dimensions yet