Hi Airbyte team! I've seen a few threads here and there about performance but nothing definitive. We would like to roll out airbyte, but are seeing some massive performance difference vs the current approach: Example dataset: full sync of 800MB / 6.5M records parquet in S3 -> Postgres Our current approach: load in pandas and push to postgres using https://github.com/d6t/d6tstack . Downside: high memory consumption due to reading data in pandas. We work around this by chunking. We're also looking at https://www.embulk.org as an alternative option to this custom code. But quite fast: full push to Postgres takes 15mins to read parquets, and only 5m30sec to push to postgres Tested the same dataset in airbyte: pull using S3 connector, record batch size set to 524288 -> push to postgres with full refresh takes 2h10mins. tested the same but push to Redshift with COPY strategy to see if that would improve, same performance I've read the optimization work is ongoing, and that 3-5m records / hour is within expected range. Is this still the case? Any way to further optimize? At this level of performance, the airbyte sync is too slow to be an option.

Hey Geert-Jan, thanks for the details. Our dev team is actively working on improving Airbyte performance — our internal testing matches your results almost exactly. Once our code freeze is over (January 3) the team will be pushing changes that have been determined to help with performance, but as of this moment you’re basically at the limit. Sorry for the lack of suggestions other than “give us two more weeks” but I promise our dev team is extremely focused on improving this ASAP.

Thanks @Jerri Comeau (Airbyte)! That's very good to know. I will keep an eye out for the next release then :-), as on the positive side: the user experience in terms of setting up the connection was very smooth. If you can get performance in the same range as above, we've love to move to airbyte as our general extract layer

Hi @Jerri Comeau (Airbyte), came across this thread while looking for info on how to improve Airbyte performance. Does the latest version 0.40.26 include the improvements you mentioned above?

Hey @Kamal Chavda! (And @Geert-Jan Van den Bogaerde) 0.40.26 includes some improvements to performance including an increase in the default RAM selection for workers, but some features (multithreading and parallel processing) are still in development/testing.

Thanks @Jerri Comeau (Airbyte)! I am running Airbyte in EKS and noticing that transferring fairly small tables from a postgres instance to redshift is taking a very long time ( as in running since yesterday afternoon and not complete yet). Is there any documentation that can help me tweak my deployment which can speed things up? (ex: increase # of workers, servers, etc.?)

Hm, good question; let me check with the dev team and see if I can find some reccomendations for tyou.

@Jerri Comeau (Airbyte) - i guess the RAM improvement alone would not be enough to make a significant difference in my use case above? seems like that would require the parallel processing? i'm looking for a 3-5x speedup, to be in the same range as our custom code. honestly (and i realize this is easy to say as an outsider, but still): not sure how we get to such a huge delta still? our own custom code pull from S3 is also serial, and it does the same thing in a fraction of the time

Hi @Jerri Comeau (Airbyte), wanted to follow-up to see if there are any recommendations to share.

@Jerri Comeau (Airbyte) - any idea on the timeline for some of the multithreading / parallel processing work in dev/testing?

Davin “Farragut” Chia from our dev team has been really active in improving our speed over the first half of Q1 — our internal testing shows that Airbyte is about 75% faster than it was at the beginning of the year (and we’re not done yet). Multithreading / parallel processing is what we’re actively working on now (now that checkpointing has been rolled out) and we’re hoping to get that feature out at the beginning of March. We know that this has been a real pain point for a lot of our users and we are definitely working on improving things as fast as possible.

Hi @Jerri Comeau (Airbyte) - Any ETA on when this feature will be available for OSS users ?

Given the vagaries of our release schedule I can’t give a specific date; we’ve had some fires that have caused a little slippage. By March 15th (or so) is what I’ve been told to say. 😃 Also, it should support Postgres out of the gate; MySQL is right behind that as I understand it.

Mssql and oracle right after?

@Jordan Fox I don’t know if Oracle is on the roadmap at this point; I know that MSSQL is next on the list. Oracle is an closed-source db that costs more than most of our customers’ quarterly operating budgets, so it’s not a priority for us right now. As our customers grow, we’ll revisit that. Also, we’ve been doing some testing with both the Airbyte application itself and with the Postgres connector that indicates we’re seeing speeds above 2MB/sec as of the latest build (0.42, released Friday). Our goal is to double that by the end of Q1 (🤞 )

That's a great goal! Glad to hear parallelization is going smoothe. Oracles new oracledb python library would be worth looking at if you guys do make it a priority in the future. It's a thin client and from my use, extremely fast and supports connection pooling.

Circling back on this to check if we can expect this feature to be released soon ?

Hey Srikanth, unfortunately we don't have any news to share about performance just yet but we'll let you all know in this thread when we have some news to share. Thanks for circling back

4 months after the last reply in this thread but I am also seeing the MSSQL -> S3 Parquet in the ranges discussed in this thread (Using Version 0.50.4). I submitted a ticket asking about this and got a response saying that the numbers (10 GB per hour speed) is what would be expected and that the support engineer did not see any existing tickets targeting parquet performance. Asking here to see if anyone else has heard or knows if there are efforts to improve performance of <source> -> S3 Parquet. Using Parquet with 780 MB of data (860,000 records) takes almost 5 minutes going to S3 Parquet but 1 and 1/2 minutes S3 CSV or S3 JSON.

At that point Larry I'd just set it up as json and have a lambda pick up files and do the conversion. Focus seems ease of use and building connectors, performance later. Not sure how much later is, but some can't wait.

Hi, any update on multithreading / parallel processing for s3 read? This still seems to be huge bottleneck on latest release (getting about 7 million records / hour reading parquets from s3. The same job in our own python code takes 15 minutes.

They released the postgresql updates and are moving onto mysql next I believe. It's not something that applies to all connectors at once. I've tested the updated postgres though and its significantly better.