Hey :wave:
I'm testing Airbyte as a potential rep...
# contributing-to-airbytej
Johan Wärlander
01/13/2022, 5:01 PMHey 👋
I'm testing Airbyte as a potential replacement for AWS DMS, and have a question about performance..
When syncing one of our larger tables, at 38 million rows (so not by any means huge, just on the larger side for our current environment), from Postgres to Snowflake, it's taking 3 full hours to finish. This is at least 2x as long as I was expecting (the DMS jobs finish in 1:30 - 2:30 depending on other workloads).
What would be the best starting point for tweaking the settings to improve this?
This is on an EC2 m5.2xlarge instance (8 cores, 32MB RAM).. But I get similar performance on our rather small Kubernetes test cluster.
u
[DEPRECATED] Marcos Marx
01/13/2022, 10:13 PM
@Johan Wärlander maybe increasing the batch size: https://github.com/airbytehq/airbyte/issues/4314
j
Johan Wärlander
01/14/2022, 5:27 AMGreat, I'll try that! 😁
Johan Wärlander
01/14/2022, 5:31 AMJust to be clear, this is still in a state where I'd have to change the connector source code directly?
k
Khai Quang Nguyen
01/14/2022, 2:48 PMYes @Johan Wärlander you'll have to implement the patch yourself because the issue is not yet solved
3 Views