Hi folks, any tip to improve performance for sync ...
# replication-troubleshootingm
Maykon Lopes
10/19/2022, 7:19 PMHi folks, any tip to improve performance for sync big tables in mysql ? Tables around 300GB to 1TB of data.
✍️ 1
Maykon Lopes
10/19/2022, 7:22 PMI changed from 5 to 20 workers configuration in .env file, but seems that it's is more related to concurrent diferent jobs at same time.
# Relevant to scaling.
MAX_SYNC_WORKERS=20
MAX_SPEC_WORKERS=20
MAX_CHECK_WORKERS=20
MAX_DISCOVER_WORKERS=20
u
user
10/19/2022, 7:58 PM@[DEPRECATED] Marcos Marx turned this thread into Zendesk ticket 2951 to ensure timely resolution!
s
Sunny Hashmi (Airbyte)
10/21/2022, 4:10 PMMost of the info on improving performance is in this doc: https://docs.airbyte.com/operator-guides/scaling-airbyte
Also, please make sure you're on the newest Airbyte + mysql connector version as there are frequent performance improvements!
Sunny Hashmi (Airbyte)
10/21/2022, 4:13 PMThere is also an open ticket on improving performance from database source connectors, please leave a thumbs up if you're interested https://github.com/airbytehq/airbyte/issues/12532
m
Maykon Lopes
10/21/2022, 7:32 PMhi Sunny, thanks for sharing it. About workers I noticed that even having free workers to be used a single table extraction is slow, would be nice if we could split/paralelize extractions based on chucks and fetch size. For example, considering that I have 20 workers to be used, and only have a single connection to extract a single table, the ideal in my point of view will be having 20 workers in parallel extraction data in this single connection, maybe some logic to get number of rows in table and divide a portion of lines per worker?
77 Views