Hi guys, i'm trying to do a full load from Mysql s...
# replication-troubleshooting
e
Hi guys, i'm trying to do a full load from Mysql source to Postgres destination, I've doing a lot syncs with small tables and it takes lake: 3 minutes to load 115,000 rows 44.6MB. but when I do some large tables the sync proces just hangs there , the screenshot says 23 and the other screenshot shows that it hangs at 68GB . It took 16 hours to get 68GB. I saw that there are some threads and posts at the forum talking about the performance connection but I couldn't find anything that helps thanks
✍️ 1
u
@[DEPRECATED] Marcos Marx turned this message into Zendesk ticket 2687 to ensure timely resolution!
s
Hey @Eduardo Aviles, can you share some more info? What version of Airbyte are you running? How much memory have you allocated for your instance? Can you share the logs? If you create a separate sync with just the table that it hangs on, is the sync successful?
e
Hi @Saj Dider (Airbyte), I'm using the latest version of airbyte 0.40.14 i'm running a EC2 (t3a.2xlarge) instance with: • 8 vCPUs • 32GB RAM • Ubuntu 22.04 • 1TB SSD I've tryed with different databases and tables, this database has 383 tables and all data is about 68GB, if tables are small airbyte finishes but still I think is really slow. I also tried the same MySQL source but instead to postgres to S3 ant it takes 15h to move all the data, about 157M records
s
What is the average row size for a few of the larger tables in your database? And have you considered splitting this large sync into smaller syncs?
e
these are the largest tables in this sync
It doesn't work for us to split the sync, we have a lot of databases ( multi tenancy ) with the same structure and we want to do a full load of this into hour datalake
s
Hey @Eduardo Aviles were you able to get answers to your questions here?
e
Hi, thanks for asking. Not really, some suggestions that I already tried like some memory configurations or something like that, but still don't know why in this scenario is really slow or why it hangs and never finishes
I've increase the memory for the connection and for the jobs but the source container for mysql memory usage is arround 350MB - 370MB during the process, so I really don't know if the configuration is working
j
@Eduardo Aviles - Any chance you could share the logs from the job in your screenshot?