I'm testing out syncing a bunch of tables from a m...
# replication-troubleshooting
m
I'm testing out syncing a bunch of tables from a mysql db to aws s3 in parquet using snappy compression. It's taking a long time and I was wondering maybe I didn't set something up right or it just takes hours and hours. I was wondering if anyone could point me to documentation or describe how to sync faster. One thought I had was to create a few pipelines sync different tables at the same time...
n
Hi Michael! Could you include the full logs? I need more info to understand what's happening here.
m
Sure, I'll figure out how to get the logs and post it here shortly
It says I'm unable to download server logs at this time, maybe due to the sync still running?
On a different not, is there a way to export / import my pipeline that I have created?
I see it mention here" https://discuss.airbyte.io/t/import-export-configuration/611, but I don't see these options
image.png
From https://docs.airbyte.com/operator-guides/scaling-airbyte/, I read that:
Copy code
Our Java connectors currently follow Java's default behaviour with container memory and will only use up to 1/4 of the host's allocated memory. e.g. On a Docker agent with 8GBs of RAM configured, a Java connector limits itself to 2Gbs of RAM and will see Out-of-Memory exceptions if this goes higher. The same applies to Kubernetes pods. You may want to customize this by setting JOB_MAIN_CONTAINER_MEMORY_REQUEST and JOB_MAIN_CONTAINER_MEMORY_LIMIT environment variables to custom values.
What percentage of the ec2 memory should be allocated to each setting?
j
kinda aside: I’ve also never been able to download the server logs locally using docker compose
n
Sync logs are enough for this - the expected sync speed should be 2-3K rows per second. If it's below that, there's a performance issue.
You can find the sync logs right on the connection page.
m
by changing the instance size and increasing these two vars JOB_MAIN_CONTAINER_MEMORY_REQUEST and JOB_MAIN_CONTAINER_MEMORY_LIMIT to 28g, I am getting 5-6k rows per second now
n
Great to hear!