@kapa.ai Are there parameters I can tune to speed up syncing from Postgres to Redshift?

New message text here

@kapa.ai I am observing that it takes 8 hours to sync 38GB of a 500+GB data source. Is there something I can do to improve the sync throughput?

A message from kapa.ai

A message from kapa.ai

Yes, there are several strategies you can employ to improve the sync throughput in Airbyte: 1. Resource Allocation: You can allocate more resources to the Airbyte instance. This can be done by adjusting the environment variables

JOB_MAIN_CONTAINER_CPU_REQUEST

,

JOB_MAIN_CONTAINER_CPU_LIMIT

,

JOB_MAIN_CONTAINER_MEMORY_REQUEST

, and

JOB_MAIN_CONTAINER_MEMORY_LIMIT

to give more resources to the pod running the sync. However, please note that simply increasing resources does not always lead to improved performance. It's important to monitor the usage of these resources to ensure they are being effectively utilized. [source](https://discuss.airbyte.io/t/1168) 2. Parallelization: If your data source consists of multiple tables, you can try to parallelize the sync jobs by creating one connection per stream. This will enable syncing multiple tables at the same time. [source](https://discuss.airbyte.io/t/1578) 3. Tuning Fetch Size: The

fetchSize

parameter determines the number of records fetched at a time from the source. Adjusting this parameter could potentially improve the sync speed. However, as of now, this parameter is dynamically set by Airbyte and cannot be manually configured. [source](https://discuss.airbyte.io/t/1578) 4. Identify Bottlenecks: It's important to identify where the bottleneck is occurring. It could be on the source side, the destination side, or somewhere in between. Once the bottleneck is identified, you can take targeted actions to alleviate it. For example, if the source is the bottleneck, you could look into optimizing the source database or using a more performant source connector if available. [source](https://discuss.airbyte.io/t/1578) Please note that the effectiveness of these strategies can vary depending on the specifics of your data source, destination, and the resources available to your Airbyte instance.

New message text here

@kapa.ai How do I tune

fetchSize

?

A message from kapa.ai

Currently, in Airbyte, the

fetchSize

parameter is dynamically set and cannot be manually configured. The

fetchSize

parameter determines the number of records fetched at a time from the source. Adjusting this parameter could potentially improve the sync speed. However, as of now, this parameter is dynamically set by Airbyte to optimize the balance between memory usage and throughput. [source](https://discuss.airbyte.io/t/1578) If you believe that adjusting the

fetchSize

could improve your sync performance, you might want to follow the discussions and updates on this topic in the Airbyte community or GitHub repository. The Airbyte team is actively working on improving database connectors and there might be future improvements related to

fetchSize

computation. [source](https://discuss.airbyte.io/t/1578)