Altan Sener
02/10/2025, 10:28 AMkapa.ai
02/10/2025, 10:28 AMvalues.yml
file. This can help improve sync speed, especially for small tables. For example: yaml global: edition: "enterprise" jobs: resources: limits: cpu: 250m memory: 500m requests: cpu: 75m memory: 150m
[Scaling Airbyte After Installation](https://docs.airbyte.com/enterprise-setup/scaling-airbyte) provides more details on this approach. 2. Adjust concurrent sync limits: You can increase the number of syncs that can run simultaneously by modifying the MAX_SYNC_WORKERS
and MAX_CHECK_WORKERS
environment variables: yaml worker: extraEnvs: - name: MAX_SYNC_WORKERS value: 5 - name: MAX_CHECK_WORKERS value: 5
3. Increase worker replicas: If you're running multiple syncs, increasing the number of worker replicas can help: yaml worker: replicaCount: 2
4. Use separate connections for small tables: As suggested in [this forum post](https://discuss.airbyte.io/t/1863), you can create separate connections for small tables. This can help with parallelization and performance, especially if you have a mix of large and small tables. 5. Consider using incremental sync modes: For tables that change frequently, using incremental sync modes like CDC (Change Data Capture) or xmin can significantly reduce sync times for subsequent syncs after the initial full sync. Remember that these configurations may need to be adjusted based on your specific setup and requirements. It's also important to monitor your system's performance and adjust accordingly.Altan Sener
02/10/2025, 11:09 AMkapa.ai
02/10/2025, 11:09 AMkapa.ai
02/10/2025, 11:09 AM