@kapa.ai, how can I scale large jobs in Airbyte to finish faster in parallel with Airbyte OSS in AWS EKS?

To scale large jobs in Airbyte OSS and improve performance on AWS EKS, you can try the following approaches: 1. Increase worker replicas: You can increase the number of worker replicas to run more jobs in parallel. Adjust the

replicaCount

in your Helm chart or Kubernetes configuration:

yaml worker:   replicaCount: ## e.g. 2

2. Adjust concurrent sync limits: Increase the

MAX_SYNC_WORKERS

and

MAX_CHECK_WORKERS

environment variables to allow more concurrent jobs:

yaml worker:   extraEnvs:     - name: MAX_SYNC_WORKERS       value: ## e.g. 5     - name: MAX_CHECK_WORKERS       value: ## e.g. 5

3. Increase resource allocation: Provide more CPU and memory to job pods by modifying the

values.yml

file:

yaml global:   jobs:     resources:       limits:         cpu: ## e.g. 250m         memory: ## e.g. 500m       requests:         cpu: ## e.g. 75m         memory: ## e.g. 150m

4. Scale your EKS cluster: Ensure your EKS cluster has enough nodes to handle the increased workload. You may need to adjust your node group settings or enable cluster autoscaling. 5. Optimize connectors: Some connectors may have specific settings that can improve performance. For example, database connectors might benefit from adjusting the

fetchSize

parameter, although this may not be directly configurable in all cases. Remember that the optimal configuration depends on your specific workload and infrastructure. You may need to experiment with different settings to find the best performance for your use case. For more detailed information on scaling Airbyte, refer to the [Scaling Airbyte documentation](https://docs.airbyte.com/enterprise-setup/scaling-airbyte).