Sean Zicari
04/28/2023, 2:31 PMkapa.ai
04/28/2023, 2:31 PMkapa.ai
04/28/2023, 2:32 PMJOB_MAIN_CONTAINER_MEMORY_REQUEST
and JOB_MAIN_CONTAINER_MEMORY_LIMIT
environment variables to custom values. This can help you manage memory usage more effectively [source](https://docs.airbyte.com/operator-guides/scaling-airbyte#memory). 2. You can increase the number of worker replicas to allow more jobs to be processed in parallel. This can be done in the Kubernetes deployment by increasing the number of replicas for the airbyte-worker
deployment [source](https://docs.airbyte.com/deploying-airbyte/on-kubernetes#increase-job-parallelism). 3. You can also configure the maximum number of each type of worker allowed to run in parallel by setting the environment variables MAX_SPEC_WORKERS
, MAX_CHECK_WORKERS
, MAX_DISCOVERY_WORKERS
, and MAX_SYNC_WORKERS
[source](https://docs.airbyte.com/understanding-airbyte/jobs#worker-parallization). For more information on scaling Airbyte, you can refer to the [Scaling Airbyte documentation](https://docs.airbyte.com/operator-guides/scaling-airbyte).