To run two

airbyte-worker

instances on the same machine to improve performance, you need to consider the following points based on the provided knowledge sources: 1. Scaling Workers: - Airbyte workers are responsible for executing operations like Discover, Read, Sync, etc. Each job has a corresponding worker executing its work. In Docker deployment, an Airbyte worker spins up at least one Docker container. Scaling Airbyte involves ensuring that the Docker container running the jobs has sufficient resources to execute its work ([source](https://docs.airbyte.com/operator-guides/scaling-airbyte#what-to-scale)). 2. Configuring Worker Parallelization: - You can configure the maximum number of each type of worker allowed to run in parallel by setting environment variables. This can help you run more jobs in parallel and increase the workload of your Airbyte instance. The relevant environment variables are: -

MAX_SPEC_WORKERS

-

MAX_CHECK_WORKERS

-

MAX_DISCOVERY_WORKERS

-

MAX_SYNC_WORKERS

- The default value for these environment variables is set to 5 ([source](https://docs.airbyte.com/understanding-airbyte/jobs#configuring-jobs--workers)). 3. Running Multiple Instances: - Running multiple Airbyte instances on the same server using Docker can be challenging due to internal code dependencies and container naming conflicts. A user reported issues with containers restarting and errors related to host not found in upstream configurations when attempting to run two instances ([source](https://github.com/airbytehq/airbyte/discussions/32809)). 4. Horizontal Scaling: - For horizontal scaling, especially in a Docker deployment, you might need to consider using orchestration tools like Kubernetes, which supports multiple workers and can handle scaling more gracefully. Airbyte supports multiple workers on Kubernetes deployment ([source](https://github.com/airbytehq/airbyte/discussions/28588)). Given these points, here are the steps you can take to run two

airbyte-worker

instances on the same machine: 1. Increase Worker Parallelization: - Adjust the environment variables to allow more workers to run in parallel. For example, you can set the

MAX_SYNC_WORKERS

to a higher value if sync jobs are your bottleneck.

bash    export MAX_SYNC_WORKERS=10

2. Consider Kubernetes for Better Scaling: - If you are facing issues with Docker deployment, consider using Kubernetes for better handling of multiple workers and scaling. Kubernetes can manage multiple pods (workers) efficiently. 3. Resource Allocation: - Ensure that your machine has sufficient resources (CPU, memory, disk space) to handle the increased number of workers. Start with a mid-sized cloud instance (e.g., 4 or 8 cores) and tune the instance size based on your workload ([source](https://docs.airbyte.com/operator-guides/scaling-airbyte#what-to-scale)). If you need more detailed guidance or run into specific issues, you can refer to the [Airbyte documentation](https://docs.airbyte.com/operator-guides/scaling-airbyte#what-to-scale) or seek help from the Airbyte community.