What's the role of Temporal Ports parameter in Airbyte worker configs? How does Airbyte use these temporal ports and how it takes part in scaling.

The

TEMPORAL_WORKER_PORTS

parameter in Airbyte worker configurations defines the local ports that the Airbyte Worker pod uses to connect to various Job pods. Specifically, ports 9001 - 9040 are exposed by default in the Helm Chart for Kubernetes deployments ([source](https://docs.airbyte.com/operator-guides/configuring-airbyte#worker-1)). ### Role of

TEMPORAL_WORKER_PORTS

1. Port Allocation: The

TEMPORAL_WORKER_PORTS

parameter specifies the range of ports that the Airbyte Worker can use to communicate with job pods. This is crucial for the worker to manage and execute tasks effectively. 2. Configuration Locations: This parameter can be defined in both the

.env

file and the

deployment.yaml

of the Airbyte worker. The ports specified in these configurations determine the pool of ports available for the worker to use ([source](https://discuss.airbyte.io/t/4074)). ### How Airbyte Uses These Ports - Job Execution: When a job is initiated, the Airbyte Worker uses one of the ports from the specified range to establish a connection with the job pod. This allows for the execution and management of the job. - Concurrency Management: The number of ports available in the

TEMPORAL_WORKER_PORTS

range can impact the number of concurrent jobs that can be run. If the number of available ports is less than the number of concurrent jobs, it can lead to job hang-ups or failures ([source](https://discuss.airbyte.io/t/1077)). ### Scaling with Temporal Ports - Concurrency Scaling: To scale the number of concurrent jobs, you need to ensure that the

TEMPORAL_WORKER_PORTS

range is sufficiently large to accommodate the maximum number of concurrent workers (

MAX_*_WORKERS

). If the port range is too small, it will limit the number of jobs that can run concurrently, thus affecting the scalability of the system ([source](https://discuss.airbyte.io/t/1077)). For more detailed information on how Airbyte scales workflow orchestration with Temporal, you can refer to the blog post on [How we scale workflow orchestration with Temporal](https://airbyte.com/blog/scale-workflow-orchestration-with-temporal). If you have further questions or need more detailed configuration guidance, you might find additional insights in the [Scaling data pipelines on Kubernetes](https://airbyte.com/blog/scaling-data-pipelines-kubernetes) blog post.

Thanks

Sorry @Talha Naeem, I'm not sure what you mean. Please try again. 👋

@kapa.ai I'm working on benchmarking Airbyte for 100 concurrent syncs. Do we need to consider any network performance metrics?

When benchmarking Airbyte for 100 concurrent syncs, it is indeed important to consider network performance metrics. However, based on the information available, network conditions might not significantly impact data sync speed in some scenarios. For instance, in a test where Airbyte was launched on AWS EC2 instead of GCP VM to have Airbyte in the same VPC as the test database, the results did not show any significant difference in sync performance. This suggests that network conditions might not be the primary bottleneck in some cases (Source: [Forum Post: MySQL source connector performance](https://discuss.airbyte.io/t/1092)). However, it is still advisable to monitor network performance metrics such as: 1. Network Throughput: Ensure that the network bandwidth is sufficient to handle the data transfer rates required for 100 concurrent syncs. 2. Latency: Lower latency can improve the performance of data syncs, especially if the source and destination are geographically distant. 3. Packet Loss: High packet loss can lead to retransmissions and reduced sync performance. Additionally, consider the following points: - Instance Resources Utilization: Ensure that the instances running Airbyte have sufficient CPU, memory, and disk I/O capacity. In one case, it was noted that only <10% of instance resources were utilized during a sync, indicating that the bottleneck might not be the network (Source: [Forum Post: Sync performance](https://discuss.airbyte.io/t/1578)). - Connector Configuration: The performance of each connector can depend on its configuration and the API being replicated. Adjusting parameters like

fetchSize

can impact performance, although in some tests, dynamic

fetchSize

did not show a noticeable impact (Source: [Forum Post: MySQL source connector performance](https://discuss.airbyte.io/t/1092)). In summary, while network performance metrics are important, they might not always be the primary factor affecting sync performance. It is crucial to also consider instance resource utilization and connector configurations. For more detailed guidance, you can refer to the Airbyte documentation and community forums: - [Airbyte Documentation](https://docs.airbyte.com/) - [Airbyte Community Forums](https://discuss.airbyte.io/) If you have specific performance issues or need further assistance, sharing detailed sync logs and metrics can help the community and support team provide more targeted advice.