Hi @kapa.ai, We are going to move close to 300 gb of data using airbyte. For this we would like to buy a physical server. What compute power and memory should we go for? We will have less close to 10 connectors working at the same time.

To move close to 300 GB of data using Airbyte with around 10 connectors working simultaneously, you should consider the following recommendations for compute power and memory based on the information available: ### Compute Power and Memory Recommendations 1. CPU: - A multi-core CPU is recommended. Based on the performance tests mentioned in the knowledge sources, a setup with 32 CPUs was used for a similar task. Therefore, a server with at least 16-32 CPUs should be sufficient for your needs. 2. Memory: - Memory is crucial, especially for handling large data volumes and multiple connectors. The knowledge sources suggest that memory usage can be significant, particularly for the Source worker which reads up to 10,000 records in memory. - For a 300 GB data transfer, a server with at least 128 GB of RAM is recommended. This is based on the example where a 128 GB memory environment was used for a similar task. 3. Disk: - Ensure you have sufficient disk space to handle the data volume. A disk space of at least 1 TB is recommended to accommodate the data and any additional overhead. ### Example Configuration - CPU: 16-32 cores - Memory: 128 GB RAM - Disk: 1 TB SSD ### Additional Considerations - Environment Variables: You may need to customize memory settings for Java connectors by setting

JOB_MAIN_CONTAINER_MEMORY_REQUEST

and

JOB_MAIN_CONTAINER_MEMORY_LIMIT

environment variables to appropriate values. This ensures that the connectors can utilize the available memory efficiently. - Scaling: According to the [Scaling Airbyte documentation](https://docs.airbyte.com/operator-guides/scaling-airbyte), you should only encounter scaling issues when running over a thousand connections. With 10 connectors, you should be well within the safe limits. For more detailed information, you can refer to the [Scaling Airbyte documentation](https://docs.airbyte.com/operator-guides/scaling-airbyte). If you have any specific performance tuning questions or encounter issues, feel free to ask for further assistance!