Please help me get an approximate calculation. If ...
# ask-aic
Charles Bockelmann
02/10/2025, 3:53 PMPlease help me get an approximate calculation. If I deploy Airbyte OSS in a GCP VM with n2-standard-4 configuration. How many requests can I process per month? I don’t need specific numbers; ranges work.
k
kapa.ai
02/10/2025, 3:54 PMTo answer your question about the approximate number of requests that can be processed per month with Airbyte OSS deployed on a GCP VM with n2-standard-4 configuration, I'll need to provide some context and caveats based on the available information. First, it's important to note that the number of requests Airbyte can process depends on various factors, including the types of connectors used, the complexity of your data, and how you've configured Airbyte. The n2-standard-4 configuration provides a good starting point, but it may not be the sole determining factor for performance. According to the [Scaling Airbyte documentation](https://docs.airbyte.com/operator-guides/scaling-airbyte), Airbyte's control plane components are generally efficient and optimized for their jobs. The documentation states: "In general, you would only encounter scaling issues when running over a thousand connections. As a reference point, the typical Airbyte user has 5 - 20 connectors and 10 - 100 connections configured." Based on this information, we can infer that an n2-standard-4 configuration should be able to handle a significant number of requests per month for a typical use case. However, without more specific information about your particular setup and requirements, it's difficult to provide an exact range. That being said, if we consider the typical user scenario mentioned in the documentation, we can make a rough estimate: - Assuming 100 connections - Each connection syncing daily This would result in approximately 3,000 sync jobs per month (100 connections * 30 days). Keep in mind that this is a very rough estimate and could vary significantly based on your specific use case. Some factors that could affect this estimate include: 1. The frequency of your syncs (hourly vs. daily) 2. The size and complexity of your data 3. The types of connectors you're using 4. Any custom configurations or optimizations you've implemented If you need to process significantly more requests, you may need to consider scaling your deployment. The [Scaling Airbyte documentation](https://docs.airbyte.com/operator-guides/scaling-airbyte) provides guidance on how to scale Airbyte for larger workloads. For a more accurate estimate tailored to your specific use case, I recommend running some tests with your actual data and connectors to gauge performance and adjust your infrastructure accordingly.
c
Charles Bockelmann
02/10/2025, 3:56 PM@kapa.ai Let’s say we have 1 source connector and 1 destination connector, both API connectors. Give me new estimations please. Thanks
k
kapa.ai
02/10/2025, 3:56 PMA message from kapa.ai
kapa.ai
02/10/2025, 3:56 PMBased on your specific scenario with 1 source connector and 1 destination connector, both being API connectors, I can provide a more focused estimation. However, it's important to note that the performance can still vary depending on the specific APIs, data volume, and sync frequency. According to the [Scaling Airbyte documentation](https://docs.airbyte.com/operator-guides/scaling-airbyte), Airbyte's performance is primarily affected by the number of concurrent syncs running at any given time. With only one connection (1 source and 1 destination), you're unlikely to run into scaling issues on an n2-standard-4 GCP VM. Let's consider some potential limitations based on the information provided: 1. API Rate Limits: The main constraint in your case is likely to be the rate limits of the APIs you're connecting to, rather than Airbyte itself. For example, the [Google Sheets API](https://docs.airbyte.com/integrations/sources/google-sheets#limitations--troubleshooting) has a limit of 300 read requests per minute per project and 60 requests per minute per user per project. 2. Airbyte's Processing: With a single connection, Airbyte should be able to handle a high number of requests per month. The n2-standard-4 configuration (4 vCPUs, 16 GB memory) should be more than sufficient for this workload. Given these factors, here's a rough estimation: - Assuming your APIs allow for frequent syncs, you could potentially run syncs every hour (24 times per day). - This would result in approximately 720 sync jobs per month (24 syncs * 30 days). However, this is a theoretical maximum. In practice, you might sync less frequently, perhaps a few times a day, which would result in 90-180 sync jobs per month (3-6 syncs per day * 30 days). Keep in mind that the actual number of API requests during each sync job depends on the amount of data being transferred and how the specific connectors are implemented. For more accurate estimates, you should: 1. Check the rate limits and quotas for your specific API connectors. 2. Consider the volume of data you're syncing and how often it needs to be updated. 3. Run some test syncs to measure the actual performance and adjust your sync frequency accordingly. Remember, with this setup, you're more likely to be limited by API rate limits than by Airbyte's capacity on the n2-standard-4 VM.