https://linen.dev logo
Join Slack
Powered by
<@U04SE163WC8> how much time a connector with 7000...
# replication-ask-ai
a
@kapa.ai how much time a connector with 700000 records should take to sync data with bigquey api is fast and it fetch data in minutes
k
Hmm, I don't know enough to give you a confident answer yet. The time it takes to sync data can depend on various factors including the specific connector being used, the network conditions, and the configuration settings. However, based on the information available, here are some insights: 1. MySQL to BigQuery Sync: - In one case, syncing ~26M rows (~40GB) took about 9.5 hours, which translates to approximately 1M rows per 20 minutes ([source](https://discuss.airbyte.io/t/1092)). - Another test showed that syncing ~21M rows (~42GB) took between 7 to 12 hours ([source](https://discuss.airbyte.io/t/1092)). 2. Performance Considerations: - The performance of the sync can be influenced by the 
fetchSize
setting, but tests have shown that dynamic 
fetchSize
did not significantly impact performance ([source](https://discuss.airbyte.io/t/1092)). - Network conditions and the load on the database can also affect sync times. 3. API-Based Connectors: - For API-based connectors, the performance can be limited by the API's rate limits and the design of the API itself. For example, the Intercom API is designed for targeted requests and not for fetching all data at once, which can result in longer sync times ([source](https://discuss.airbyte.io/t/2089)). Given that your API is fast and can fetch data in minutes, the bottleneck might be in the data transfer and processing stages. You might want to consider the following to optimize performance: - Increase Resources: Allocate more memory and CPU to the Airbyte instance. - Parallelize Syncs: If possible, split the data into smaller chunks and sync them in parallel. - Optimize Network: Ensure that the network connection between the source and destination is optimized. For more detailed guidance, you can refer to the [Airbyte documentation](https://docs.airbyte.com/) or the specific connector documentation you are using.
a
For syncing (522 MB) data @kapa.ai
k
A message from kapa.ai
Hmm, I don't know enough to give you a confident answer yet. The time it takes to sync data can depend on various factors including the specific connector being used, the network conditions, and the configuration settings. However, based on the information available, here are some insights: 1. Performance Expectations: - In one case, syncing ~2GB of data took approximately 1 hour and 15 minutes ([source](https://github.com/airbytehq/airbyte/discussions/28726)). - Another user reported that syncing 300MB of data took approximately 140 seconds, which translates to a transfer rate of ~2.15MB/s ([source](https://discuss.airbyte.io/t/3631)). 2. Performance Considerations: - The performance of the sync can be influenced by the 
fetchSize
setting, but tests have shown that dynamic 
fetchSize
did not significantly impact performance ([source](https://github.com/airbytehq/airbyte/discussions/28726)). - Network conditions and the load on the database can also affect sync times. 3. API-Based Connectors: - For API-based connectors, the performance can be limited by the API's rate limits and the design of the API itself. For example, the Yandex Metrica API has a timeout period of 2 hours and is recommended to sync data once a day ([source](https://docs.airbyte.com/integrations/sources/yandex-metrica#performance-considerations)). Given that your API is fast and can fetch data in minutes, the bottleneck might be in the data transfer and processing stages. You might want to consider the following to optimize performance: - Increase Resources: Allocate more memory and CPU to the Airbyte instance. - Parallelize Syncs: If possible, split the data into smaller chunks and sync them in parallel. - Optimize Network: Ensure that the network connection between the source and destination is optimized. For more detailed guidance, you can refer to the [Airbyte documentation](https://docs.airbyte.com/) or the specific connector documentation you are using.
Open in Slack
PreviousNext
Powered by