Hi, using airbyte to stream data from places like shopify into Google Cloud Big-Query. Using the recommended Google cloud Storage and default normalized data option for transformations. The streams are slow and wondering if anyone has recommendation to optimize for faster streaming?
n
Nataly Merezhuk (Airbyte)
02/07/2023, 10:58 PM
Hi, Felix! If the speed is 2-3K rows per second, that is expected. For scaling Airbyte, check this doc out:
https://docs.airbyte.com/operator-guides/scaling-airbyte/
Speed is something we're focusing on this quarter, but I don't have any specifics for you yet!