Hi all, we started with using Airbyte on EC2 instance, we just created a small dev instance before moving to production to evaluate the work Our use case is to move streaming data from BigQuery to Snowflake, but unfortunately as soon as we try and push more data it fails We managed to move small amount of data easily (100.000 of rows in less than 2 min) but when we try and scale up (we tried with 1.000.000 rows - table had 12Gb) it failed and wasn’t able to finish -->

Job either timed out or was cancelled

Would be interested to know if anyone had similar issues and were able to solve it, we can scale up our instance and we plan to do it for prd, but not sure if this is the solution at the end?

Hey Dragan, could you give me more details on what Airbyte/connector versions you're using and give me the full logs that include this error?

Hi Nataly, so here is the issue and the logs will be shared in private message We scaled up and actually were able to push large amount of data, but when we try to sync even more, the query fails and it fails on BQ side, the error in question is

Response too large to return. Consider specifying a destination table in your job configuration

There is no configuration in Airbyte to enable this or maybe I am missing this, but this blocks us from pushing more data from A to B We were on t3.medium but that wasn’t enough and scaled up using BigQuery 0.2.2 and Snowflake 0.4.38

Thanks for the logs! I've found an open issue exactly for this: https://github.com/airbytehq/airbyte/issues/6533 I think the best course of action right now is to comment on the issue to help it gain traction, I don't see a suggested workaround right now.

Thanks, hopefully this is resolved relatively soon, since this is a major blocker for us to use Airbyte at the moment