Hey, we’re looking to use Airbyte to ETL data from BigQuery to Postgres, but the tasks are taking a very long time.
We currently have an in-house solution which extracts data to GCS and executes a COPY to load the into postgres, which takes about 10 minutes.
Unfortunately, the bigquery source -> postgres destination using Airbyte is taking over 3 hours.
We’ve not moving too much data, about 30M rows, about 3gb worth of data.
Are there any ways of speeding this up using these connectors?
Would an S3 -> postgres be faster?