Hi airbyte community, has anyone been able to resolve issues of Postgres<>BigQuery jobs always failing at Normalization step after upgrading airbyte? It seemingly gets all the way through normalization successfully and then just fails at the end. Job is able to run with normalization turned off. At this point it seems like our only option is to revert upgrades. Logs attached 🙏

Hello Nick, I didn’t find any relevant error in your logs beside the failure in normalization. Do you have other connections sending data to bigquery? What version were you using before for both connectors can you take a look in previous logs?

Thanks for taking a look @Marcos Marx (Airbyte) - here is a log from our old setup a couple months ago - we were running Airbyte

0.35.12-alpha

previously. I think I was able to upgrade both postgres source and bigquery destination to latest versions without issue. Big difference here is our jump in Airbyte version

Can I suggest to try doing only one table to test the sync? Maybe this give us some tips about the problem

Thanks @Marcos Marx (Airbyte) I've done that - here is an example

Sync one table worked?

I’ll note that it does look like some recent upgrades on our side moved us to a different data region within the US

I see, are you using staging method with bq?

I threw up my hands and wrote my own normalization step as this was a mission critical pipeline. But to answer your question yes we are using GCS staging with BQ

Any updates from my side I'll return to you!