Chris (deprecated profile)

06/16/2021, 2:35 PM
About the Structured/Repeated fields in BigQuery… Since BigQuery does not allow
Arrays of Arrays of XXX
, you need to build
Arrays of Record of Arrays of XXX
instead I know Stitch is choosing to add a 
 every-time you have a record/array field when creating the columns in BQ: Actually, whenever there is an
Array of
they automatically build a
Record of Array of
using that extra
field. So you might end up with things like *`friend_ids.value.value` *in the case of arrays of arrays… I’m wondering how does other do it? or how should Airbyte do this?
cc @Ping / @Subodh (Airbyte) ?

Prashanth Patali

06/16/2021, 5:55 PM
I think the goal should be to produce exactly the same result as BigQuery does itself when you tell it to load newline-delimited JSON. I can experiment to see what it does with arrays of arrays.