https://linen.dev logo
c

Chris (deprecated profile)

06/16/2021, 2:35 PM
About the Structured/Repeated fields in BigQuery… Since BigQuery does not allow
Arrays of Arrays of XXX
, you need to build
Arrays of Record of Arrays of XXX
instead I know Stitch is choosing to add a 
.value
 every-time you have a record/array field when creating the columns in BQ: https://www.stitchdata.com/docs/replication/loading/google-bigquery-storing-nested-data-structures#storing-nested-records Actually, whenever there is an
Array of
they automatically build a
Record of Array of
using that extra
.value
field. So you might end up with things like *`friend_ids.value.value` *in the case of arrays of arrays… I’m wondering how does other do it? or how should Airbyte do this?
cc @Ping / @Subodh (Airbyte) ?
p

Prashanth Patali

06/16/2021, 5:55 PM
I think the goal should be to produce exactly the same result as BigQuery does itself when you tell it to load newline-delimited JSON. I can experiment to see what it does with arrays of arrays.
2 Views