Title
#feedback-and-requests
u

user

07/07/2021, 6:40 PM
Nice demo, even though I only caught the last part 😳 How many levels does the basic normalization do? e.g. in Snowflake if I have a VARIANT col that then has nested attributes that also may have nested attributes etc. At some point these need to go into separate tables, so curious what you guys chose to do here
6:45 PM
i know it does at least 1 level, but the Mongo source is broken slightly, so I'm not sure if it would go farther if it cloud
6:45 PM
the 2nd level array/object is not in JSON syntax, so it just dumps giant strings
7:00 PM
It depends mostly on the source if the source describes the level of nesting, then normalization can extract/explode these in separate tables recursively This is why from the demo of @[DEPRECATED] Marcos Marx, he started with only one raw stream
_airbyte_raw_poke_pokemon
but he ended up with multiple normalized tables: (some are multiple nested level deep)
Ryan Thomson

Ryan Thomson

07/07/2021, 7:03 PM
In the case of source-mongo, there’s discussion on it in order to provide more complete schemas into Airbyte’s catalog so normalization can leverage it:https://github.com/airbytehq/airbyte/issues/3428#issuecomment-871527949
u

user

07/07/2021, 7:04 PM
Otherwise, because the source says: here is a column and all it says it’s that it’s a big object blob, then normalization doesn’t know how to unnest it (and does not do data scans etc to discover it either)