Nice demo, even though I only caught the last part 😳 How many levels does the basic normalization do? e.g. in Snowflake if I have a VARIANT col that then has nested attributes that also may have nested attributes etc. At some point these need to go into separate tables, so curious what you guys chose to do here

i know it does at least 1 level, but the Mongo source is broken slightly, so I'm not sure if it would go farther if it cloud

the 2nd level array/object is not in JSON syntax, so it just dumps giant strings

It depends mostly on the source if the source describes the level of nesting, then normalization can extract/explode these in separate tables recursively This is why from the demo of @[DEPRECATED] Marcos Marx, he started with only one raw stream

_airbyte_raw_poke_pokemon

but he ended up with multiple normalized tables: (some are multiple nested level deep)

In the case of source-mongo, there’s discussion on it in order to provide more complete schemas into Airbyte’s catalog so normalization can leverage it: https://github.com/airbytehq/airbyte/issues/3428#issuecomment-871527949

Otherwise, because the source says: here is a column and all it says it’s that it’s a big object blob, then normalization doesn’t know how to unnest it (and does not do data scans etc to discover it either)