A complete solution for open data platforms, enterprise data catalogs, data lakes and data management. Open source, mature, fully-featured and production ready.

DataHub

Hi all
:snowflake:*Snowflake-dbt* *issues*
Here are some I keep encountering and can't find solution or even sense to:
1. in the dbt recipe, when using `entities_enabled.sources: 'NO'`, the sources, which are then considered as datasets, don't get the snowflake symbol
2. My dbt entities always have their urns in uppercase. Oddly, when ingesting snowflake with `convert_urns_to_lowercase: False`, the snowflake entities are separate from their dbt counterpart. When not using that flag, they merge together.
3. When ingesting snowflake with lowercase urns, some tables appear twice in the lineage view as the up/downstream of each other (two, identical squares - same stats, urns, etc,).
I would appreciate any tip regarding any of these issues

While I am not fully versed in these sources, I believe the general recommendation is to lowercase consistently when dealing with sources whose case can vary.

<@U01GZEETMEZ> Do you mind providing some guidance for Ben?

1. Sources should _not be ingested_ at all if entities_enabled.sources is set to 'NO'. Are you saying that these are still being ingested? 
2. When this flag is _not_ provided, then Snowflake URNs will be lower cased. DBT typically produces lower cased URNs so this is why they are aligning! 
3. Do you mind sending a screenshot of this? My hunch is that stateful_ingestion was not enabled, and so when you changed from uppercasing to lowercasing, new entities were minted on DataHub (duplicates). Stateful ingestion configurations should prevent that if its enabled. Can you confirm that you are not configuring stateful ingest?

Just curious, do you have multiple dbt projects where one model could be the source of others?

Hi <@U01GCJKA8P9>
1. Yes, when using `sources: 'NO'`, the sources are ingested as regular datasets. It’s good for me since it allows me to ingest lineage from s3 to them (as opposed to when they are ingested as sources). It’s just weird that datasets don’t get the Snowflake symbol and sources do (I would expect it to be the other way around).
2. I agree and am aware of that. In our Snowflake wharehouse, all names are fully UPPERCASE, so our dbt project adheres to that - I think that’s why dbt is ingested UPPERCASE. That’s also why I used the flag for the Snowflake ingestion. What’s weird is that, against my (I think reasonable) expectation, when snowflake is ingested UPPERCASEly, the entities end up separate from dbt entities. When ingest lowercasely, they merge properly.
3. Indeed, I’m not setting `stateful_ingestion`. Do you recommend setting it in dbt as well?
.
Thanks a lot!

Hi <@U04BRQ3JZH7>, are these issues still affecting you here?

3. Yes I do recommend.

1. <@U01GZEETMEZ> Do you understand why this is happening?

re 1 - that behavior definitely seems a bit strange. One possibility is that if you ran multiple dbt ingestions with different configs, once with sources enabled and then a subsequent one without, the previous “merge association” data between dbt and snowflake might still be lingering

I was thinking exactly that: Since stateful ingestion was not previous enabled, the sources that you may have ingested on an earlier run may still be in your system