hey everyone!
Quick question, regarding dbt ingest...
# ingestionb
bright-window-42671
08/27/2021, 7:06 PMhey everyone!
Quick question, regarding dbt ingestion, does datahub pull in anything in the meta variable in the schema.yml? Right now it isn't even pulling in the column descriptions from this file. I know they are written correctly because they are showing up in the dbt docs but not in the metadata that datahub pulls in (code in thread)
bright-window-42671
08/27/2021, 7:06 PM Copy code
version: 2
models:
- name: dm_applications
columns:
- name: application_id
description: A unique ID given to each application
meta:
description2: another description
contains_pii: true
tests:
- unique
- not_nullm
mammoth-bear-12532
08/27/2021, 7:33 PM
Hey @bright-window-42671: in this case is there a bigquery table backed by this model?
mammoth-bear-12532
08/27/2021, 7:34 PM
the current ingestion system is sensitive to the order in which you run ingestion for dbt and the underlying platform, since the "schema metadata" is clobbered by the last writer
mammoth-bear-12532
08/27/2021, 7:36 PM
one quick way to check what is happening is to set the
sink Copy code
sink:mammoth-bear-12532
08/27/2021, 7:36 PM
Copy code
sink:mammoth-bear-12532
08/27/2021, 7:37 PM
ugh!!
mammoth-bear-12532
08/27/2021, 7:37 PM
sink:
type: file
config:
filename: dbt_mces.json
mammoth-bear-12532
08/27/2021, 7:37 PM
and re-run dbt ingestion
mammoth-bear-12532
08/27/2021, 7:38 PM
if you look at the dbt_mces.json that are generated, that should give you a good idea of what it is pulling out and what is being ignored
b
bright-window-42671
08/27/2021, 8:48 PMthanks for the suggestions @mammoth-bear-12532. This model pull from a snowflake instance. I'll try digging into the json output and see what I find