<@U088WCP3B4P>, please see if you can help here: I...
# ask-elementary-ai
j
@kapa.ai, please see if you can help here: I'm using the Elementary package to measure the coverage of descriptions using the
dbt_models
and
dbt_columns
tables. In
dbt_columns
, a blank description is shown as
null
, while in
dbt_models
, a blank description appears as an empty string (
''
). I understand that both tables are updated during
on-run-end
, and I found the macro that seems to update the descriptions using
node_dict.get('description')
for models and
column_node.get('description')
for
dbt_columns.description
. However, I'm wondering where exactly is the
node
object being taken from? Is it derived from
manifest.json
,
catalog.json
(generated by `dbt run`/`build`)? btw, in manifest all blank description is empty string and in catalog.json all blank description(comment) is null
k
A message from kapa.ai
Great question! Based on the available knowledge sources, here’s what can be confirmed: - The Elementary dbt package loads metadata about models and columns into the dbt_models and dbt_columns tables, respectively. This metadata includes descriptions, which are extracted from your dbt project’s YAML files and dbt artifacts. - The process for populating these tables involves macros that extract fields from the dbt artifacts during execution and insert them into the tables. Specifically, the documentation states that Elementary uploads fields from the run-results and the graph object, which includes data from manifest.json, catalog.json, and sources. The relevant models are updated either via on-run-end hooks or as post-hooks on the models themselves, depending on the artifact type. Regarding your specific question about where the node object is sourced from: - The “node” (for models) and “column_node” (for columns) are derived from the dbt graph object, which is a combination of the manifest.json, catalog.json, and sources. This means that the macro is pulling the description field from these artifacts, and the behavior you observe (null vs. empty string) is a result of how these fields are represented in the underlying dbt artifacts: manifest.json uses an empty string for blank descriptions, while catalog.json uses null for blank comments/descriptions. To summarize: - The node object is sourced from the dbt graph object, which is built from manifest.json, catalog.json, and sources generated by dbt run/build. The difference in how blank descriptions are represented (null vs. empty string) reflects the underlying dbt artifact structure, not a bug in Elementary’s extraction logic. References: - [How Elementary uploads dbt artifacts?](https://docs.elementary-data.com/dbt/dbt-artifacts) - [dbt artifacts models and their data sources](https://docs.elementary-data.com/dbt/dbt-artifacts#dbt-artifacts-models) If you need more detail on the exact macro logic, the documentation does not provide the full macro code, but it does confirm the artifacts used as sources for these tables.
j
@kapa.ai thank you for the answer. So we can say dbt_models description is from manifest.json so blank description is empty string, while dbt_columns description is from catalog.json so blank description is null. right? is it possible that you show me the exactly source code of gathering nodes for each
k
A message from kapa.ai
Based on the available knowledge sources, here’s what can be confirmed: - The dbt_models table in Elementary is populated with model metadata, including descriptions, from the dbt project's current graph, which is built from dbt artifacts such as manifest.json and catalog.json. However, the documentation does not explicitly state that dbt_models.description comes exclusively from manifest.json or that dbt_columns.description comes exclusively from catalog.json. It only states that the graph object includes data from manifest, catalog, and sources, and that the macros extract the relevant fields from these artifacts during execution [How Elementary uploads dbt artifacts?](https://docs.elementary-data.com/dbt/dbt-artifacts) [dbt artifacts models and their data sources](https://docs.elementary-data.com/dbt/dbt-artifacts#dbt-artifacts-models). - Your observation about blank descriptions being empty strings in manifest.json and null in catalog.json is consistent with dbt’s artifact structure, but the knowledge sources do not explicitly confirm that Elementary always uses manifest.json for dbt_models and catalog.json for dbt_columns. - Regarding the exact source code for how nodes are gathered for each table: The knowledge sources reference macros that extract fields from the artifacts, but they do not provide the full source code or the exact macro logic. There are some code snippets in Slack discussions showing that the macro uses node_dict.get('description') for models and column_node.get('description') for columns, but the full context or file is not provided [source](https://slack.com/archives/C02CTC89LAX/p1698150269410739). In summary: - It is likely that dbt_models.description is sourced from manifest.json (empty string for blank), and dbt_columns.description is sourced from catalog.json (null for blank), but the knowledge sources do not explicitly confirm this mapping or provide the exact source code for node gathering. - The documentation confirms that both manifest and catalog are used as part of the dbt graph object, but does not detail the precise extraction logic or file locations. If you need the exact source code, you may need to review the Elementary dbt package source directly (e.g., the macros in the package repository), as the documentation and Slack threads do not include the full implementation.
j
@kapa.ai, Is there any particular reason for this difference?
k
A message from kapa.ai
I'm sorry, but the knowledge sources do not contain any explicit information about the reason for the difference in how blank descriptions are represented (empty string in dbt_models vs. null in dbt_columns). The sources confirm the observed behavior and discuss where the data is extracted from, but they do not provide an explanation or rationale for this implementation detail. If you need a definitive answer, you may need to consult the Elementary or dbt source code directly or reach out to the maintainers for clarification.