Hello <@U088WCP3B4P>! So we have a weird interacti...
# ask-elementary-ai
a
Hello @kapa.ai! So we have a weird interaction where our staging elementary dashboard works perfectly and have lots of data, but our prod dashboard is nearly empty despite having daily runs of dbt tests and elementary. I've noticed something and would like you to confirm if my hypothesis may be right. Basically, in staging, our elementary dataset (which is stored on BigQuery) takes the id of the merge request followed by elementary, so something like "MR_152_elementary" for example. But in prod, my dataset is just called "elementary". I wanted a clean name so i've basically added a macro to get only "elementary" as the name, basically getting the schema value written in my dbt_project.yml (where it's obviously written "+schema: elementary"). But i'm starting to think that this may have been a bad idea, because i've been reading in the documentation an example where it says that for elementary models the suffix is '_elementary'. I've read this on the page "Elementary in production" in the documentation. So i'm thinking, is it mandatory to have it written like "something_elementary" for it to work and correctly recognise it as the elementary dataset for the dashboard ?
k
A message from kapa.ai
Your hypothesis is understandable, but it is not mandatory for the Elementary schema/dataset to have the "_elementary" suffix for the platform to work correctly. The "_elementary" suffix is a recommended convention to avoid mixing Elementary models with your other dbt models, but it is not a technical requirement enforced by Elementary. What is important is that the schema/dataset name you configure in your dbt_project.yml and the one you specify in your profiles.yml (for the Elementary CLI) match exactly where the Elementary tables are actually being written and read from. The Elementary CLI (edr) does not infer the schema name based on conventions or suffixes; it looks for the tables in the schema/dataset you explicitly provide in your profiles.yml under the elementary profile. If your macro sets the schema to just "elementary" in prod, then your profiles.yml for the Elementary CLI must also point to "elementary" as the dataset/schema for prod. The documentation recommends the "_elementary" suffix for clarity and separation, but it is not a requirement for the dashboard or CLI to function. The key is consistency between where dbt writes the Elementary models and where the CLI expects to find them. If your prod dashboard is empty, it is likely due to a mismatch between where the data is being written and where the CLI is looking for it, not because of the lack of a suffix. Double-check that your dbt runs and your edr commands are both targeting the same schema/dataset in prod. Here’s a relevant excerpt from the documentation: > To avoid mixing with your existing models, we recommend configuring a dedicated schema for the Elementary models using the dbt custom schema option. Here is an example configuration, that creates a schema with the suffix '_elementary' for elementary models: > > dbt_project.yml > >
Copy code
> models: >   elementary: >     +schema: elementary >
> > [Elementary in production](https://docs.elementary-data.com/oss/deployment-and-configuration/elementary-in-production) So, you can use "elementary" as your schema name, as long as it is consistent across your dbt and Elementary CLI configurations. The suffix is a best practice, not a requirement. If your prod dashboard is empty, focus on ensuring that both dbt and edr are using the same schema/dataset, and that your dbt tests are actually populating the Elementary tables in that location.
a
@kapa.ai Ok so i'm trying to investigate further because i'm not seeing any issues with the schema/dataset names, they are correct on both sides. So this is the cmd i'm using in Airflow :
Copy code
dbt test --store-failures -t composer-prod &&
              dbt run -s elementary --profiles-dir . --profile elementary -t elementary-prod &&
              edr send-report --env prod --profiles-dir . -t elementary-prod --gcs-bucket-name *** --bucket-file-path *** --update-bucket-website true -d 30 -ch *** -st *** &&
              edr monitor --env prod --profiles-dir . -t elementary-prod -ch *** -st ***
I've hidden some sensible information with *. So i'm wondering if i've been using elementary badly, because i'm running dbt run-s elementary each time after a dbt test. But do i really need it or can i move directly to the edr cmds ?
k
A message from kapa.ai
You do not need to run
dbt run -s elementary
every time after
dbt test
. According to multiple sources, the recommended practice is: - Run
dbt run -s elementary
only on initial setup or when upgrading the Elementary package to a new minor version
(i.e., when there are schema changes). This command creates or updates the necessary Elementary tables and views. - For regular operation, the on-run-end hooks and the Elementary CLI (
edr monitor
,
edr send-report
, etc.) handle the collection and reporting of test results. You can skip the
dbt run -s elementary
step in your daily/regular pipeline runs unless you are specifically updating the schema or have made changes to the Elementary package itself. Here are some relevant references: - "`dbt run --select elementary` is only needed to build the tables on installation or a minor version update. The on-run-end hooks populate these tables" [Slack answer](https://slack.com/archives/C02CTC89LAX/p1741272625948369). - "You need to run
dbt run -s elementary
on 2 occasions: 1. When you first install elementary dbt package on an env... 2. When you upgrade to a new Elementary minor version..." [Slack answer](https://slack.com/archives/C02CTC89LAX/p1685412868454149). - "There is no need to run
dbt run -s elementary
after first setup, except for versions updates, for all models to align" [Slack answer](https://slack.com/archives/C02CTC89LAX/p1711543686145199). In summary: You can safely remove the
dbt run -s elementary
step from your regular Airflow job. Just run it on initial setup or after upgrading the Elementary package. For daily runs, running
dbt test
followed by your
edr
commands is sufficient.
a
@kapa.ai Ok so i've removed the dbt run -s elementary cmd, hoping that it would mean my prod dashboard would now have data but this is still not the case. I would like you to clear a misunderstanding i have. Does the elementary dashboard's data comes from the run_results.json file generated by DBT, or does it come from the elementary dataset ?
k
A message from kapa.ai
The data shown in the Elementary dashboard comes from the tables in your Elementary dataset/schema in your data warehouse, not directly from the
run_results.json
file generated by dbt. Elementary uses dbt's on-run-end hooks to write results (such as test results, run results, etc.) into dedicated tables in your warehouse (e.g.,
elementary_test_results
,
dbt_run_results
, etc.). The Elementary CLI (
edr
) then reads from these tables to generate the dashboard and reports. The
run_results.json
file is not used directly by the dashboard; instead, its contents are loaded into the warehouse tables by the dbt package hooks and models. Here’s a relevant explanation: > all of the source data for the report is located in your
elementary
schema. Documentation on all of the models there can be found [here](https://github.com/elementary-data/dbt-data-reliability/blob/master/models/dbt_artifacts.yml). > Regarding the "Monitored tables" specifically, it is generated using a union between
dbt_models
and
dbt_sources
to find all the tables, then joining with
dbt_tests
to see which tables have monitors configured for them. > [Slack answer](https://slack.com/archives/C02CTC89LAX/p1707205953561139) And: > elementary_test_results - data about all test results (dbt, elementary, other packages) that Elementary needs for generating the UI. Data is loaded at on-run-end after tests are executed. (incremental) > dbt_run_results - data from the dbt results object, loaded at an on-run-end after each invocation that generates results. So you would get results for models, tests, snapshots, etc. (incremental) > [Slack answer](https://slack.com/archives/C02CTC89LAX/p1662059505282089) So, the dashboard does not read from
run_results.json
directly, but from the warehouse tables populated by the dbt package
. If your dashboard is empty, the issue is likely with data not being written to or read from the correct tables in your warehouse, not with the presence or absence of the
run_results.json
file.