I commented on the github issue already but also wanted to bring up here how great it would be to have relationship tests against sources have a table displayed. Anyway Elementary is awesome and thanks for building it!

Hi all. We are just implementing a DQ framework in our DBT pipelines and just found Elementary. Looks an amazing project. Quick question product feature question.. Can we take the results of the Elementary tests and use them to update our models by 'tagging' the data it's testing, e.g. with a DQ flag indicating the row fails a DQ test? Reason for asking is we are trying to implement the write audit publish (WAP) pattern where we inspect the raw data and publish it as source tables into a tested 'silver' layer . We are using Databricks. Ideally we'd want to mark rows (each row is an event in the raw data) that fail a DQ check so consumers can decide on whether to use the data. Elementary seems to have the types of tests we want to run and great reports but can it be used to 'tag' the data it's testing?

Hey 👋 I'm having problems with copy result query from Test results. All

<=

and

>=

are replaced by

>=

or

<=

Because of that all queries have invalid syntax and can't be executed. I've checked the compiled tests in dbt and they are ok. I'm using elementary 0.9.2 and dbt package version 0.9.0

Hello everyone. Are you guys developing an integration for elementary and astronomer cosmos ?

I don’t think this is currently possible, but here is a problem I wish Elementary could solve: Let’s say you have a test for uniqueness that fails in a staging model. This causes failures in downstream models as well for unique values. It would be great to be able to see those dependencies somehow and know that the failure in staging is causing the other failures and if you fix that one all others will resolve. We spend a lot of time chasing down the causes of test failures when this happens. It happens in more complicated testing scenarios too and would be nice to know ‘all these tests are related/failing for the same reason, no need to troubleshoot them all’

Hey, I'm having problems with dashboard on initial load

If I had one wish, it would be that the

dbt_columns

table contained column-level tags, as we use these in our staging and transformation layers to indicate primary keys. While

dbt_tests

will give me this tag whenever we have a uniqueness test on a primary key, any model where uniqueness is determined by a unique combination of columns lacks this tag, as the test is associated at the model level and not the column. Are there any plans for expanding the utility of

dbt_columns

in the future?

👋 Hello, team!

Any plans to support Lightdash in the near future?

Do we have a dashboard which shows the source freshness

Hello team, I have a question about the alerts configuration. Do you have any plan to use the dbt’s groups information to send alerts of elementary? dbt 1.5+ is supporting the groups feature as a part of the features of data contracts. Personally, it would be good to take advantage of the information in the

groups

property to send elementary’s alert, as we can configure the

owner

,

subscriber

and

channel

.

groups:
  - name: finance
    owner:
      # 'name' or 'email' is required; additional properties allowed
      email: <mailto:finance@jaffleshop.com|finance@jaffleshop.com>
      slack: finance-data
      github: finance-data-team

https://docs.getdbt.com/docs/build/groups

Elementary gives a really good visibility in case "tests" fails for specific models. Somehow I cannot say the same for model runs failures. When a model run fail I would like to see the the reason of the failure. The information is currently available in the table

dbt_run_results

inside the message field. Adding something similar to: •

View tests runs

and

Inspect last result

will be nice to have to for the models runs. Not sure if this is something that you already planned or already requested, but as part of dbt observability tool I would like to have as much as I can also in term of model runs. PS: I'm not really sure if this is the place to ask for feature or give feedback 😄 Thanks.

Hello All, I was thinking about feature which estimates ETL completion time, incase of some failure, restart of ETL or full load, so we can inform our stackholders that data will be available in x minutes/hours.

Hi, I just introduced elementary to the rest of my team and everyone is really enjoying it. So far, I have heard 2 feature requests a few times and I think both are great ideas (if they aren't already done somehow and I am missing something) 1. More visibility into models without tests. It would be great to be able to generate or see a list of models without tests and also to toggle "untested" or "no-test" in the lineage graph. It would be nice to see how untested data is affecting the up/downstream models. 2. The ability to see tests that "fail" in the ui without actually having the job fail. So, an additional status, like "Fail" - "Would have failed" - "warn" - "pass" , where the "would have failed" was flagged as a failure but pushed through in the job. Thanks for the great product!

hi everyone! thanks for the great product! Is it possible to track test duration the same way as models? It would be helpful for us to spot long-running tests

Hey Elementary Team, Thank you for taking the time to make all the features that are available in this open-source package. Saves people like a me a lot of time when wanting to test data quality. My only question/comment/concern is this: I feel that the detection method for anomalies isn’t quite aligned with how Z scores/t scores are used. The training period should be isolated from the detection period to calculate the sample mean (assumed population mean) otherwise if we include the data point being tested in the mean calculation then the Z score will definitely be skewed by the new test data point. For example, if I’m trying to perform a volume anomaly test for a table that I know receive a single monthly manual upload, I would calculate an average of the previous X months worth of data (I’m assuming they were successfully uploaded). Then for the upcoming month I would want to test if this new volume data point statistically belongs to that “population” of successfully uploaded table volumes. Including the new data point in the sample mean calculation doesn’t seem to make sense as it could damper the Z score which might not be caught with the default anomaly_sensitivity. Just thought I’d bring this up and if I’m totally wrong I would love to learn where my assumptions/understandings are wrong. Thanks Elementary!

Hey folks blob wave. We’ve been working on replacing our current notification systems for dbt models with Elementary’s. However, a few of our analysts have requested that we leave the on success messages that we already send out. These basically give them a summary saying that their model has run successfully in x seconds. They use this in the morning for an at-a-glance check that everything went well with their pipelines. Is there any chance that Elementary will be adding this as a feature in the future? 🤞

Some notes regarding dbt-elementary. I just thought it may be interesting for community. I'm running now every dbt model as separate airflow task using K8sPodOperator. In such setup elementary on-run-start and on-run-end hooks take 1-2 minutes. Let's say it's 1,5 minutes average. For the project with many models (~350) and for hourly processing schedule interval it becomes a problem. Especially for simple models with usual run duration up to 1 minute. So possible solution may be to group models somehow and run them grouped. However we may face some obstacles in this case: • we'll lose model-level lineage in airflow • logging in airflow will become strange, so need to find a way how to work with rendered queries in case of failures • retries in dbt may be a proble (I did not try this feature yet, so it's time to check it out). I suppose that setting specific

DBT_TARGET_PATH

for every models group may help in this case Will appreciate any thoughts, suggestions regarding optimal elementary usage on big dbt projects with ~low latency required.

On more thing regarding dbt multiprojects: I've managed to setup smth like

elementary-umbrella

dbt project which contains only views which actually are union from source dbt-elementary projects models. Using

dbt_utils.union_relations

macro it was quite easy to do. Thanks to such approach I can now generate 1 elementary report which covers all my dbt projects on 1 web page.

hey Team, after upgrading to 0.14.1 we have an issue: on-run-end hook is very slow: it takes up to 3 mins vs 10 seconds on version 0.11.2. Is there a way to understand what slows it down?

Hi all! We are happy users of Elementary but there is one design decision that we keep bumping into: the fact that everything is uploaded using

on-run-end

hook, i.e., only after the whole dbt run has finished. Are there any plans of changing that in the not too distant future, i.e., update the internal tables during the run? (Sorry if this topic is already covered somewhere — searched but didn’t manage to find here nor in Github issues. Background in the thread 🧵)

Hi Elementary team, I tried 0.16.0 which was just released. I haven’t identified the root cause. But, let me share the error I encountered on an

edr report

command. Even though I set the dbt profile

elementary

in the dbt project,

edr report

couldn’t find it.

edr report \
                --env "dev" \
                --profiles-dir . \
                --profile-target "dev" \
                --target-path "edr_target" \
                --file-path "edr_target/index.html"

2024-08-05 17:03:46 — INFO — Running with edr=0.16.0
{"data": {"exc": "Runtime Error\n  Could not find profile named 'elementary'"}, "info": {"category": "", "code": "Z002", "extra": {}, "invocation_id": "219d2dd9-1811-47dd-9b96-503ea63307ee", "level": "error", "msg": "Encountered an error:\nRuntime Error\n  Could not find profile named 'elementary'", "name": "MainEncounteredError", "pid": 47466, "thread": "MainThread", "ts": "2024-08-05T08:03:47.056099Z"}}
2024-08-05 17:03:47 — ERROR — Unable to get the latest invocation: Failed to run dbt command.
Runtime Error
  Could not find profile named 'elementary'

hi there! Could anyone advice if I can use different backend type for elementary models? Currently I have all models materialised in AWS Athena. And I want to have elementary models inside AWS RDS (which is actually Postgres). Is it possible ? I'll provide my

dbt_profiles.yml

in thread

My experience regarding support for the open source solution of elementary has drastically dropped in the last couple of months. More than half a year the support was great and collaboration was also fine, but recent times this is a lot poorer. I understand you guys have to focus on your cloud solution in order to become a viable company, however some sla's on PR / issues in Github would be really appreciated. For example I filed a bug and have not heard of it since the 4th of august. More recent we also opened a PR and a ticket for something and we would like to know if we are wasting our time on improving the project or not. If open source contribution is no longer of interest for you guys at least let us know so, then we can decide what we do with it.

Hi Elementary Team, I’ve been following the MCP Server release—really exciting direction. I wanted to share a recurring challenge we face and see if it’s something Elementary could help address down the line. We use Fivetran, Snowflake, dbt, and Elementary to manage our stack. One of our biggest pain points is tracking the downstream impact of changes made in upstream source systems like Salesforce, HubSpot, NetSuite, and Zendesk. Fivetran gives us logs when schemas change (like renamed or dropped fields), but there’s no current way to tie those upstream changes directly to the dbt models and reports they impact—at least not before something breaks. Would Elementary consider building a feature that helps map source-level schema changes (detected through Fivetran or another integration) to the dbt models and exposures that rely on them? Even better if those changes could trigger proactive alerts before they cause downstream failures. This is strongly desired by our platform owners(Salesforce Developers/Admins/Product Owners for example). This kind of source-to-model observability would fill a major gap and give us stronger confidence in making changes upstream without accidentally breaking things downstream. Right now, no single tool we've found fully owns that "upstream-to-downstream proactive impact" space. We could be patching it together in a custom way with: • Fivetran logs + dbt contracts + Elementary observability It’s doable, but clunky. A feature like MCP Server could bridge that gap—if it starts tying source changes directly into model-level lineage and alerting. I thought about this when I saw your update/release today. Happy to chat more if it’s helpful. Just wanted to put this on your radar. Thanks, Dan Halabe https://elementary-community.slack.com/archives/C02BPB78J4T/p1754322932249029

Hey Elementary Team, I wonder if it is possible to route alerts only to the owner, but not to the subscriber if a test fails for technical reasons (eg. connection or syntax error)? Non-data stakeholders get quite confused if they receive an alert, but no further info than “Test failed: …” I saw there was a similar discussion in the past.

Hello everyone, We are currently using Elementary with Slack integration - anyhow recently, due to the broader adoption of PagerDuty in the company we were researching how to integrate PagerDuty in different systems, Elementary is one of them (at least for some relevant data-models). I understood that PagerDuty comes with the cloud support, anyhow switching to cloud only for that it seems an overkill to us - it will be hard to have leadership buy in for that. I'm wondering if a PR that integrates PagerDuty in the open source version will be accepted. Happy to discuss this more @Itamar Hartstein if necessary, before propose a a PR.

how do I remove the "Powered by Elementary" message and the "*elementary-data.com*" links users are complaining that it's creating noise in our channel. thanks! cc @Ella Katz @Or Avidov @Maayan Salom found you guys since you guys are pretty active in the channel

We run dbt Core and Elementary on Jenkins. The Elementary upload step takes 17 minutes, while the dbt run itself takes 23 minutes. Why is the upload so slow, and how can we improve it? cc @Maayan Salom @Noya Offer

poetry run edr report --env prod --profiles-dir ../ || echo "WARNING: Elementary report generation failed, continuing..."

                                    # Create elementary directory and move files
                                    if [ -d "./edr_target" ]; then
                                        mkdir -p ./elementary
                                        mv ./edr_target/elementary_report.html ./elementary/index.html || echo "WARNING: Failed to move Elementary HTML report, continuing..."
                                        mv ./edr_target/elementary_output.json ./elementary/elementary_output.json || echo "WARNING: Failed to move Elementary JSON report, continuing..."

                                        aws s3 sync --region=us-west-2 ./elementary <s3://prod-ba-finance-dbt-docs/elementary> --quiet || echo "WARNING: Elementary S3 sync failed, continuing..."

                                        poetry run edr monitor --profiles-dir /app --project-dir /app/intacct_models --env prod --report-url <https://ba-finance-dbt.prod.internal.equipmentshare.com/elementary/#> || echo "WARNING: Elementary monitoring failed, continuing..."
                                    fi

Hi, which command specifically is slow for you? The upload to s3 or the report generation?