https://linen.dev logo
Join Slack
Powered by
<@U088WCP3B4P> for report generation using `edr` c...
# ask-elementary-ai
l
@kapa.ai for report generation using 
edr
command, how far back does 
edr
collect data?
k
A message from kapa.ai
New message text here
l
@kapa.ai looking in the UI it seems that it's at least 10 days (UI doesn't let to zoom out more than that)
k
A message from kapa.ai
edr
itself still defaults to collecting 7 days of data when you run 
edr report
/ 
edr monitor send-report
without extra flags.[\[Troubleshooting\]](https://docs.elementary-data.com/oss/general/troubleshooting) The UI “zoom out” limit you’re seeing (around ~10 days) is a front-end display constraint and doesn’t change how far back 
edr
actually pulls data. To control how far back data is collected for the report, use:
Copy code
bash edr report --days-back 30
(or another number of days you want). This is the documented way to get 14, 30, 45 days, etc., into the report even if the default UI range looks shorter.[\[Slack: 7 days default\]](https://elementary-community.slack.com/archives/C02CTC89LAX/p1686465000985369;[Anomaly guide](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/Anomaly-troubleshooting-guide))
l
@kapa.ai Thanks, let's move on to another question. I want to create an anomaly that measures column relationships. For example, given models 
transaction
and 
products
, I want to know overtime the fraction of transactions with a product id not present in the products table
k
A message from kapa.ai
You can do this with Elementary, but it’s a two-step pattern: build a model that calculates the metric you care about over time, then put an Elementary anomaly test on that model. ### 1. Build a model for “fraction of bad transactions” Create a dbt model (e.g. 
transactions_missing_products
) that joins 
transactions
to 
products
and computes the fraction of transactions whose 
product_id
is not found in 
products
, ideally per day (or other time grain) so Elementary can use it as a time series:
Copy code
sql -- models/transactions_missing_products.sql select   t.transaction_date::date as ds,   count(*) as total_tx,   count_if(p.product_id is null) as missing_product_tx,   100.0 * count_if(p.product_id is null) / count(*) as missing_product_rate from {{ ref('transactions') }} t left join {{ ref('products') }} p   on t.product_id = p.product_id group by 1
Now 
missing_product_rate
is your “fraction of transactions with product_id not in products” per day. ### 2. Add an Elementary anomaly test on the metric Use a column anomaly test on 
missing_product_rate
to detect spikes/drops over time. Configure the model’s timestamp and the test in YAML:
Copy code
yaml version: 2  models:   - name: transactions_missing_products     config:       elementary:         timestamp_column: ds   # the date column in the model     columns:       - name: missing_product_rate         tests:           - elementary.column_anomalies:               column_anomalies:                 - average      # monitor the average value over time               time_bucket:                 period: day                 count: 1
elementary.column_anomalies
will track 
missing_product_rate
over time and alert when it deviates from its historical pattern. [[column anomalies](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/column-anomalies); [column anomalies config](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/column-anomalies#test-configuration)] ### Alternative: generic ratio via metrics If you prefer a more generic metric pattern (e.g., multiple dimensions), you can: 1. Build a model that has the numerator and denominator columns you care about. 2. Use 
elementary.collect_metrics
with a 
metrics
entry that computes a ratio (using an appropriate metric type from the metrics configuration). [[Metrics intro](https://docs.elementary-data.com/features/anomaly-detection/metrics)] However, the explicit “ratio model + `column_anomalies`” approach above is the clearest way, and it matches how Elementary suggests handling cross-table metrics and referential-style checks. [[Add tests examples](https://docs.elementary-data.com/data-tests/add-elementary-tests#adding-tests-examples); [Detection & coverage](https://docs.elementary-data.com/best-practices/detection-and-coverage)]
l
@kapa.ai 'm trying to specify arguments for my anomaly tests, it seems that 
arguments
keyword is not allowed under 
elementary.dimension_anomalies
as DBT test command complains with: macro 'dbt_macro__test_dimension_anomalies' takes no keyword argument 'arguments'
k
A message from kapa.ai
New message text here
l
@kapa.ai I want to define a dbt data recency tests. It should take a few parameters in: model, date/timestamp column, date part, interval and where condition for filtering
k
A message from kapa.ai
Elementary doesn’t provide a threshold-style “data recency”/SLA test with arbitrary parameters (date part, interval, where) beyond what’s documented for its freshness anomaly tests. Instead, you have two main options: --- ## 1. Use 
elementary.freshness_anomalies
(anomaly-based) This test learns the usual update cadence from history and alerts when freshness is anomalously worse (by default 
anomaly_direction: spike
). It does not take an explicit “interval must be < X hours” parameter.[\[freshness_anomalies\]](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/freshness-anomalies) Example config, parametrizing model, timestamp column and filter:
Copy code
yaml version: 2  models:   - name: your_model_name          # <model>     tests:       - elementary.freshness_anomalies:           timestamp_column: your_ts_col    # <date/timestamp column> (required)           where_expression: "your_condition"  # <where condition>, optional           time_bucket:                      # controls the bucket granularity             period: day                     # <date part: hour|day|week|month>             count: 1                        # <interval in that period>           # optional additional params:           # anomaly_sensitivity: 2           # detection_period:           #   period: day           #   count: 2           # training_period:           #   period: day           #   count: 14
Key points: - 
timestamp_column
is mandatory.[\[freshness config\]](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/freshness-anomalies#test-configuration) - 
where_expression
lets you filter the rows used for the calculation (e.g. a specific partition/window).[\[where_expression\]](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/where-expression) - 
time_bucket.period
and 
.count
play the role of a date part and interval, defining how data is sliced over time.[\[time_bucket\]](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/time-bucket) This is suitable when you want “alert me if the update pattern changes” rather than a hard SLA like “error if older than 24h.” Several users looking for strict “> X hours stale” behavior were pointed to dbt native freshness instead, because Elementary’s freshness tests are anomaly-based.[\[Slack: SLA vs anomalies\]](https://elementary-community.slack.com/archives/C02CTC89LAX/p1683294589948059;[Another SLA Q](https://elementary-community.slack.com/archives/C02CTC89LAX/p1708964023340349)) --- ## 2. Use dbt’s native source/model freshness for strict thresholds For a strict recency/SLA test (“fail if data older than N hours/days”) Elementary recommends using dbt’s native freshness config (especially on sources). Example from a user:[\[dbt freshness example\]](https://elementary-community.slack.com/archives/C02CTC89LAX/p1708964023340349)
Copy code
yaml sources:   - name: my_source     tables:       - name: my_table         loaded_at_field: "CAST(modifiedon AS TIMESTAMP)"  # <date/timestamp column>         freshness:           error_after: {count: 24, period: hour}          # <interval, date part>           filter: modifiedon > DATE_SUB(CURRENT_DATE(), INTERVAL 3 DAY)  # <where condition>
This natively supports: - 
period
/ 
count
= your date part & interval - 
filter
= your where condition and Elementary can surface dbt freshness results in its UI when mapped, as one user mentioned resolving their need this way.[\[mapped dbt freshness\]](https://elementary-community.slack.com/archives/C02CTC89LAX/p1708964023340349) --- Because the knowledge sources do not describe an Elementary test macro that takes exactly the parameter list you described as first‑class arguments (model, column, date part, interval, where) and enforces a hard SLA, I can’t give you a ready-made Elementary macro for that without making it up. The documented paths are: - anomaly-based: 
elementary.freshness_anomalies
- strict recency/SLA: dbt native freshness configuration.
l
@kapa.ai when I define a generic test, where can I define a description for the test?
k
A message from kapa.ai
You can define a description for a generic dbt test using the 
meta.description
field in the test’s YAML (or in the test SQL’s config block, for singular tests). ### 1. Generic test defined in YAML (schema.yml) Add 
meta.description
under the test configuration:
Copy code
yaml columns:   - name: my_column     tests:       - dbt_expectations.expect_column_values_to_not_be_null:           row_condition: "magnitud_km is null"           meta:             description: "This is the test description"
This pattern is what Elementary support recommends for customizing test descriptions.[[Change test description](https://elementary-community.slack.com/archives/C02CTC89LAX/p1678208351698859); [Volume anomalies description](https://elementary-community.slack.com/archives/C02CTC89LAX/p1702640567056559)] You can do the same with built‑in dbt tests:
Copy code
yaml tests:   - unique:       meta:         description: "The column should contain only unique values"
[[Custom description](https://elementary-community.slack.com/archives/C02CTC89LAX/p1708443408220759)] ### 2. Singular (SQL) tests For tests defined as 
.sql
files, set the description inside the 
config
block using `meta`:
Copy code
sql {{ config(     severity = 'error',     tags = ["test"],     meta = {       "description": "This test checks for wrong dates in the finance_daily_sales table."     } ) }}
[[Singular test config](https://elementary-community.slack.com/archives/C02CTC89LAX/p1715167583747759); [Singular meta example](https://elementary-community.slack.com/archives/C02CTC89LAX/p1679919138029609)] Elementary then surfaces this 
meta.description
in the UI and alerts for that test.
l
@kapa.ai I don't want to create a new description everytime I use the test, can I have global decription?
k
A message from kapa.ai
In dbt/Elementary today you cannot define a global description for a test type (e.g. “use this description for all 
unique
tests”) and have dbt automatically apply it everywhere. Elementary support explicitly notes that: - dbt does not allow adding `meta`/config “globally for different tests,” so you can’t set a single description once and reuse it for all uses of that test.[[Global descriptions not possible](https://elementary-community.slack.com/archives/C02CTC89LAX/p1670943766224959)] - As a partial workaround, Elementary ships some built‑in default descriptions for common dbt tests (and some dbt‑expectations tests), so if you don’t override 
meta.description
, those defaults will be shown in Elementary.[[Built‑in test descriptions](https://elementary-community.slack.com/archives/C02CTC89LAX/p1670943766224959)] So: - If you want a custom description, you must set 
meta.description
on each test usage. - If you’re fine with a standard wording, you can rely on Elementary’s built‑in descriptions for supported tests instead of defining your own.
l
@kapa.ai how do I remove certain tests? I don't want them to be present in the report anymore
k
A message from kapa.ai
To remove certain tests so they no longer appear in Elementary reports, you have a few options depending on what you mean by “remove”: ### 1. Disable the dbt test itself (recommended) If you don’t want a test to run or appear going forward, disable it in your dbt config (e.g. in the YAML where it’s defined):
Copy code
yaml tests:   - some_test_name:       enabled: false
Or using dbt’s shorthand:
Copy code
yaml tests:   - some_test_name:       +enabled: false
Once you change the config, run any dbt command (e.g. 
dbt run
or 
dbt test
); Elementary’s on-run-end hook will pick up the updated test definitions and update 
dbt_tests
, so the disabled test will no longer be considered active. [[Delete test handling](https://elementary-community.slack.com/archives/C02CTC89LAX/p1679692437666359); [Disable via enabled](https://elementary-community.slack.com/archives/C08HW57FGN9/p1757061385015699)] Historical results for that test remain in 
elementary_test_results
, but because Elementary joins against 
dbt_tests
(current definitions), deleted/disabled tests effectively disappear from the report. [[Delete test handling](https://elementary-community.slack.com/archives/C02CTC89LAX/p1679692437666359)] ### 2. Remove the test from your YAML If you physically delete the test from your schema YAML, the behavior is the same as disabling: - Next dbt run: Elementary re-uploads tests to 
dbt_tests
without that test. - Reports no longer show it because it no longer exists in 
dbt_tests
. [[Delete test handling](https://elementary-community.slack.com/archives/C02CTC89LAX/p1679692437666359)] ### 3. Exclude tests from runs (but keep them defined) If you only want to hide them by not running them anymore (and thus not generating new results): - Use dbt selectors in your CI / scheduled runs, e.g.:
Copy code
bash dbt test --exclude tag:sensitive
or for Elementary tests specifically:
Copy code
bash dbt build --exclude elementary dbt build --exclude test_name:schema_changes
[[Exclude elementary tests](https://elementary-community.slack.com/archives/C02CTC89LAX/p1693288324394649); [Exclude schema_changes](https://elementary-community.slack.com/archives/C02CTC89LAX/p1694770754861869)] Existing history may still be in the report for a while, but no new results will be added. ### 4. For 
edr report
email attachments with sensitive tests If the issue is sensitive tests in the HTML attachment, there is currently no way to filter specific tests out of the HTML; you can only filter the summary with selectors, or skip the HTML attachment entirely:
Copy code
bash edr send-report --disable html_attachment
This sends just the summary (which can already be limited via 
--select
), not the full detailed HTML with all tests. [[Report filtering limitation](https://elementary-community.slack.com/archives/C02CTC89LAX/p1696506753295779)] --- Most robust way to “not have them in the report anymore”: 1. Disable or delete the tests in dbt. 2. Run a dbt command so Elementary refreshes 
dbt_tests
. After that, they won’t appear as active tests in Elementary reports.
l
@kapa.ai what is 
zero_percent
column anomaly?
k
A message from kapa.ai
zero_percent
is a column anomaly monitor that tracks the percentage of zero values in a numeric column over time. - It’s one of the supported column monitors for 
elementary.column_anomalies
and 
elementary.all_columns_anomalies
and only applies to numeric columns.[\[Column monitors](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/column-anomalies)\] - Internally, Elementary computes the share of rows where the column value equals 
0
in each time bucket, then uses anomaly detection to flag significant deviations from the historical pattern.\[[All columns anomalies](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/all-columns-anomalies); [Column anomalies](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/column-anomalies)\] You enable it by listing it under 
column_anomalies
, for example:
Copy code
yaml columns:   - name: has_been_subscribed     data_tests:       - elementary.column_anomalies:           column_anomalies:             - zero_percent           timestamp_column: created_timestamp
(In YAML it must be in a list, e.g. 
column_anomalies: [zero_percent]
or as a list under it.)\[[Slack example](https://elementary-community.slack.com/archives/C08HW57FGN9/p1761301129998949)\]
l
@kapa.ai I've created an anomaly test with the following config:
Copy code
tests:
          - elementary.column_anomalies:
              column_anomalies: [average]
              tags: ["anomaly"]
In the UI two tests show up, one with test type 
generic
and another with type 
average
. Why is 
generic
showing up?
k
A message from kapa.ai
This happens when Elementary records both: 1. A result for the specific sub‑test (
average
in your case), and 2. A “fallback” / unspecialized result that gets classified as 
generic
. There are two documented situations where this occurs: 1. Anomaly test can’t compute a score (e.g., not enough data, wrong timestamp type, etc.) When Elementary can’t calculate an anomaly score for the configured monitor (like 
average
), it may log the result as a generic test instead of as the specific sub‑type, which is confusing but expected behavior today.[[Monthly bucket Slack](https://elementary-community.slack.com/archives/C02CTC89LAX/p1737832587221769?thread_ts=1737832587.221769&amp;cid=C02CTC89LAX); [Variance/stddev Slack](https://elementary-community.slack.com/archives/C02CTC89LAX/p1687331770365499?thread_ts=1687331770.365499&amp;cid=C02CTC89LAX)] Examples that can lead to this: - Not enough historical buckets to train (you’ll often see “Not enough data to calculate anomaly scores” in the logs).[[Anomaly training Slack](https://elementary-community.slack.com/archives/C02CTC89LAX/p1681298359181089?thread_ts=1681298359.181089&amp;cid=C02CTC89LAX)] - Misconfigured or missing 
timestamp_column
, or wrong data type for that column (e.g., string instead of timestamp/date).[[Timestamp type issue](https://elementary-community.slack.com/archives/C02CTC89LAX/p1737832587221769?thread_ts=1737832587.221769&amp;cid=C02CTC89LAX)] 2. UI misclassification / duplication of anomaly tests There are known UI issues where anomaly tests are counted or displayed twice or under the wrong type (e.g., freshness test shown once as 
freshness
and once as 
generic
, or only the last anomaly test per column shown).[[Freshness generic bug](https://github.com/elementary-data/elementary/issues/1557); [Multiple anomaly tests per column](https://github.com/elementary-data/elementary/issues/1772)] In those cases: - The underlying dbt/YAML definition is fine. - The duplication/`generic` label is a reporting bug rather than a configuration problem. Given your config is very simple:
Copy code
yaml tests:   - elementary.column_anomalies:       column_anomalies: [average]       tags: ["anomaly"]
the most likely explanations are: - Training/data issue causing one result to fall back to `generic`; check 
dbt test
logs for “Not enough data to calculate anomaly scores” and verify your 
timestamp_column
(if configured on the model) exists and is a proper date/timestamp type.[[Column anomalies test](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/column-anomalies)] - Or you’re hitting the known reporting bug where the same anomaly test appears as both its real sub‑type and as 
generic
in the UI.[[Freshness generic bug](https://github.com/elementary-data/elementary/issues/1557)] If you look directly in 
elementary_test_results
, you’ll probably see only one logical test definition, but with some rows having 
test_sub_type = 'average'
and possibly others marked as generic if they failed to compute properly.
l
@kapa.ai does 
timestamp_column
has to actual timestamp (time precision) or date type is ok?
k
A message from kapa.ai
A date type is fully OK. Elementary’s docs and Slack answers say the 
timestamp_column
should be a column that represents the creation/load time, and explicitly state that “date type also works” — it doesn’t have to be a full timestamp with time-of-day precision.[\[timestamp_column docs\]](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/timestamp-column) So you can safely use a 
DATE
(or warehouse equivalent) column as 
timestamp_column
, as long as it’s a proper date/datetime type (not a plain string).[[timestamp_column docs](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/timestamp-column); [Hourly anomaly example](https://elementary-community.slack.com/archives/C02CTC89LAX/p1673462017898399?thread_ts=1673462017.898399&amp;cid=C02CTC89LAX)]
33 Views
Open in Slack
PreviousNext
Powered by