https://linen.dev logo
Join Slack
Powered by
hi <@U088WCP3B4P> please share the detailed exampl...
# ask-elementary-ai
o
hi @kapa.ai please share the detailed example with description for dimension_anomalies if there are id, date, stock_qty and loaded_at columns
k
A message from kapa.ai
Here’s a concrete pattern you can adapt to a table with: - 
id
- 
date
(business / event date) - 
stock_qty
- 
loaded_at
(when the row was loaded) ### What 
dimension_anomalies
will do here `elementary.dimension_anomalies`: - Counts rows grouped by the configured 
dimensions
(e.g. 
date
, or combinations like 
date, stock_qty_bucket
). [[Dimension anomalies intro](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/dimension-anomalies)] - Tracks those counts over time buckets based on a 
timestamp_column
(usually your load/partition timestamp). [[Dimensions config](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/dimensions); [Dim anomalies config](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/dimension-anomalies#test-configuration)] - Alerts if the distribution of counts per dimension changes unexpectedly (spikes/drops). [[Add tests – dimension](https://docs.elementary-data.com/data-tests/add-elementary-tests)] With your columns, a common setup is: - Use 
loaded_at
as the 
timestamp_column
(how Elementary buckets data over time). - Use 
date
as the 
dimension
(business date whose distribution you care about). ### Example 1 – Detect spikes/drops per business date
Copy code
yaml version: 2  models:   - name: stock_snapshot     description: "Daily stock levels by item"     config:       elementary:         # Time axis for anomaly detection (how data is arriving to the table)         timestamp_column: loaded_at      tests:       - elementary.dimension_anomalies:           # Group by business date – monitor row counts per date over time           dimensions:             - date           # Optional filter – for example, only monitor recent dates           where_expression: "date >= current_date - interval '30' day"           # Optional explicit time bucket (default is 1 day)           time_bucket:             period: day             count: 1
What this does: - For each 
loaded_at
day (time bucket), it counts rows per 
date
. - Builds a baseline of “normal” row counts per 
date
value. - Fails if, for example, rows for 
2024-01-05
suddenly drop to near zero compared to historical behavior. [[Dimension anomalies](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/dimension-anomalies); [Dimensions param](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/dimensions)] ### Example 2 – Segment by a stock quantity bucket (dimension as SQL expression) You can turn 
stock_qty
into buckets and use that as a dimension:
Copy code
yaml models:   - name: stock_snapshot     config:       elementary:         timestamp_column: loaded_at      tests:       - elementary.dimension_anomalies:           dimensions:             # Bucket stock_qty as a dimension             - "case                  when stock_qty < 10 then 'low'                  when stock_qty between 10 and 100 then 'medium'                  else 'high'                end"           where_expression: "date >= current_date - interval '30' day"           time_bucket:             period: day             count: 1
Here Elementary will: - Group rows per bucket (
low
/ 
medium
/ 
high
) for each time bucket. - Detect if, for example, “low” stocks suddenly become much more (or less) frequent than usual. ### Key points to keep in mind - 
dimensions
is a list of columns or SQL expressions used in the 
GROUP BY
. [[Dimensions config](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/dimensions)] - Use low-cardinality dimensions (date, status buckets, qty buckets, etc.), not highly unique IDs. [[Dimension anomalies intro](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/dimension-anomalies)] - 
timestamp_column
controls how data is sliced into time buckets; if you omit it, Elementary monitors total counts per dimension over all time. [[Dim anomalies config](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/dimension-anomalies#test-configuration)] If you share what behavior you want to catch (e.g., “missing dates”, “sudden surge of low stock”), I can tailor the 
dimensions
and 
where_expression
more precisely.
o
how to check that for each id daily we have no anomalies with stock_qty column? @kapa.ai
k
A message from kapa.ai
You can do this with 
column_anomalies
+ 
dimensions
, using: - 
loaded_at
(or similar) as the 
timestamp_column
– defines the time axis. - 
id
as a dimension – so each id is monitored separately. - Optional 
date
as another dimension – so you segment by both id and business date. 
dimensions
tells Elementary to group metrics by those fields and detect anomalies per group. [[Dimensions param](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/dimensions); [Column anomalies config](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/column-anomalies#test-configuration); [Changelog – column anomalies by dimensions](https://docs.elementary-data.com/changelog#column-anomalies-by-dimensions)] ### Example: per-id daily anomalies on 
stock_qty
Copy code
yaml version: 2  models:   - name: stock_snapshot     description: "Daily stock quantity per id"     config:       elementary:         # Time axis for anomaly detection (usually load / partition time)         timestamp_column: loaded_at      columns:       - name: stock_qty         tests:           - elementary.column_anomalies:               # Let Elementary run default numeric metrics (min, max, avg, etc.)               # or specify explicitly, e.g. ['average', 'zero_percent']               # column_anomalies: ['average']                # Monitor per id per date               dimensions:                 - id                 - date                # Optional time bucketing – defaults to 1 day if omitted               time_bucket:                 period: day                 count: 1
What this does: - For every day (by 
loaded_at
), Elementary calculates 
stock_qty
metrics for each 
(id, date)
pair. [[Column anomalies](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/column-anomalies); [Dimensions param](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/dimensions)] - It builds a historical baseline per 
(id, date)
and flags anomalies when a metric (e.g. average 
stock_qty
) for a specific id/date deviates significantly from its past behavior. If 
id
has very high cardinality, this can be heavy; the docs recommend using lower-cardinality dimensions where possible. [[Dimensions param](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/dimensions)]
31 Views
Open in Slack
PreviousNext
Powered by