https://linen.dev logo
Join Slack
Powered by
<@U088WCP3B4P> I need to implement a anomaly detec...
# ask-elementary-ai
s
@kapa.ai I need to implement a anomaly detection to track the volume of information that recipe per day, in case that this the number drop raise a alert for that.
k
Thinking completed You can do this with the 
elementary.volume_anomalies
test, using daily time buckets and 
anomaly_direction: drop
so it only alerts when the row count falls. I'm assuming "recipe" is the name of your table or model, so replace it with your actual model name.
Copy code
models:   - name: recipe     config:       elementary:         timestamp_column: created_at   # replace with your timestamp column     data_tests:       - elementary.volume_anomalies:           arguments:             time_bucket:               period: day               count: 1             anomaly_direction: drop           config:             severity: warn   # optional
How it works: • The test counts rows per time bucket, daily by default. It compares each bucket in the detection period (last 2 days by default) against the buckets in the training period (14 days by default). If it finds an anomaly in the detection period, the test fails. [Volume anomalies] • The test only runs on completed time buckets. If you run it during the day, it evaluates yesterday as the last complete bucket. [Volume anomalies] • 
anomaly_direction: drop
flags only values below the expected range. It does not change how the range is calculated. [anomaly_direction] • A 
timestamp_column
is highly recommended. Without one, the test counts the table's total rows. [Test configuration] Optional: ignore small drops. To fail only on drops above a given percentage, add 
ignore_small_changes
. For example, with 
drop_failure_percent_threshold: 10
, the test fails only if the row count drops by more than 10% compared to the mean of the training period. [ignore_small_changes example]
Copy code
arguments:             anomaly_direction: drop             ignore_small_changes:               drop_failure_percent_threshold: 10
Optional: fixed percentage thresholds. If you already know the acceptable range, such as "alert if the table shrinks by more than 10 percent", the docs suggest 
volume_threshold
instead. It compares the newest completed bucket against the previous one using explicit percentages. [Volume anomalies] Getting the alert in Slack. Run 
edr monitor
after your dbt tests, with a Slack webhook or token configured. [Slack alerts]
Copy code
edr monitor --slack-webhook <your_slack_webhook_url>
👍 0 button 👎 0 button ℹ️ Tip: Mention @elementary.ai in the thread for follow-ups or DM the app for 1-to-1 help.
s
@kapa.ai I need to implement this for diferent tables with the same parameter, how can add more reusable for this test and also add a custom name for the test
k
Thinking completed Reusing the test across tables I couldn't find a way in the knowledge sources to define one anomaly test and apply it to several tables in open-source Elementary. Tests in dbt are attached to specific models or columns, so you need a definition per table. [Global tests not supported] Elementary Cloud can add anomaly monitors across your assets, but the open-source version cannot do this without extra configuration. [Cloud vs OSS] The workaround is to generate the YAML with a macro. A community member shared this macro, which loops over the models in a schema, optionally filtered by tags, and prints a schema YAML with the same Elementary tests for each model. I haven't tested it, and it isn't official. Here it is, trimmed to a 
volume_anomalies
test that alerts on drops:
Copy code
{% macro generate_schema_yml(schema_name, include_tags=[]) %}
  {% set models_in_schema = [] %}
  {% for model in graph.nodes.values() %}
    {% if schema_name in model.schema
       and model.resource_type == "model"
       and (include_tags | length == 0 or model.tags | select("in", include_tags) | list | length > 0) %}
      {% do models_in_schema.append(model.name) %}
    {% endif %}
  {% endfor %}

  {% set schema_yaml %}
version: 2

models:
{% for model in models_in_schema %}
  - name: {{ model }}
    tests:
      - elementary.volume_anomalies:
          anomaly_direction: drop
          time_bucket:
            period: day
            count: 1
          meta:
            description: "Daily volume drop check for {{ model }}"
          config:
            severity: warn
            elementary:
              timestamp_column: "your_timestamp_column"
{% endfor %}
  {% endset %}

  {{ print(schema_yaml) }}
{% endmacro %}
Generate the file with:
Copy code
dbt run-operation generate_schema_yml --args '{"schema_name": "bronze", "include_tags" :[" "]}' --quiet  | sed $'s/\x1b\[[0-9;]*m//g' > models/bronze/br_anomaly_tests.yml
[Schema macro] Other users generate the YAML with an external script instead. [Script workaround] Custom test name • You can set 
name:
on the test, and dbt will print it in the console. It is stored in the 
elementary.dbt_tests
table, but it was not shown in the Elementary report for 
volume_anomalies
. This was reported as a bug. [Naming tests] • Support staff also confirmed that overriding the displayed test name is not currently possible. [Overwrite name] • Workaround: set 
meta.description
. It appears in the report under each test result and in alerts. The downside is that it overrides the default description. [Test descriptions]
Copy code
- elementary.volume_anomalies:
    meta:
      description: "Your custom description here"
You can also use tags to group tests and make them easier to manage. [Test descriptions] 👍 0 button 👎 0 button ℹ️ Tip: Mention @elementary.ai in the thread for follow-ups or DM the app for 1-to-1 help.
s
@kapa.ai I got this error in local: Runtime Error Could not find profile named ’elementary
k
Thinking completed This error means 
edr
can't find a profile named 
elementary
in your 
profiles.yml
. It looks for that exact name and doesn't use your main dbt project profile. [Profile error thread] What to check 1. Add a profile named 
elementary
. The file needs a top-level 
elementary
key, and the name must match exactly, including case and indentation. [Missing profile] Elementary uses dbt's connection profile format, so dbt-core users can just add a profile with this name. [CLI install]elementary: outputs: default: type: [your warehouse type] # ... other connection details ... schema: [schema name] # elementary schema, usually [schema name]_elementary threads: [1 or more]The error message in your post shows a stray character in the name (
’elementary
). If you typed the profile name by hand, check that it has no curly quotes or extra characters. 2. Check the 
profiles.yml
location. By default, the file is looked up in 
~/.dbt/
. If it's elsewhere, pass the directory with 
--profiles-dir
. [Missing profile]edr report --profiles-dir /path/to/your/profiles 3. Pass a directory, not a file. 
--profiles-dir
expects the folder that contains 
profiles.yml
, not the path to the file itself. [Profiles dir thread] 4. Check the target. 
--profile-target
(or 
-t
) only selects a target inside the 
elementary
profile. The target must exist under 
outputs
. [Profile error thread] 5. Check file name and formatting. The file must be named exactly 
profiles.yml
, and its YAML indentation must be correct. [Missing profile] If it still fails, share your 
profiles.yml
with credentials redacted, plus the exact 
edr
command you ran, and I can help narrow it down. 👍 0 button 👎 0 button ℹ️ Tip: Mention @elementary.ai in the thread for follow-ups or DM the app for 1-to-1 help.
s
@kapa.ai Could not find adapter type snowflake!
k
Thinking completed The knowledge sources don't describe this exact Snowflake error, so I can't give a confirmed diagnosis. They do cover two likely causes: a missing adapter in the environment where 
edr
runs, and invalid profile credentials. 1. Install the Snowflake adapter in the same environment as edr The CLI install depends on your dbt version. [CLI install] dbt Core 1.x: install the CLI with the matching extra, which installs dbt Core and the warehouse adapter alongside it:
Copy code
pip install 'elementary-data[dbt1-snowflake]'
dbt v2 (Fusion): install the CLI without extras, then install dbt v2 itself by following dbt's installation guide. Adapters are built in, so no warehouse-specific package is needed.
Copy code
pip install elementary-data
Then run 
edr --help
to confirm the installation worked. 2. Check your virtual environment A community member hit an error when running 
edr report
and fixed it by installing everything inside an activated virtual environment in the dbt project directory. They installed 
dbt-core
, 
dbt-snowflake
, and 
elementary-data
there. They also noted that the virtual environment is isolated from global installations, so the packages must be installed in the active environment. [Venv fix]
Copy code
pip install dbt-core
pip install dbt-snowflake
pip install elementary-data
3. Check your elementary profile credentials In a similar Slack thread (a Postgres user got "Couldn't find adapter type postgres!"), support staff pointed to invalid credentials in the elementary profile and asked the user to check them. [Slack thread] Make sure your profile has 
type: snowflake
and valid details. The docs recommend keypair auth: [Snowflake profile]
Copy code
elementary:
  outputs:
    default:
      type: snowflake
      account: [account id]
      user: [username]
      role: [user role]
      private_key_path: [path/to/private.key]
      private_key_passphrase: [passphrase for the private key, if key is encrypted]
      database: [database name]
      warehouse: [warehouse name]
      schema: [schema name]_elementary
      threads: 4
If it still fails, share the full error output, your dbt version, and how you installed 
edr
, with credentials redacted. 👍 0 button 👎 0 button ℹ️ Tip: Mention @elementary.ai in the thread for follow-ups or DM the app for 1-to-1 help.
s
@kapa.ai this is my profile: default: target: dev # Default target to use outputs: dev: # Development environment type: snowflake # Connection identifiers (placeholder examples, see adapter-specific pages for supported configs) account: DWTBRWL-asasas database: nexus_dev schema: dbt_sramirez warehouse: WH_DBT_DEV authenticator: externalbrowser user: saul.castro@asasasas.com threads: 2 # Number of parallel threads elementary: outputs: default: type: snowflake account: DWTBRWL-KJasB2asas3as18 authenticator: externalbrowser user: saul.castro@asasasas.com warehouse: WH_DBT_QA database: nexus_qa schema: elementary threads: 2
k
Thinking completed Your profile has a few things to fix, based on the knowledge sources. 1. externalbrowser isn't supported by the CLI Your 
elementary
profile uses 
authenticator: externalbrowser
. Support threads say this method isn't supported by the Elementary CLI, because it needs interactive browser authentication. In one case it caused a login loop and repeated prompts, and the user fixed it by switching to key-pair login. [Login loop fix] [MFA thread] Another user hit a 
JSONDecodeError
when the CLI printed the browser login message while parsing dbt output. [Log parse error] The docs recommend key-pair auth for Snowflake: [Snowflake profile]
Copy code
elementary:
  outputs:
    default:
      type: snowflake
      account: [account id]
      user: [username]
      role: [user role]
      private_key_path: [path/to/private.key]
      private_key_passphrase: [passphrase for the private key, if key is encrypted]
      database: [database name]
      warehouse: [warehouse name]
      schema: [schema name]_elementary
      threads: 4
2. The schema name probably doesn't match Your 
elementary
profile uses 
schema: elementary
. Elementary tables are created by default in 
<schema_name>_elementary
, and the elementary profile must point to where they actually are. A community thread resolved a similar problem by adding the missing 
_elementary
suffix. [Schema suffix] Since your dbt schema is 
dbt_sramirez
, the Elementary schema would typically be 
dbt_sramirez_elementary
. Check where 
elementary_test_results
actually lives in your warehouse and use that schema. The same applies to the database, since your 
elementary
profile points to 
nexus_qa
while your dbt target uses 
nexus_dev
. 3. Other things to check • Account identifiers differ. Your two profiles use different 
account
values. This may be intentional, but verify it, since Elementary needs to connect to the warehouse where its tables live. • Profile name. The 
elementary
top-level key is correct. 
edr
looks for exactly that name. [Missing profile] • Adapter installation. The knowledge sources don't confirm the cause of your "Could not find adapter type snowflake!" error. Make sure 
dbt-snowflake
is installed in the same active environment as 
edr
, for example with 
pip install 'elementary-data[dbt1-snowflake]'
. [CLI install] Also, remove your email and account identifiers from anything you share publicly. 👍 0 button 👎 0 button ℹ️ Tip: Mention @elementary.ai in the thread for follow-ups or DM the app for 1-to-1 help.
s
@kapa.ai why pip3 with python 3.9 install elementary-data-0.20.1
k
Thinking completed The knowledge sources don't say why pip3 on Python 3.9 resolves to 
elementary-data
0.20.1, so I can't confirm the reason. • Python 3.9 is supported. Support staff say Elementary supports Python 3.9 to 3.12, in line with the versions dbt supports. [Slack thread] Python 3.8 support was dropped in 0.18.0. [Release 0.18.0] • 0.20.1 may be the dbt package version. The only 0.20.1 release note in the sources is for the dbt package, which fixed the dbt Fusion version check to require major version 2. [dbt package 0.20.1] The sources don't show whether that number also applies to the CLI package on PyPI. • Newer releases are possible. Python 3.13 support was merged and was due to be released soon. [Python 3.13 issue] The sources don't list which CLI versions came afterwards. Generally, pip installs the newest release whose declared Python requirement matches your interpreter. This is general pip behavior, not something the knowledge sources state. If a newer release required a higher Python version than 3.9, pip would fall back to an older one. I can't confirm that this is the case here. If you want a specific version, pin it, for example 
pip3 install elementary-data==<version>
. If that version isn't compatible, pip will report the Python requirement in its error. If you tell me which version you expected, I can look into it further. 👍 0 button 👎 0 button ℹ️ Tip: Mention @elementary.ai in the thread for follow-ups or DM the app for 1-to-1 help.
5 Views
Open in Slack
PreviousNext
Powered by