hey all, I am giving elementary a try. So far so g...
# community-supporta
Aditya Vishwakarma
10/14/2023, 4:42 AMhey all, I am giving elementary a try. So far so good. love the simplicity of it.
One issue we see is that most of our tests have severity warning, as we don't want to fail the pipeline. We would like more fine grained alerting levels. Like P0 - P5. So that we can route it to different slack channels. any ideas on how to achieve it?
a
Alex
10/15/2023, 8:51 PMHey @Aditya Vishwakarma
After implementing elementary in several places. I have to tell you: don't go this route.
Simplify. P0-P5 is way too complex.
Alex
10/15/2023, 8:52 PMThat said you can use tags and send alert to different channels with elementary.
https://docs.elementary-data.com/quickstart/send-slack-alerts
Alex
10/15/2023, 8:53 PMFor the severity there is only 2 options: warning and error. If you establish thresholds to warn if and error if only above certain number of failures.
a
Aditya Vishwakarma
10/16/2023, 1:27 AMI get where you are coming from. Basically we define our alerts at 3 levels:
1. ERROR - stop the pipeline. this maps well into DBT error
2. WARN - data issue, but pipeline working. this goes to analyst or owners to drill down and figure out what happened.
3. NOTICE - basically a test which is still being developed. will eventually result in a WARN test when SLA are more concrete
a
Alex Alves
10/16/2023, 7:17 AMSeverity is a dbt feature and there are only 2 options. In your case, warn and notice should be warn and you can use tags to differentiate them.
p
Parris Gjerde
02/16/2024, 5:13 PMAre there other alert channels, besides Slack and MS Teams, that have been developed? For instance, cloudwatch or splunk?
a
Alex Alves
02/16/2024, 5:16 PMPagerDuty and Opsgenie.
👍 1
Alex Alves
02/16/2024, 5:19 PMIf you are using the open source, with the same orchestrator you are using for edr, you could run a python script that sends the alerts to Splunk api for example from the data saved in elementary alerts table.
p
Parris Gjerde
02/16/2024, 5:21 PMYeah, I figured querying the table would be an option. Thanks.
Parris Gjerde
02/19/2024, 2:02 PMQuestion, are there limitations in using elementary per what's available on dbt hub?
Parris Gjerde
02/19/2024, 2:12 PMAlso, what are best practices as for managing the datasets themselves? Should the tables be truncated, for instance?
a
Alex Alves
02/19/2024, 5:02 PMI am not sure if I understand correctly your first question. There is no limitation as far as i know.
The tables created by elementary are managed by the dbt package, you don’t have to worry about them
p
Parris Gjerde
02/19/2024, 8:11 PMMy question is around how long the data lasts in the elementary tables. Elementary dumps a lot of data into the tables and they could get large. Do we need to manage their contents or does elementary provide mechanisms to manage the data sets? Thanks.
o
Ofek Weiss
03/20/2024, 9:07 AMHey @Parris Gjerde, currently there is no mechanism to manage the datasets, but thats an interesting idea, you are more than welcome to truncate them in your warehouse as you wish, you can also open a github issue if you would like and we can see how that can be integrated into elementary
31 Views