Hi all, We’re running into noise issues with test...
# community-support
g
Hi all, We’re running into noise issues with test failure alerts and want to see if there’s a better way to handle this with Elementary. We use tests heavily, and Elementary’s Slack alerts are super helpful, but alerts often repeat for the same problematic rows even after stakeholders have been notified. Ideally, we’d like to “mark” these IDs so they aren’t alerted again unless new failures appear. Currently, we: • Use a date column to limit alerts to rows created in the last 24 hours • Add a 24-hour suppression interval But this isn’t ideal, as we risk missing new failing rows for the same test within that window. It would be cleaner if we could track alerted IDs instead of fine-tuning lookback windows and suppression intervals. Is there a straightforward way to achieve this with Elementary OSS? If not, any other ideas or best practices would be appreciated. Thanks in advance!
👀 1
g
Hi @George Tzanakis, There’s an option for this in the CLI. You can add the following to your
edr monitor
command:
edr monitor --suppression-interval {HOURS}
You’ll find an explanation of this flag in this section of the docs Let me know if anything is unclear!
g
Hey Guy, thanks for your reply. However, as I was saying in my message I'm using this already, I also explain why this is not a perfect solution.
g
Sorry for missing that part of your message! What do you think about limiting the rows by filtering on those created since the last test execution, instead of using a fixed 24-hour window? You could also leverage the alerts table managed by Elementary. This table contains information about sent alerts, including the timestamp of the last alert per test.
g
Hi Guy and thank you for the feedback.
What do you think about limiting the rows by filtering on those created since the last test execution, instead of using a fixed 24-hour window?
For us (and I believe for others, too), that would be a massive improvement towards reducing the noise in our channel and making our alerting more lean and meaningful. I would imagine it as a new flag in the
edr monitor
command, something like --ignore-alerts-since-last-execution, or something with a better name :). Do you think you could incorporate it into edr? That would be amazing..
You could also leverage the alerts table managed by Elementary. This table contains information about sent alerts, including the timestamp of the last alert per test.
Hm, I see. This however sounds like we would have to create some sort of join with this table, or, in any case, sounds like a custom workaround possibly involving custom tests, right? Ideally for us it would be something as streamlined as possible; just use regular dbt/dbt_utils/dbt_expectations tests without having to worry about workarounds and custom solutions..
g
Hi George, You're right. it would definitely be helpful if there were no need to implement custom tests and everything worked seamlessly. That way, you'd be able to use standard tests without processing the same rows twice. I'll discuss this feature with my team and let you know what we think!
🦾 1
🙏 1
g
Thanks a lot for considering this, Guy, hope you find the idea valuable