Ofek Weiss
07/20/2023, 2:10 PMdbt run -s elementary after dbt deps for this new minor version_ ❗
elementary What's New? elementary
• New Seasonality Options: hour_of_day and hour_of_week 🌞🌙
◦ Thanks to the brilliant contribution from @Moss Pauly (🤩), you now have two new seasonality options: hour_of_day and hour_of_week.
◦ These options will help in making the anomaly detection tests more accurate for datasets with strong seasonality impact.
◦ Check out the docs .
• Data for calculating Quality Score 📊
◦ We made the following changes to help calculating data sets quality score (refer to this great article for more info):
▪︎ We've added two new columns, elementary_test_resuts.failed_row_count and dbt_tests.quality_dimension. This data enables you to gain deeper insights into data quality and better track your data pipeline's health.
▪︎ We automatically fill these fields for common test (dbt, dbt_utils, and dbt_expectations), if you would like them to work for other test, or overwrite the default ones, you will need to fill the test's meta with:
• quality_dimension - Name of the quality dimension this test measures.
• failed_row_count_calc - An SQL expression to calculate the amount of failed rows from the test (count(*), sum(n_records), etc).
◦ Thank you @Saurabh Jha for working with us on this!
◦ This feature is still experimental, and is not reflected in the report nor the alerts.
• Alert Suppression in CLI 🚦
◦ Now, you can suppress alerts in the Command Line Interface (CLI) using the --suppression-interval flag.
◦ This feature gives you more control over your alerts and helps you manage notifications effectively.
◦ Check out the docs .
elementary Other Changes elementary
• Order Slack messages by test run time ⏰. This makes it easier to track test results and prioritize action items - Thanks @Willi Mueller for suggesting!
• Set SQL expression as timestamp_column - You can now set an SQL expression as the timestamp column, offering more flexibility in handling time-based data. Fantastic idea from @Sham Sreedharan, and solves use cases raised by @Shay Misgav, @Renee Cooper, @Steve Luk and @FABIEN RYCKOORT.
• Support for dbt where Parameter - Thanks @Taha Bel Khayate for surfacing.
elementary Bugfix elementary
• We've resolved a pesky bug that caused dbt_columns to malfunction in specific access configurations when using BigQuery.
• Table anomalies tests fix - Reported by @Scott Anderson !
• New dimension values will no linger cause dimension anomalies test to fail - Thanks @Jason Deng for reporting!
As always - We are looking forward to talk to you here on Slack (#C02CTC89LAX, #C03LPJ5EYU9), and happy to schedule calls!