https://linen.dev logo
Join Slack
Powered by
Hi all! I'm trying to use elementary report as an ...
# community-support
r
Hi all! I'm trying to use elementary report as an easy and convenient way to get insights into failed data quality tests. As I've been facing report charts inconsistencies while using elementary version 0.9.0 our team decided to switch to the new one 0.12.0. We use Snowflake as data WH. Well, first of all I droped elementary schema with all the underlying tables and views and following quickstart guide created them anew using 
dbt run --select elementary
command. At this point I executed 
dbt test
and got an information about 1125 running tests, 3 out of them were failed and 1122 passed. Afterwards I've created a report website using 
edr report
with elementary client being of the latest 0.12.0 version. The result seems strange to me, there is no mentioning of failed tests as well as general amount of unique test is 471. Meanwhile in DBT_TESTS elementary schema table information about all the 1125 dbt tests is stored. Could someone help me to resolve this confusion please? Is it the 
edr report
command building website poorly or did I miss something? Thanks is advance!
r
I am sure there is a bug in Elementary , after running column anomalies , all metrics and send-report are failing
r
That is useful to know, thanks a lot for sharing, will removing column anomaly tests help to get rig of inconsistencies? And maybe you know if there are other kinds of tests causing troubles (e.g volume or schema change test)?
m
Hey, we had a bug and fix it in 0.12.1. It's not a bug in any of the tests, it was in logging the results.
@Rostyslav Ilnytskyi can you please report if an upgrade solved it?
r
I have updated in the packages.yml as below
Copy code
packages:
    ## Docs: <https://docs.elementary-data.com>
    ## Monitor your data quality, operation and performance directly from your dbt project
  - package: elementary-data/elementary
    version: 0.12.1
But when running the pip install elementary , it installs 0.12
Copy code
[2023-11-02, 21:27:27 EDT] {subprocess.py:74} INFO - Running command: ['bash', '-c', 'edr send-report --s3-bucket-name abcd --bucket-file-path nginx/edr/index.html --project-dir /opt/airflow/dbt-dir/dbt/ --profiles-dir /opt/airflow/dbt-dir/ -c /mnt/dir -st *** -ch alerts']
[2023-11-02, 21:27:27 EDT] {subprocess.py:85} INFO - Output:
[2023-11-02, 21:27:30 EDT] {subprocess.py:92} INFO -     ________                          __
[2023-11-02, 21:27:30 EDT] {subprocess.py:92} INFO -    / ____/ /__  ____ ___  ___  ____  / /_____ ________  __
[2023-11-02, 21:27:30 EDT] {subprocess.py:92} INFO -   / __/ / / _ \/ __ `__ \/ _ \/ __ \/ __/ __ `/ ___/ / / /
[2023-11-02, 21:27:30 EDT] {subprocess.py:92} INFO -  / /___/ /  __/ / / / / /  __/ / / / /_/ /_/ / /  / /_/ /
[2023-11-02, 21:27:30 EDT] {subprocess.py:92} INFO - /_____/_/\___/_/ /_/ /_/\___/_/ /_/\__/\__,_/_/   \__, /
[2023-11-02, 21:27:30 EDT] {subprocess.py:92} INFO -                                                  /____/
[2023-11-02, 21:27:30 EDT] {subprocess.py:92} INFO - 
[2023-11-02, 21:27:30 EDT] {subprocess.py:92} INFO - Any feedback and suggestions are welcomed! join our community here - <https://bit.ly/slack-elementary>
[2023-11-02, 21:27:30 EDT] {subprocess.py:92} INFO - 
[2023-11-02, 21:27:30 EDT] {subprocess.py:92} INFO - 2023-11-03 01:27:30 — INFO — Running with edr=0.12.0
[2023-11-02, 21:27:30 EDT] {subprocess.py:92} INFO - 2023-11-03 01:27:30 — INFO — Running dbt --log-format json deps --project-dir /mnt/working-dir/edr/elementary/monitor/dbt_project --profiles-dir /opt/airflow/dbt-dir/
[2023-11-02, 21:27:36 EDT] {subprocess.py:92} INFO - 2023-11-03 01:27:36 — INFO — Running with dbt=1.7.0
[2023-11-02, 21:27:36 EDT] {subprocess.py:92} INFO - 2023-11-03 01:27:36 — INFO — Updating lock file in file path: /mnt/working-dir/edr/elementary/monitor/dbt_project/package-lock.yml
[2023-11-02, 21:27:36 EDT] {subprocess.py:92} INFO - 2023-11-03 01:27:36 — INFO — Installing dbt-labs/dbt_utils
[2023-11-02, 21:27:36 EDT] {subprocess.py:92} INFO - 2023-11-03 01:27:36 — INFO — Installed from version 0.8.6
[2023-11-02, 21:27:36 EDT] {subprocess.py:92} INFO - 2023-11-03 01:27:36 — INFO — Updated version available: 1.1.1
[2023-11-02, 21:27:36 EDT] {subprocess.py:92} INFO - 2023-11-03 01:27:36 — INFO — Installing elementary-data/elementary
[2023-11-02, 21:27:36 EDT] {subprocess.py:92} INFO - 2023-11-03 01:27:36 — INFO — Installed from version 0.12.0
[2023-11-02, 21:27:36 EDT] {subprocess.py:92} INFO - 2023-11-03 01:27:36 — INFO — Updated version available: 0.12.1
[2023-11-02, 21:27:36 EDT] {subprocess.py:92} INFO - 2023-11-03 01:27:36 — INFO —
[2023-11-02, 21:27:36 EDT] {subprocess.py:92} INFO - 2023-11-03 01:27:36 — INFO — Updates available for packages: ['dbt-labs/dbt_utils', 'elementary-data/elementary']
[2023-11-02, 21:27:36 EDT] {subprocess.py:92} INFO - Update your versions in packages.yml, then run dbt deps
[2023-11-02, 21:28:05 EDT] {subprocess.py:92} INFO - 2023-11-03 01:28:05 — INFO — edr (0.12.0) and Elementary's dbt package (0.12.1) are compatible.
[2023-11-02, 21:28:14 EDT] {subprocess.py:92} INFO - 2023-11-03 01:28:14 — INFO — Elementary's database and schema: '"DS_DEV_MART.PRODUCT_METRICS_elementary"'
[2023-11-02, 21:28:14 EDT] {subprocess.py:92} INFO - 2023-11-03 01:28:14 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_test_results", "macro_args": {"days_back": 7, "invocations_per_test": 720, "disable_passed_test_metrics": false}} --project-dir /mnt/working-dir/edr/elementary/monitor/dbt_project --profiles-dir /opt/airflow/dbt-dir/
[2023-11-02, 21:28:32 EDT] {subprocess.py:92} INFO - [31;20m2023-11-03 01:28:32 — ERROR — Could not generate the report - Error: Failed to run dbt command.
[2023-11-02, 21:28:32 EDT] {subprocess.py:92} INFO - Please reach out to our community for help with this issue.[0m
[2023-11-02, 21:28:32 EDT] {subprocess.py:92} INFO - NoneType: None
[2023-11-02, 21:28:33 EDT] {subprocess.py:96} INFO - Command exited with return code 1
m
@Ram pip is for the Python package. The fix is in thr dbt package. The latest Elementary versions are - Python: 0.12.0 dbt package: 0.12.1 The steps for upgrading the dbt package are: 1. Update packages.yml 2. Run 'dbt deps' 3. Run 'dbt run --select elementary'
r
Yes i ran all above steps
m
Ok @Ram , seems like your upgrade was successful. However edr is failing to generate a report. I know @Erik Zaadi was helping you with this but I don't know the details. It is unrelated to the issue @Rostyslav Ilnytskyi reported.
r
I believe this Run ‘dbt run --select elementary’ is one time not every run right
m
Yes
👍 1
r
Good news finally its working after i ran again 
'dbt run --select elementary'
Now only one issue pending to resolve i will tag you the other message i posted
🎉 1
r
Hello, @Maayan Salom I've updated dbt elementary package to 0.12.1v and now report seems much more trustworthy, thanks for suggestion. But through weekend I noticed new types of errors emerging, first one has this message and happens during 
dbt test
posthook
Copy code
Object '<DBNAME>.ELEMENTARY.FILTERED_INFORMATION_SCHEMA_COLUMNS' does not exist or not authorized.
Table FILTERED_INFORMATION_SCHEMA_COLUMNS really is not present in elementary schema but INFORMATION_SCHEMA_COLUMS do exists. Also 
edr monitor
command throws the next error:
Copy code
invalid identifier 'ALERTS_IN_TIME_LIMIT.FRESHNESS_ERROR_AFTER'
Maybe you know what should be done to resolve the issues?
n
Hey @Rostyslav Ilnytskyi It looks like you need to run 
dbt run -s elementary
. See upgrade instructions here.
r
@Noy Arie Hello! Yeah, seems like I've missed the point, after running this command errors are gone, thanks a lot 😊
n
Happy I could help 🙂
🙌 1
36 Views
Open in Slack
PreviousNext
Powered by