here It was running all days but today it failed ``` 2023 1 Elementary Community #community-support

@here It was running all days but today it failed ...

Ram

10/27/2023, 2:18 AM

@here It was running all days but today it failed

Copy code

[2023-10-26, 22:20:06 EDT] {subprocess.py:74} INFO - Running command: ['bash', '-c', 'edr send-report --s3-bucket-name xxxx --bucket-file-path /edr/index.html --project-dir /opt/airflow/dbt/abc_metrics --profiles-dir /opt/airflow/dbt-dir/ -c /mnt/working-dir -st *** -ch dbt_alerts']
[2023-10-26, 22:20:06 EDT] {subprocess.py:85} INFO - Output:
[2023-10-26, 22:20:09 EDT] {subprocess.py:92} INFO -     ________                          __
[2023-10-26, 22:20:09 EDT] {subprocess.py:92} INFO -    / ____/ /__  ____ ___  ___  ____  / /_____ ________  __
[2023-10-26, 22:20:09 EDT] {subprocess.py:92} INFO -   / __/ / / _ \/ __ `__ \/ _ \/ __ \/ __/ __ `/ ___/ / / /
[2023-10-26, 22:20:09 EDT] {subprocess.py:92} INFO -  / /___/ /  __/ / / / / /  __/ / / / /_/ /_/ / /  / /_/ /
[2023-10-26, 22:20:09 EDT] {subprocess.py:92} INFO - /_____/_/\___/_/ /_/ /_/\___/_/ /_/\__/\__,_/_/   \__, /
[2023-10-26, 22:20:09 EDT] {subprocess.py:92} INFO -                                                  /____/
[2023-10-26, 22:20:09 EDT] {subprocess.py:92} INFO - 
[2023-10-26, 22:20:09 EDT] {subprocess.py:92} INFO - Any feedback and suggestions are welcomed! join our community here - <https://bit.ly/slack-elementary>
[2023-10-26, 22:20:09 EDT] {subprocess.py:92} INFO - 
[2023-10-26, 22:13:27 EDT] {subprocess.py:92} INFO - 2023-10-27 02:13:27 — INFO — Running with edr=0.12.0
[2023-10-26, 22:13:27 EDT] {subprocess.py:92} INFO - 2023-10-27 02:13:27 — INFO — Running dbt --log-format json deps --project-dir /mnt/working-dir/edr/elementary/monitor/dbt_project --profiles-dir /opt/airflow/dbt-dir/
[2023-10-26, 22:13:33 EDT] {subprocess.py:92} INFO - 2023-10-27 02:13:33 — INFO — Running with dbt=1.6.6
[2023-10-26, 22:13:33 EDT] {subprocess.py:92} INFO - 2023-10-27 02:13:33 — INFO — Installing dbt-labs/dbt_utils
[2023-10-26, 22:13:33 EDT] {subprocess.py:92} INFO - 2023-10-27 02:13:33 — INFO — Installed from version 0.8.6
[2023-10-26, 22:13:33 EDT] {subprocess.py:92} INFO - 2023-10-27 02:13:33 — INFO — Updated version available: 1.1.1
[2023-10-26, 22:13:33 EDT] {subprocess.py:92} INFO - 2023-10-27 02:13:33 — INFO — Installing elementary-data/elementary
[2023-10-26, 22:13:33 EDT] {subprocess.py:92} INFO - 2023-10-27 02:13:33 — INFO — Installed from version 0.12.0
[2023-10-26, 22:13:33 EDT] {subprocess.py:92} INFO - 2023-10-27 02:13:33 — INFO — Up to date!
[2023-10-26, 22:13:33 EDT] {subprocess.py:92} INFO - 2023-10-27 02:13:33 — INFO —
[2023-10-26, 22:13:33 EDT] {subprocess.py:92} INFO - 2023-10-27 02:13:33 — INFO — Updates available for packages: ['dbt-labs/dbt_utils']
[2023-10-26, 22:13:33 EDT] {subprocess.py:92} INFO - Update your versions in packages.yml, then run dbt deps
[2023-10-26, 22:14:01 EDT] {subprocess.py:92} INFO - [33;20m2023-10-27 02:14:01 — WARNING — You are using incompatible versions between edr (0.12.0) and Elementary's dbt package (0.9.4).
[2023-10-26, 22:14:01 EDT] {subprocess.py:92} INFO -  To fix please update your packages.yml, and run:
[2023-10-26, 22:14:01 EDT] {subprocess.py:92} INFO - dbt deps && dbt run --select elementary
[2023-10-26, 22:14:01 EDT] {subprocess.py:92} INFO - [0m
[2023-10-26, 22:14:09 EDT] {subprocess.py:92} INFO - 2023-10-27 02:14:09 — INFO — Elementary's database and schema: '"PR_METRICS_elementary"'
[2023-10-26, 22:14:10 EDT] {subprocess.py:92} INFO - 2023-10-27 02:14:10 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_test_results", "macro_args": {"days_back": 7, "invocations_per_test": 720, "disable_passed_test_metrics": false}} --project-dir /mnt/working-dir/edr/elementary/monitor/dbt_project --profiles-dir /opt/airflow/dbt-dir/
[2023-10-26, 22:14:26 EDT] {subprocess.py:92} INFO - [31;20m2023-10-27 02:14:26 — ERROR — Could not generate the report - Error: Failed to run dbt command.
[2023-10-26, 22:14:26 EDT] {subprocess.py:92} INFO - Please reach out to our community for help with this issue.[0m
[2023-10-26, 22:14:26 EDT] {subprocess.py:92} INFO - NoneType: None
[2023-10-26, 22:14:26 EDT] {subprocess.py:96} INFO - Command exited with return code 1
[2023-10-26, 22:14:27 EDT] {taskinstance.py:1909} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/mnt/working-dir/avalara_ds/airflow/operators/dbt_bash.py", line 97, in execute
    super().execute(context)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/operators/bash.py", line 194, in execute
    raise AirflowException(
airflow.exceptions.AirflowException: Bash command failed. The command returned a non-zero exit code 1.

👀 1

Anna Vasileva

10/27/2023, 8:41 AM

The same for me. Version 0.12.0

Or Avidov

10/27/2023, 9:40 AM

Hi @Ram and @Anna Vasileva! according to this log it seems like you are using incompatible versions between elementary CLI and dbt package -

Copy code

WARNING — You are using incompatible versions between edr (0.12.0) and Elementary's dbt package (0.9.4).

Or Avidov

10/27/2023, 9:41 AM

Could you please upgrade to the latest dbt package by editing your packages.yml file and then run

dbt run -s elementary

Ram

10/27/2023, 2:55 PM

@here Even after upgrade same error

Copy code

[2023-10-27, 10:46:49 EDT] {subprocess.py:92} INFO - 
[2023-10-27, 10:46:49 EDT] {subprocess.py:92} INFO - 2023-10-27 14:46:49 — INFO — Running with edr=0.12.0
[2023-10-27, 10:46:49 EDT] {subprocess.py:92} INFO - 2023-10-27 14:46:49 — INFO — Running dbt --log-format json deps --project-dir /mnt/working-dir/edr/elementary/monitor/dbt_project --profiles-dir /opt/airflow/dbt-dir/
[2023-10-27, 10:46:55 EDT] {subprocess.py:92} INFO - 2023-10-27 14:46:55 — INFO — Running with dbt=1.6.6
[2023-10-27, 10:46:55 EDT] {subprocess.py:92} INFO - 2023-10-27 14:46:55 — INFO — Installing dbt-labs/dbt_utils
[2023-10-27, 10:46:55 EDT] {subprocess.py:92} INFO - 2023-10-27 14:46:55 — INFO — Installed from version 0.8.6
[2023-10-27, 10:46:55 EDT] {subprocess.py:92} INFO - 2023-10-27 14:46:55 — INFO — Updated version available: 1.1.1
[2023-10-27, 10:46:55 EDT] {subprocess.py:92} INFO - 2023-10-27 14:46:55 — INFO — Installing elementary-data/elementary
[2023-10-27, 10:46:55 EDT] {subprocess.py:92} INFO - 2023-10-27 14:46:55 — INFO — Installed from version 0.12.0
[2023-10-27, 10:46:55 EDT] {subprocess.py:92} INFO - 2023-10-27 14:46:55 — INFO — Up to date!
[2023-10-27, 10:46:55 EDT] {subprocess.py:92} INFO - 2023-10-27 14:46:55 — INFO —
[2023-10-27, 10:46:55 EDT] {subprocess.py:92} INFO - 2023-10-27 14:46:55 — INFO — Updates available for packages: ['dbt-labs/dbt_utils']
[2023-10-27, 10:46:55 EDT] {subprocess.py:92} INFO - Update your versions in packages.yml, then run dbt deps
[2023-10-27, 10:47:23 EDT] {subprocess.py:92} INFO - [33;20m2023-10-27 14:47:23 — WARNING — You are using incompatible versions between edr (0.12.0) and Elementary's dbt package (0.9.4).
[2023-10-27, 10:47:23 EDT] {subprocess.py:92} INFO -  To fix please update your packages.yml, and run:
[2023-10-27, 10:47:23 EDT] {subprocess.py:92} INFO - dbt deps && dbt run --select elementary
[2023-10-27, 10:47:23 EDT] {subprocess.py:92} INFO - [0m
[2023-10-27, 10:47:31 EDT] {subprocess.py:92} INFO - 2023-10-27 14:47:31 — INFO — Elementary's database and schema: '"DS_DEV_MART.PRODUCT_METRICS_elementary"'
[2023-10-27, 10:47:32 EDT] {subprocess.py:92} INFO - 2023-10-27 14:47:32 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_test_results", "macro_args": {"days_back": 7, "invocations_per_test": 720, "disable_passed_test_metrics": false}} --project-dir /mnt/working-dir/edr/elementary/monitor/dbt_project --profiles-dir /opt/airflow/dbt-dir/
[2023-10-27, 10:47:48 EDT] {subprocess.py:92} INFO - [31;20m2023-10-27 14:47:48 — ERROR — Could not generate the report - Error: Failed to run dbt command.

Ram

10/27/2023, 3:01 PM

packages.yml

Copy code

packages:
    ## Docs: <https://docs.elementary-data.com>
    ## Monitor your data quality, operation and performance directly from your dbt project
  - package: elementary-data/elementary
    version: 0.12.0
    ## contains macros that can be (re)used across dbt projects
  - package: dbt-labs/dbt_utils
    version: 1.1.1
    ## helps assert expectations for different dataset characteristics and column values
  - package: calogica/dbt_expectations
    version: 0.10.0
    ## Macros that generate dbt code, and log it to the command line
  - package: dbt-labs/codegen
    version: 0.11.0

Or Avidov

10/27/2023, 4:10 PM

@Ram it seems like the old package is still the issue based on the log you shared above.. Can you please make sure to run the following commands inside your dbt project folder in the following order - •

dbt deps

•

dbt run -s elementary

Afterwards try running

edr report

again?

Ram

10/27/2023, 4:26 PM

Ok i ran •

dbt deps

•

dbt run -s elementary

Ram

10/27/2023, 4:27 PM

then i ran dbt test , its failing first time , it was running fine till before running above commands All the column anomalies failed why ? Example as below

Copy code

16:17:15  Failure in test elementary_column_anomalies_metric_setup_1__sum__metric_value__365__month__1__product_name_Calculations_AvaTax_ (models/events/metric_setup.yml)
16:17:15    Got 1 result, configured to fail if != 0

Ram

10/27/2023, 4:29 PM

Copy code

models:
  - name:  metric_setup
    config:
      elementary:
        timestamp_column: 'metric_date'
    columns:
      - name: metric_value
        tests:
          - elementary.column_anomalies:
              column_anomalies:
                - sum
              time_bucket:
                period: month
                count: 1
              days_back: 365
              config:
                # optional - change severity
                severity: ERROR
              anomaly_sensitivity: 1
              where_expression: "product_name = 'Returns' "

Ram

10/27/2023, 4:30 PM

What am i missing here

Ram

10/27/2023, 4:34 PM

Can some one help me out , its failing in out production

Or Avidov

10/27/2023, 4:36 PM

Maybe try generating a report to see if there's a good reason why the test is failing?

Ram

10/27/2023, 4:46 PM

We always run the steps as below dbt test --> dbt monitor --> dbt send-report

Ram

10/27/2023, 4:46 PM

Right now i am running dbt run before i run dbt test

👍 1

Or Avidov

10/27/2023, 4:54 PM

it's just that in the report will be the easiest to see the metric that this test measures over time and if the failure makes sense or not

Ram

10/27/2023, 5:23 PM

Ram

10/27/2023, 5:58 PM

should i run command

Ram

10/27/2023, 5:58 PM

edr report

Ram

10/27/2023, 6:04 PM

Ram

10/27/2023, 6:05 PM

image.png

Ram

10/27/2023, 6:05 PM

this issue is happening only after i ran

dbt deps && run -s elementary

Ram

10/27/2023, 6:06 PM

But was working fine before until yesterday

Ram

10/27/2023, 6:08 PM

but some other metris is passed with same message

Ram

10/27/2023, 6:39 PM

ALso i ran edr command in bash operator in airflow still same error even though packages are updated

Copy code

[2023-10-27, 14:35:58 EDT] {subprocess.py:92} INFO -   / __/ / / _ \/ __ `__ \/ _ \/ __ \/ __/ __ `/ ___/ / / /
[2023-10-27, 14:35:58 EDT] {subprocess.py:92} INFO -  / /___/ /  __/ / / / / /  __/ / / / /_/ /_/ / /  / /_/ /
[2023-10-27, 14:35:58 EDT] {subprocess.py:92} INFO - /_____/_/\___/_/ /_/ /_/\___/_/ /_/\__/\__,_/_/   \__, /
[2023-10-27, 14:35:58 EDT] {subprocess.py:92} INFO -                                                  /____/
[2023-10-27, 14:35:58 EDT] {subprocess.py:92} INFO - 
[2023-10-27, 14:35:58 EDT] {subprocess.py:92} INFO - Any feedback and suggestions are welcomed! join our community here - <https://bit.ly/slack-elementary>
[2023-10-27, 14:35:58 EDT] {subprocess.py:92} INFO - 
[2023-10-27, 14:35:58 EDT] {subprocess.py:92} INFO - 2023-10-27 18:35:58 — INFO — Running with edr=0.12.0
[2023-10-27, 14:35:58 EDT] {subprocess.py:92} INFO - 2023-10-27 18:35:58 — INFO — Running dbt --log-format json deps --project-dir /mnt/working-dir/edr/elementary/monitor/dbt_project --profiles-dir /opt/airflow/dbt-dir/
[2023-10-27, 14:36:04 EDT] {subprocess.py:92} INFO - 2023-10-27 18:36:04 — INFO — Running with dbt=1.6.6
[2023-10-27, 14:36:04 EDT] {subprocess.py:92} INFO - 2023-10-27 18:36:04 — INFO — Installing dbt-labs/dbt_utils
[2023-10-27, 14:36:04 EDT] {subprocess.py:92} INFO - 2023-10-27 18:36:04 — INFO — Installed from version 0.8.6
[2023-10-27, 14:36:04 EDT] {subprocess.py:92} INFO - 2023-10-27 18:36:04 — INFO — Updated version available: 1.1.1
[2023-10-27, 14:36:04 EDT] {subprocess.py:92} INFO - 2023-10-27 18:36:04 — INFO — Installing elementary-data/elementary
[2023-10-27, 14:36:04 EDT] {subprocess.py:92} INFO - 2023-10-27 18:36:04 — INFO — Installed from version 0.12.0
[2023-10-27, 14:36:04 EDT] {subprocess.py:92} INFO - 2023-10-27 18:36:04 — INFO — Up to date!
[2023-10-27, 14:36:04 EDT] {subprocess.py:92} INFO - 2023-10-27 18:36:04 — INFO —
[2023-10-27, 14:36:04 EDT] {subprocess.py:92} INFO - 2023-10-27 18:36:04 — INFO — Updates available for packages: ['dbt-labs/dbt_utils']
[2023-10-27, 14:36:04 EDT] {subprocess.py:92} INFO - Update your versions in packages.yml, then run dbt deps
[2023-10-27, 14:36:32 EDT] {subprocess.py:92} INFO - 2023-10-27 18:36:32 — INFO — edr (0.12.0) and Elementary's dbt package (0.12.0) are compatible.
[2023-10-27, 14:36:40 EDT] {subprocess.py:92} INFO - 2023-10-27 18:36:40 — INFO — Elementary's database and schema: '"DS_DEV_MART.PRODUCT_METRICS_elementary"'
[2023-10-27, 14:36:40 EDT] {subprocess.py:92} INFO - 2023-10-27 18:36:40 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_test_results", "macro_args": {"days_back": 7, "invocations_per_test": 720, "disable_passed_test_metrics": false}} --project-dir /mnt/working-dir/edr/elementary/monitor/dbt_project --profiles-dir /opt/airflow/dbt-dir/
[2023-10-27, 14:36:57 EDT] {subprocess.py:92} INFO - [31;20m2023-10-27 18:36:57 — ERROR — Could not generate the report - Error: Failed to run dbt command.
[2023-10-27, 14:36:57 EDT] {subprocess.py:92} INFO - Please reach out to our community for help with this issue.[0m
[2023-10-27, 14:36:57 EDT] {subprocess.py:92} INFO - NoneType: None
[2023-10-27, 14:36:57 EDT] {subprocess.py:96} INFO - Command exited with return code 1
[2023-10-27, 14:36:57 EDT] {taskinstance.py:1909} ERROR - Task failed with exception

Or Avidov

10/28/2023, 8:12 AM

Now it seems to be a different error, if the command failed how did you get the report? Also - can you please share your edr.log and the report with me in private? this will help me debug the issue further

Ram

10/28/2023, 12:17 PM

The report I got was when I ran in my local But above error is when ran in bash operator in airflow

Or Avidov

10/29/2023, 8:33 AM

@Ram can you please send me in private the

edr_target

folder? you can probably zip it and send it over so I can further debug this?

Ram

10/30/2023, 4:37 PM

I am not sure how to get edr_target , coz we run in pod and it gets deleted after run or so .. Can we get from tables ?

Ram

10/30/2023, 5:43 PM

@Or Avidov What does belwo command do ? Does it truncate and overide the table

Copy code

dbt run --select elementary

Ram

10/30/2023, 5:47 PM

I have DM you the message with screenshot

Ram

10/30/2023, 8:33 PM

@Or Avidov Sent the log to your DM

48 Views

Open in Slack

Previous Next