https://linen.dev logo
Join Slack
Powered by
Hi! I recently upgraded our elementary (the dbt pa...
# community-support
c
Hi! I recently upgraded our elementary (the dbt package and the command line) 0.10.3 -> 0.14.1 and 0.10.0 -> 0.14.1. I ran 
dbt run -s elementary
and tested things locally, and all ran fine. But on our MWAA environment I'm getting errors when running 
edr report
, they are intermittent but almost every time, and there are a few different ones. I will put the errors in thread, would appreciate any help! - EDIT: ran 
dbt run -s elementary
again and the first run was successful. I'm keeping my 🤞 that it's fixed it, will let you know
OK, so on further investigation, we were actually having errors before the upgrade, so not necessarily upgrade related. Example errors:
Copy code
[2024-03-22 07:12:42,351] 2024-03-22 07:12:42 — INFO — Running with edr=0.14.1
[2024-03-22 07:12:55,267] 2024-03-22 07:12:55 — INFO — edr (0.14.1) and Elementary's dbt package (0.14.1) are compatible.
[2024-03-22 07:12:58,941] 2024-03-22 07:12:58 — INFO — Elementary's database and schema: '"DEV_CASPIAN.elementary"'
[2024-03-22 07:12:58,942] 2024-03-22 07:12:58 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_test_results", "macro_args": {"days_back": 7, "invocations_per_test": 720, "disable_passed_test_metrics": false}} --project-dir /usr/local/airflow/.local/lib/python3.11/site-packages/elementary/monitor/dbt_project --profiles-dir /usr/local/airflow/dags/dbt/profiles --target snowflake-mwaa
[2024-03-22 07:14:48,008] 2024-03-22 07:14:48 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_source_freshness_results", "macro_args": {"days_back": 7, "invocations_per_test": 720}} --project-dir /usr/local/airflow/.local/lib/python3.11/site-packages/elementary/monitor/dbt_project --profiles-dir /usr/local/airflow/dags/dbt/profiles --target snowflake-mwaa
[2024-03-22 07:14:58,777] 2024-03-22 07:14:58 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_models", "macro_args": {"exclude_elementary": true}} --project-dir /usr/local/airflow/.local/lib/python3.11/site-packages/elementary/monitor/dbt_project --profiles-dir /usr/local/airflow/dags/dbt/profiles --target snowflake-mwaa
[2024-03-22 07:15:03,704] 2024-03-22 07:15:03 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_sources", "macro_args": {}} --project-dir /usr/local/airflow/.local/lib/python3.11/site-packages/elementary/monitor/dbt_project --profiles-dir /usr/local/airflow/dags/dbt/profiles --target snowflake-mwaa
[2024-03-22 07:15:08,046] 2024-03-22 07:15:08 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_exposures", "macro_args": {}} --project-dir /usr/local/airflow/.local/lib/python3.11/site-packages/elementary/monitor/dbt_project --profiles-dir /usr/local/airflow/dags/dbt/profiles --target snowflake-mwaa
[2024-03-22 07:15:12,612] 2024-03-22 07:15:12 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_models_runs", "macro_args": {"days_back": 7, "exclude_elementary": true}} --project-dir /usr/local/airflow/.local/lib/python3.11/site-packages/elementary/monitor/dbt_project --profiles-dir /usr/local/airflow/dags/dbt/profiles --target snowflake-mwaa
[2024-03-22 07:15:56,806] 2024-03-22 07:15:56 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_dbt_models_test_coverage", "macro_args": {}} --project-dir /usr/local/airflow/.local/lib/python3.11/site-packages/elementary/monitor/dbt_project --profiles-dir /usr/local/airflow/dags/dbt/profiles --target snowflake-mwaa
[2024-03-22 07:16:10,384] 2024-03-22 07:16:10 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_nodes_depends_on_nodes", "macro_args": {"exclude_elementary": true}} --project-dir /usr/local/airflow/.local/lib/python3.11/site-packages/elementary/monitor/dbt_project --profiles-dir /usr/local/airflow/dags/dbt/profiles --target snowflake-mwaa
[2024-03-22 07:16:19,040] 2024-03-22 07:16:19 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_models_latest_invocation", "macro_args": {}} --project-dir /usr/local/airflow/.local/lib/python3.11/site-packages/elementary/monitor/dbt_project --profiles-dir /usr/local/airflow/dags/dbt/profiles --target snowflake-mwaa
[2024-03-22 07:16:27,378] 2024-03-22 07:16:27 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_models_latest_invocations_data", "macro_args": {}} --project-dir /usr/local/airflow/.local/lib/python3.11/site-packages/elementary/monitor/dbt_project --profiles-dir /usr/local/airflow/dags/dbt/profiles --target snowflake-mwaa
[2024-03-22 07:16:37,833] 2024-03-22 07:16:37 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_test_results", "macro_args": {"days_back": 7, "invocations_per_test": 720, "disable_passed_test_metrics": false}} --project-dir /usr/local/airflow/.local/lib/python3.11/site-packages/elementary/monitor/dbt_project --profiles-dir /usr/local/airflow/dags/dbt/profiles --target snowflake-mwaa
[2024-03-22 07:18:22,433] Command exited with return code -9
[2024-03-22 07:18:22,458] Task failed with exception
Traceback (most recent call last):
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/operators/bash.py", line 210, in execute
    raise AirflowException(
airflow.exceptions.AirflowException: Bash command failed. The command returned a non-zero exit code -9.
[2024-03-22 07:18:22,465] Marking task as FAILED. dag_id=financials_segment_data_pipeline, task_id=elementary.elementary_report_task, execution_date=20240321T183000, start_date=20240322T071240, end_date=20240322T071822
[2024-03-22 07:18:22,491] Failed to execute job 385116 for task elementary.elementary_report_task (Bash command failed. The command returned a non-zero exit code -9.; 29875)
And:
Copy code
[2024-03-21 00:02:56,111] 2024-03-21 00:02:56 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_test_results", "macro_args": {"days_back": 7, "invocations_per_test": 720, "disable_passed_test_metrics": false}} --project-dir /usr/local/airflow/.local/lib/python3.11/site-packages/elementary/monitor/dbt_project --profiles-dir /usr/local/airflow/dags/dbt/profiles --target snowflake-mwaa
[2024-03-21 00:05:30,198] Traceback (most recent call last):
[2024-03-21 00:05:30,198]   File "/usr/local/airflow/.local/lib/python3.11/site-packages/elementary/clients/dbt/dbt_runner.py", line 88, in _run_command
[2024-03-21 00:05:30,202]     result = subprocess.run(
[2024-03-21 00:05:30,202]              ^^^^^^^^^^^^^^^
[2024-03-21 00:05:30,202]   File "/usr/local/lib/python3.11/subprocess.py", line 571, in run
[2024-03-21 00:05:30,202]     raise CalledProcessError(retcode, process.args,
[2024-03-21 00:05:30,202] subprocess.CalledProcessError: Command '['dbt', '--log-format', 'json', 'run-operation', 'elementary.log_macro_results', '--args', '{"macro_name": "elementary_cli.get_test_results", "macro_args": {"days_back": 7, "invocations_per_test": 720, "disable_passed_test_metrics": false}}', '--project-dir', '/usr/local/airflow/.local/lib/python3.11/site-packages/elementary/monitor/dbt_project', '--profiles-dir', '/usr/local/airflow/dags/dbt/profiles', '--target', 'snowflake-mwaa']' died with <Signals.SIGKILL: 9>.
[2024-03-21 00:05:30,202] During handling of the above exception, another exception occurred:
[2024-03-21 00:05:30,202] Traceback (most recent call last):
[2024-03-21 00:05:30,202]   File "/usr/local/airflow/.local/bin/edr", line 8, in <module>
[2024-03-21 00:05:30,205]     sys.exit(cli())
[2024-03-21 00:05:30,205]              ^^^^^
[2024-03-21 00:05:30,206]   File "/usr/local/airflow/.local/lib/python3.11/site-packages/click/core.py", line 1130, in __call__
[2024-03-21 00:05:30,208]     return self.main(*args, **kwargs)
[2024-03-21 00:05:30,208]            ^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-03-21 00:05:30,209]   File "/usr/local/airflow/.local/lib/python3.11/site-packages/click/core.py", line 1055, in main
[2024-03-21 00:05:30,209]     rv = self.invoke(ctx)
[2024-03-21 00:05:30,209]          ^^^^^^^^^^^^^^^^
[2024-03-21 00:05:30,209]   File "/usr/local/airflow/.local/lib/python3.11/site-packages/elementary/cli/cli.py", line 67, in invoke
[2024-03-21 00:05:30,210]     return super().invoke(ctx)
[2024-03-21 00:05:30,210]            ^^^^^^^^^^^^^^^^^^^
[2024-03-21 00:05:30,210]   File "/usr/local/airflow/.local/lib/python3.11/site-packages/click/core.py", line 1657, in invoke
[2024-03-21 00:05:30,210]     return _process_result(sub_ctx.command.invoke(sub_ctx))
[2024-03-21 00:05:30,210]                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-03-21 00:05:30,210]   File "/usr/local/airflow/.local/lib/python3.11/site-packages/click/core.py", line 1404, in invoke
[2024-03-21 00:05:30,211]     return ctx.invoke(self.callback, **ctx.params)
[2024-03-21 00:05:30,211]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-03-21 00:05:30,211]   File "/usr/local/airflow/.local/lib/python3.11/site-packages/click/core.py", line 760, in invoke
[2024-03-21 00:05:30,211]     return __callback(*args, **kwargs)
[2024-03-21 00:05:30,212]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-03-21 00:05:30,212]   File "/usr/local/airflow/.local/lib/python3.11/site-packages/click/decorators.py", line 26, in new_func
[2024-03-21 00:05:30,212]     return f(get_current_context(), *args, **kwargs)
[2024-03-21 00:05:30,212]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-03-21 00:05:30,212]   File "/usr/local/airflow/.local/lib/python3.11/site-packages/elementary/monitor/cli.py", line 712, in send_report
[2024-03-21 00:05:30,213]     sent_report_successfully = data_monitoring.send_report(
[2024-03-21 00:05:30,214]                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-03-21 00:05:30,214]   File "/usr/local/airflow/.local/lib/python3.11/site-packages/elementary/monitor/data_monitoring/report/data_monitoring_report.py", line 221, in send_report
[2024-03-21 00:05:30,215]     self.send_test_results_summary(
[2024-03-21 00:05:30,215]   File "/usr/local/airflow/.local/lib/python3.11/site-packages/elementary/monitor/data_monitoring/report/data_monitoring_report.py", line 284, in send_test_results_summary
[2024-03-21 00:05:30,216]     tests_api = TestsAPI(
[2024-03-21 00:05:30,216]                 ^^^^^^^^^
[2024-03-21 00:05:30,216]   File "/usr/local/airflow/.local/lib/python3.11/site-packages/elementary/monitor/api/tests/tests.py", line 39, in __init__
[2024-03-21 00:05:30,217]     self.test_results_db_rows = self._get_test_results_db_rows(
[2024-03-21 00:05:30,217]                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-03-21 00:05:30,217]   File "/usr/local/airflow/.local/lib/python3.11/site-packages/elementary/monitor/api/tests/tests.py", line 51, in _get_test_results_db_rows
[2024-03-21 00:05:30,217]     return self.tests_fetcher.get_all_test_results_db_rows(
[2024-03-21 00:05:30,217]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-03-21 00:05:30,217]   File "/usr/local/airflow/.local/lib/python3.11/site-packages/elementary/monitor/fetchers/tests/tests.py", line 22, in get_all_test_results_db_rows
[2024-03-21 00:05:30,219]     run_operation_response = self.dbt_runner.run_operation(
[2024-03-21 00:05:30,219]                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-03-21 00:05:30,219]   File "/usr/local/airflow/.local/lib/python3.11/site-packages/elementary/clients/dbt/dbt_runner.py", line 160, in run_operation
[2024-03-21 00:05:30,219]     success, command_output = self._run_command(
[2024-03-21 00:05:30,219]                               ^^^^^^^^^^^^^^^^^^
[2024-03-21 00:05:30,219]   File "/usr/local/airflow/.local/lib/python3.11/site-packages/elementary/clients/dbt/dbt_runner.py", line 99, in _run_command
[2024-03-21 00:05:30,219]     raise DbtCommandError(err, command_args, logs=logs)
[2024-03-21 00:05:30,220] elementary.exceptions.exceptions.DbtCommandError: Failed to run dbt command.
[2024-03-21 00:05:32,014] Command exited with return code 1
[2024-03-21 00:05:32,055] Task failed with exception
Traceback (most recent call last):
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/operators/bash.py", line 210, in execute
    raise AirflowException(
airflow.exceptions.AirflowException: Bash command failed. The command returned a non-zero exit code 1.
[2024-03-21 00:05:32,063] Marking task as FAILED. dag_id=financials_segment_data_pipeline, task_id=elementary.elementary_report_task, execution_date=20240320T230820, start_date=20240320T235954, end_date=20240321T000532
[2024-03-21 00:05:32,099] Failed to execute job 377625 for task elementary.elementary_report_task (Bash command failed. The command returned a non-zero exit code 1.; 8231)
[2024-03-21 00:05:32,232] Task exited with return code 1
[2024-03-21 00:05:32,268] 0 downstream tasks scheduled from follow-on schedule check
I have run 
dbt run -s elementary
again, and run the report and I had a successful run. It's hard to get the 
edr.log
off MWAA where it runs, but if I have further problems I will let you know (as I said, the problems were intermittent, so not sure if they will reoccur)
35 Views
Open in Slack
PreviousNext
Powered by