Peter
10/24/2024, 6:54 AMedr report locally from vscode (devcontainer): I get ERROR — Unable to get the latest invocation: Failed to run dbt command. Encountered an error while running operation: Database Error ('communication error', ConnectionRefusedError(111, 'Connection refused'))
• dbt `1.8.3`; `elementary-data==0.15.1`; and the dbt package is 0.15.2
• I was able to run this command before and we’re running it daily from the same container on airflow
• I’m able to run dbt run -s elementary (also with --full-refresh)
• detailed logs in the thread
• any help would be appreciated 🙏Peter
10/24/2024, 6:58 AM2024-10-24 06:46:50 — DEBUG — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_elementary_database_and_schema", "macro_args": {}} --project-dir /usr/local/lib/python3.10/site-packages/elementary/monitor/dbt_project
2024-10-24 06:48:08 — ERROR — Failed to parse Elementary's database and schema.
2024-10-24 06:48:08 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_test_results", "macro_args": {"days_back": 7, "invocations_per_test": 720, "disable_passed_test_metrics": false}} --project-dir /usr/local/lib/python3.10/site-packages/elementary/monitor/dbt_project
2024-10-24 06:49:26 — ERROR — Could not generate the report - Error: Failed to run dbt command.
Encountered an error while running operation: Database Error
('communication error', ConnectionRefusedError(111, 'Connection refused'))
Please reach out to our community for help with this issue.[0m
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/elementary/clients/dbt/dbt_runner.py", line 88, in _run_command
result = subprocess.run(
File "/usr/local/lib/python3.10/subprocess.py", line 526, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['dbt', '--log-format', 'json', 'run-operation', 'elementary.log_macro_results', '--args', '{"macro_name": "elementary_cli.get_test_results", "macro_args": {"days_back": 7, "invocations_per_test": 720, "disable_passed_test_metrics": false}}', '--project-dir', '/usr/local/lib/python3.10/site-packages/elementary/monitor/dbt_project']' returned non-zero exit status 1.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/elementary/monitor/api/report/report.py", line 50, in get_report_data
tests_api = TestsAPI(
File "/usr/local/lib/python3.10/site-packages/elementary/monitor/api/tests/tests.py", line 43, in __init__
self.test_results_db_rows = self._get_test_results_db_rows(
File "/usr/local/lib/python3.10/site-packages/elementary/monitor/api/tests/tests.py", line 55, in _get_test_results_db_rows
return self.tests_fetcher.get_all_test_results_db_rows(
File "/usr/local/lib/python3.10/site-packages/elementary/monitor/fetchers/tests/tests.py", line 25, in get_all_test_results_db_rows
run_operation_response = self.dbt_runner.run_operation(
File "/usr/local/lib/python3.10/site-packages/elementary/clients/dbt/dbt_runner.py", line 160, in run_operation
success, command_output = self._run_command(
File "/usr/local/lib/python3.10/site-packages/elementary/clients/dbt/dbt_runner.py", line 99, in _run_command
raise DbtCommandError(err, command_args, logs=logs)
elementary.exceptions.exceptions.DbtCommandError: Failed to run dbt command.
Encountered an error while running operation: Database Error
('communication error', ConnectionRefusedError(111, 'Connection refused'))Peter
10/24/2024, 7:03 AM[0m17:04:03.074099 [debug] [MainThread]: Acquiring new redshift connection 'master'
[0m17:04:03.081336 [debug] [ThreadPool]: Acquiring new redshift connection 'list_live'
[0m17:04:03.092111 [debug] [ThreadPool]: Using redshift connection "list_live"
[0m17:04:03.092904 [debug] [ThreadPool]: On list_live: /* {"app": "dbt", "dbt_version": "1.8.3", "profile_name": "komoot-redshift", "target_name": "dev", "connection_name": "list_live"} */
select distinct nspname from pg_namespace
[0m17:04:03.093635 [debug] [ThreadPool]: Opening a new connection, currently in state init
[0m17:04:03.096364 [debug] [ThreadPool]: Redshift adapter: Establishing connection using ssl with `sslmode` set to 'prefer'.To connect without ssl, set `sslmode` to 'disable'.
[0m17:04:03.097612 [debug] [ThreadPool]: Redshift adapter: Connecting to redshift with username/password based auth...
[0m17:04:03.157679 [debug] [ThreadPool]: Redshift adapter: Error running SQL: /* {"app": "dbt", "dbt_version": "1.8.3", "profile_name": "komoot-redshift", "target_name": "dev", "connection_name": "list_live"} */
select distinct nspname from pg_namespace
[0m17:04:03.158459 [debug] [ThreadPool]: Redshift adapter: Rolling back transaction.
[0m17:04:03.159386 [debug] [ThreadPool]: Redshift adapter: Error running SQL: macro list_schemas
[0m17:04:03.160079 [debug] [ThreadPool]: Redshift adapter: Rolling back transaction.
[0m17:04:03.160582 [debug] [ThreadPool]: On list_live: No close available on handle
[0m17:04:03.161395 [debug] [MainThread]: Connection 'master' was properly closed.
[0m17:04:03.161888 [debug] [MainThread]: Connection 'list_live' was properly closed.
[0m17:04:03.162435 [info ] [MainThread]:
[0m17:04:03.162961 [info ] [MainThread]: Finished running in 0 hours 0 minutes and 0.09 seconds (0.09s).
[0m17:04:03.163661 [error] [MainThread]: Encountered an error:
Database Error
('communication error', ConnectionRefusedError(111, 'Connection refused'))
[0m17:04:03.164750 [debug] [MainThread]: Resource report: {"command_name": "run", "command_wall_clock_time": 36.3623, "process_user_time": 32.85004, "process_kernel_time": 0.725583, "process_mem_max_rss": "367784", "process_in_blocks": "584", "command_success": false, "process_out_blocks": "0"}
[0m17:04:03.165691 [debug] [MainThread]: Command `dbt run` failed at 17:04:03.165622 after 36.36 seconds
[0m17:04:03.166794 [debug] [MainThread]: Flushing usage eventsPeter
10/24/2024, 7:04 AMselect distinct nspname from pg_namespace line. I’m also able to query that from within vscode.Joost Boonzajer Flaes
10/27/2024, 12:08 PMPeter
10/29/2024, 3:05 PMedr report : PermissionError: [Errno 13] Permission denied: '/usr/local/lib/python3.10/site-packages/elementary/monitor/dbt_project/target/partial_parse.msgpack'
• to resolve this, I’ve switched to running the command with sudo: sudo edr report
• unfortunately it meant that the user was switched and our environ variables were ‘forgotten’
• since we use environ vars in our profiles.yml, the connection was refused…
solution
• either switch to super user and then re-export the env vars
• or change the directory permissions sudo chmod -R +x '/usr/local/lib/python3.10/site-packages/elementary/monitor/dbt_project
thank you @Joost Boonzajer Flaes for you help 🙏