https://linen.dev logo
Join Slack
Powered by
hey everyone! Since couple of days, generating the...
# community-support
i
hey everyone! Since couple of days, generating the report in production fails (see trace in thread) with the following error 
elementary.exceptions.exceptions.DbtCommandError: Failed to run dbt command.
, even if nothing changed in terms of project structure, variables, and infrastructure. We run the commands from within a container, and if I do so locally (with the same env vars) the command succeeds, if I run it through AWS ECS it does not
Copy code
2024-10-21T07:55:51.168Z
	
2024-10-21 07:55:51 — INFO — edr (0.15.1) and Elementary's dbt package (0.15.2) are compatible.
	
2024-10-21T07:55:56.562Z
	
2024-10-21 07:55:56 — INFO — Elementary's database and schema: '"INSIGHT_PROD.ELEMENTARY"'
	
2024-10-21T07:55:56.563Z
	
2024-10-21 07:55:56 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_test_results", "macro_args": {"days_back": 16, "invocations_per_test": 720, "disable_passed_test_metrics": false}} --project-dir /usr/local/lib/python3.9/site-packages/elementary/monitor/dbt_project
	
2024-10-21T07:57:35.950Z
	
2024-10-21 07:57:35 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_source_freshness_results", "macro_args": {"days_back": 16, "invocations_per_test": 720}} --project-dir /usr/local/lib/python3.9/site-packages/elementary/monitor/dbt_project
	
2024-10-21T07:57:47.182Z
	
2024-10-21 07:57:47 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_models", "macro_args": {"exclude_elementary": true}} --project-dir /usr/local/lib/python3.9/site-packages/elementary/monitor/dbt_project
	
2024-10-21T07:57:54.206Z
	
2024-10-21 07:57:54 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_sources", "macro_args": {}} --project-dir /usr/local/lib/python3.9/site-packages/elementary/monitor/dbt_project
	
2024-10-21T07:58:01.567Z
	
2024-10-21 07:58:01 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_exposures", "macro_args": {}} --project-dir /usr/local/lib/python3.9/site-packages/elementary/monitor/dbt_project
	
2024-10-21T07:58:09.032Z
	
2024-10-21 07:58:09 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_singular_tests", "macro_args": {}} --project-dir /usr/local/lib/python3.9/site-packages/elementary/monitor/dbt_project
	
2024-10-21T07:58:15.626Z
	
2024-10-21 07:58:15 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_models_runs", "macro_args": {"days_back": 16, "exclude_elementary": true}} --project-dir /usr/local/lib/python3.9/site-packages/elementary/monitor/dbt_project
	
2024-10-21T07:59:01.996Z
	
[31;20m2024-10-21 07:59:01 — ERROR — Could not generate the report - Error: Failed to run dbt command.
	
2024-10-21T07:59:01.996Z
	
Please reach out to our community for help with this issue.[0m
	
2024-10-21T07:59:01.996Z
	
Traceback (most recent call last):
	
2024-10-21T07:59:01.996Z
	
File "/usr/local/lib/python3.9/site-packages/elementary/clients/dbt/dbt_runner.py", line 88, in _run_command
	
2024-10-21T07:59:01.996Z
	
result = subprocess.run(
	
2024-10-21T07:59:01.996Z
	
File "/usr/local/lib/python3.9/subprocess.py", line 528, in run
	
2024-10-21T07:59:01.996Z
	
raise CalledProcessError(retcode, process.args,
	
2024-10-21T07:59:01.996Z
	
subprocess.CalledProcessError: Command '['dbt', '--log-format', 'json', 'run-operation', 'elementary.log_macro_results', '--args', '{"macro_name": "elementary_cli.get_models_runs", "macro_args": {"days_back": 16, "exclude_elementary": true}}', '--project-dir', '/usr/local/lib/python3.9/site-packages/elementary/monitor/dbt_project']' died with <Signals.SIGKILL: 9>.
	
2024-10-21T07:59:01.996Z
	
During handling of the above exception, another exception occurred:
	
2024-10-21T07:59:01.996Z
	
Traceback (most recent call last):
	
2024-10-21T07:59:01.996Z
	
File "/usr/local/lib/python3.9/site-packages/elementary/monitor/api/report/report.py", line 85, in get_report_data
	
2024-10-21T07:59:01.996Z
	
models_runs = models_api.get_models_runs(
	
2024-10-21T07:59:01.996Z
	
File "/usr/local/lib/python3.9/site-packages/elementary/monitor/api/models/models.py", line 39, in get_models_runs
	
2024-10-21T07:59:01.996Z
	
model_runs_results = self.models_fetcher.get_models_runs(
	
2024-10-21T07:59:01.996Z
	
File "/usr/local/lib/python3.9/site-packages/elementary/monitor/fetchers/models/models.py", line 24, in get_models_runs
	
2024-10-21T07:59:01.996Z
	
run_operation_response = self.dbt_runner.run_operation(
	
2024-10-21T07:59:01.996Z
	
File "/usr/local/lib/python3.9/site-packages/elementary/clients/dbt/dbt_runner.py", line 160, in run_operation
	
2024-10-21T07:59:01.996Z
	
success, command_output = self._run_command(
	
2024-10-21T07:59:01.996Z
	
File "/usr/local/lib/python3.9/site-packages/elementary/clients/dbt/dbt_runner.py", line 99, in _run_command
	
2024-10-21T07:59:01.996Z
	
raise DbtCommandError(err, command_args, logs=logs)
	
2024-10-21T07:59:01.996Z
	
elementary.exceptions.exceptions.DbtCommandError: Failed to run dbt command.
	
2024-10-21T07:59:03.592Z
	
cleanup
	
2024-10-21T07:59:03.592Z
	
cleanup
j
Hi Igor! Hard to say but we can do a couple of steps to try and debug. Are you running the same pip packages (ie, are the versions hardcoded or is there difference between local/cloud?) Are you able to exec into the ECS container? If so, can you run 
dbt run
/`dbt test` there without any issues? And is there any more info in your edr.log (if you can access this?) It should sit in 
edr_target/edr.log
in the container
i
hey Joost, I can't really access the container right now because I have some issues with connecting to the container through ssh 😕 I can exec any dbt command within the container (they actually run just before the elementary report command) all requirements are the same local - cloud is there a way I can print the log in the console? that would be the easiest way to access them (or somehow have a verbose command)
j
Not a way to print it to the console afaik. Hmm interesting - and just to double check, you are connecting to the same elementary schema (prod/dev)?
i
yeah exactly same env in my local machine, this is the proof:
Copy code
edr report --env prod --days-back 16 --file-path test_report.html
    ________                          __
   / ____/ /__  ____ ___  ___  ____  / /_____ ________  __
  / __/ / / _ \/ __ `__ \/ _ \/ __ \/ __/ __ `/ ___/ / / /
 / /___/ /  __/ / / / / /  __/ / / / /_/ /_/ / /  / /_/ /
/_____/_/\___/_/ /_/ /_/\___/_/ /_/\__/\__,_/_/   \__, /
                                                 /____/

You are using Elementary 0.15.1, however version 0.16.1 is available.
Consider upgrading by running: "/usr/local/bin/python3.9 -m pip install --upgrade elementary-data"

Any feedback and suggestions are welcomed! join our community here - <https://bit.ly/slack-elementary>

2024-10-21 09:04:58 — INFO — Running with edr=0.15.1
2024-10-21 09:05:23 — INFO — edr (0.15.1) and Elementary's dbt package (0.15.2) are compatible.
2024-10-21 09:05:32 — INFO — Elementary's database and schema: '"INSIGHT_PROD.ELEMENTARY"'
2024-10-21 09:05:32 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_test_results", "macro_args": {"days_back": 16, "invocations_per_test": 720, "disable_passed_test_metrics": false}} --project-dir /usr/local/lib/python3.9/site-packages/elementary/monitor/dbt_project
2024-10-21 09:08:31 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_source_freshness_results", "macro_args": {"days_back": 16, "invocations_per_test": 720}} --project-dir /usr/local/lib/python3.9/site-packages/elementary/monitor/dbt_project
2024-10-21 09:08:48 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_models", "macro_args": {"exclude_elementary": true}} --project-dir /usr/local/lib/python3.9/site-packages/elementary/monitor/dbt_project
2024-10-21 09:08:59 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_sources", "macro_args": {}} --project-dir /usr/local/lib/python3.9/site-packages/elementary/monitor/dbt_project
2024-10-21 09:09:10 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_exposures", "macro_args": {}} --project-dir /usr/local/lib/python3.9/site-packages/elementary/monitor/dbt_project
2024-10-21 09:09:22 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_singular_tests", "macro_args": {}} --project-dir /usr/local/lib/python3.9/site-packages/elementary/monitor/dbt_project
2024-10-21 09:09:32 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_models_runs", "macro_args": {"days_back": 16, "exclude_elementary": true}} --project-dir /usr/local/lib/python3.9/site-packages/elementary/monitor/dbt_project
2024-10-21 09:11:26 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_dbt_models_test_coverage", "macro_args": {}} --project-dir /usr/local/lib/python3.9/site-packages/elementary/monitor/dbt_project
2024-10-21 09:11:54 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_nodes_depends_on_nodes", "macro_args": {"exclude_elementary": true}} --project-dir /usr/local/lib/python3.9/site-packages/elementary/monitor/dbt_project
2024-10-21 09:12:13 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_models_latest_invocation", "macro_args": {}} --project-dir /usr/local/lib/python3.9/site-packages/elementary/monitor/dbt_project
2024-10-21 09:12:26 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_models_latest_invocations_data", "macro_args": {}} --project-dir /usr/local/lib/python3.9/site-packages/elementary/monitor/dbt_project
j
I see, thanks for sharing that! I think the issue is probably here: 
subprocess.CalledProcessError: Command '['dbt', '--log-format', 'json', 'run-operation', 'elementary.log_macro_results', '--args', '{"macro_name": "elementary_cli.get_models_runs", "macro_args": {"days_back": 16, "exclude_elementary": true}}', '--project-dir', '/usr/local/lib/python3.9/site-packages/elementary/monitor/dbt_project']' died with <Signals.SIGKILL: 9>.
The error you're seeing, 
subprocess.CalledProcessError: Command '['dbt', ...] died with <Signals.SIGKILL: 9>
means that the 
dbt
process was forcibly killed. Signal 
SIGKILL
(9) is typically sent when a process exceeds available resources or is manually killed. So it looks like the container is running into it's limits which can be an issue with the resources allocated to it or a timeout. I don't think it's a timeout issue. Can you try increasing memory/cpu?
You can probably verify this in the ECS monitoring
i
yeah I was wondering the same if it was a memory issue 😕 thanks for the hint, I am gonna check this
j
Yes but it should be relatively simple to increase the limits for the container! 🙂
i
right, gonna try that
👍 1
j
Let me know if it resolves the issue!
🙌 1
i
thanks, that was it! it works now 🙂
j
Amazing 🙂 Thanks for letting me know!
🙌 1
30 Views
Open in Slack
PreviousNext
Powered by