Is there a way to reopen this issue? https://github.com/elementary-data/elementary/issues/761 Should I open a new one?

Hey @Francesco Mucio! Can you share more information about the issues you’re having before we’ll reopen it?

to me it look like the same problem. I am running Elementary in a Airflow K8S pod (we even raised the pod ram to 16GiB and reduce Elementary to process 1 day only of data), this was fine till a week ago. Now the pod gets terminated with the following error:

'state': {'running': None,
                                              'terminated': {'container_id': '<containerd://f4548ea025b662a2fd7004fd0a6a46c02e7b17e108b3408d460d9326290ab35>b',
                                                             'exit_code': 1,
                                                             'finished_at': datetime.datetime(2023, 12, 4, 0, 56, 46, tzinfo=tzlocal()),
                                                             'message': None,
                                                             'reason': 'OOMKilled',
                                                             'signal': None,
                                                             'started_at': datetime.datetime(2023, 12, 4, 0, 51, 24, tzinfo=tzlocal())},
                                              'waiting': None}}],

This is what I get before the termination:

[2023-12-04, 00:51:38 UTC] {pod_manager.py:418} INFO - [base] 2023-12-04 00:51:38 — INFO — edr (0.12.0) and Elementary's dbt package (0.12.1) are compatible.
[2023-12-04, 00:51:40 UTC] {pod_manager.py:418} INFO - [base] 2023-12-04 00:51:40 — INFO — Elementary's database and schema: '"prod_dataplatform.elementary"'
[2023-12-04, 00:51:40 UTC] {pod_manager.py:418} INFO - [base] 2023-12-04 00:51:40 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_test_results", "macro_args": {"days_back": 1, "invocations_per_test": 720, "disable_passed_test_metrics": false}} --project-dir /usr/local/lib/python3.10/site-packages/elementary/monitor/dbt_project
[2023-12-04, 00:56:46 UTC] {pod_manager.py:418} INFO - [base] [31;20m2023-12-04 00:56:46 — ERROR — Could not generate the report - Error: Failed to run dbt command.
[2023-12-04, 00:56:46 UTC] {pod_manager.py:418} INFO - [base] Please reach out to our community for help with this issue.[0m
[2023-12-04, 00:56:46 UTC] {pod_manager.py:418} INFO - [base] NoneType: None

I also found this recent issue on Github which seems similar. The fact that the log provides little information makes it very hard to debug this issue. Is there anything I can try to investigate this problem?

Can you maybe try to turn elementary off and run dbt without it once just to make sure it’s 100% related to elementary run? Did you change any versions? (you said it was running fine until a week ago - did anything change a week ago?)

the pod is only running Elementary, the dbt run happens in a separated pod, we just copy the dbt artifacts over

Got you, and what about the versions? Did you recently upgrade?

I think we upgraded to 0.12.0 or 0.12.1

Interesting thanks! Last thing - any chance you can get me the

edr.log

from this pod?

not easily

Thanks!

i'm facing this problem aswell. Started with 4GB limit, increases to 8 GB RAM limit. Now its crashing again because RAM Limit. Any idea on how to improve this?

for me the fix was to rerun the elementary model to update the tables

Hey @Andre Almeida can you try @Francesco Mucio’s solution and let us know if it helped?

ok. when i'm dealing again with the issue, i will try it. let you know later