<@U088WCP3B4P>, we are looking for ways that my `e...
# ask-elementary-ai
f
@kapa.ai, we are looking for ways that my
edr monitor
command will perform better in Airflow. We now execute
edr monitor --filters tags:my-team --profile-target default --maximum-columns-in-alert-samples 25
. We run into issues on airflow where the command fails with
died with <Signals.SIGKILL: 9>
.
What can we do to reduce memory usage of that command?
k
A message from kapa.ai
The error
died with <Signals.SIGKILL: 9>
almost always indicates that the process was killed due to running out of memory (OOM) in your Airflow environment. This is a common issue when running
edr monitor
or similar Elementary Data commands, especially when processing large amounts of data or with limited resources. Here are several strategies, based on community experience and support recommendations, to reduce memory usage and improve the reliability of your
edr monitor
command in Airflow: 1. Limit the Data Window with
--days-back
By default, Elementary processes 7 days of data, which can be a lot. Reducing the window with
--days-back 1
(or another small number) significantly decreases the amount of data loaded into memory:
Copy code
edr monitor --days-back 1 --filters tags:my-team --profile-target default --maximum-columns-in-alert-samples 25
This is the most frequently recommended solution and has resolved OOM issues for many users [see example](https://slack.com/archives/C02CTC89LAX/p1685079843099879), [another example](https://slack.com/archives/C02CTC89LAX/p1663241888773889). 2. Reduce Execution Limits If you have a lot of test executions, you can further limit memory usage by lowering the
--executions-limit
parameter (default is 720). For example:
Copy code
edr monitor --days-back 1 --executions-limit 120 --filters tags:my-team --profile-target default --maximum-columns-in-alert-samples 25
Lowering this value has helped users avoid OOM errors [see here](https://slack.com/archives/C02CTC89LAX/p1701791821617429). 3. Increase Airflow Task Memory If possible, increase the memory allocated to your Airflow worker or Kubernetes pod. Many users have found that the default memory is insufficient for Elementary's workload, especially with larger datasets [see discussion](https://slack.com/archives/C02CTC89LAX/p1710250385626209). 4. Limit Columns in Alert Samples You are already using
--maximum-columns-in-alert-samples 25
, which is good. If you can reduce this further, it may help, but the biggest impact comes from limiting the data window and executions. 5. Monitor and Tune Monitor the actual memory usage of your Airflow tasks. Sometimes, even with reduced parameters, the memory required can be substantial, and further tuning or hardware scaling may be necessary [see example](https://slack.com/archives/C02CTC89LAX/p1685079843099879). Summary: Start by adding
--days-back 1
and, if needed, reduce
--executions-limit
. If you still encounter OOM errors, increase the memory allocated to your Airflow worker or pod. These steps are the most effective and widely recommended by both the Elementary team and community members. If you continue to have issues, consider sharing your
edr.log
with Elementary support for more targeted troubleshooting, as sometimes specific queries or data volumes can cause unexpected spikes in memory usage. Let me know if you need help with the exact command syntax or further troubleshooting!