@kapa.ai

edr report

getting OOM-killed, looking for ways to reduce its memory footprint We run

edr report

(edr

0.26.0

, dbt package

0.26.0

, Snowflake) inside a container (AWS ECS Fargate). On one of our larger projects the report generation gets OOM-killed — the process exits with

Killed

(SIGKILL) and we see

resource_tracker: ... leaked semaphore objects

at shutdown, right after it loads test results. From the logs, the underlying macro call is:

elementary_cli.get_test_results
days_back: 7, invocations_per_test: 720,
disable_passed_test_metrics: false, skip_test_result_rows: false

We invoke it simply as:

edr report --profiles-dir ... --target-path ... --file-path ...

We understand we can throw more memory at it, but we'd like to reduce what the report loads into memory too. Questions: 1. Are there

edr report

CLI flags (or config) to control

invocations_per_test

,

days_back

,

disable_passed_test_metrics

, and

skip_test_result_rows

?

720

invocations/test seems high — is that the default, and can it be lowered? 2. What's the recommended memory for

edr report

on larger projects? 3. Any guidance on generating the report with a lower memory footprint (pagination/streaming, splitting the window, etc.)? Thanks a lot — happy to share more logs or config if helpful! 🙏

Thinking completed 1. CLI Flags to Control Data Loading Yes, there are direct CLI flags mapping to each of the arguments you see in the log [Linen Conversation; Report Size Fix]: •

--days-back

(maps to

days_back

): The default is

7

days. Lowering this to

3

or

1

is the single most effective way to cut down memory usage [Report Size Fix; Memory Tuning]. •

--executions-limit

(maps to

invocations_per_test

): Yes,

720

is the default value [Executions Limit Context]. It sets how many historical test invocations per test are loaded into memory and displayed. You can lower this significantly to values like

100

,

50

, or even

10

[Linen Conversation; Memory Tuning]. •

--disable-passed-test-metrics true

(maps to

disable_passed_test_metrics

): Excludes metric graphs and historical trends for tests that passed while still displaying the test status, significantly reducing query overhead and memory consumption [Linen Conversation]. •

--disable-samples true

(maps to

skip_test_result_rows

): Omits row-level failure samples from the report data [Linen Conversation; Report Size Fix]. You can combine these options directly:

edr report \
  --profiles-dir ... \
  --days-back 3 \
  --executions-limit 50 \
  --disable-passed-test-metrics true \
  --disable-samples true

[Report Size Fix] 2. Recommended Memory for Larger Projects Because

edr report

builds a self-contained static HTML file with all the loaded model, test, and execution data embedded as JSON, large projects are inherently memory-intensive [Linen Conversation; Serving Lag Context]. Community experience and support recommendations indicate that containers/workers often require 4 GB to 8 GB (or higher) of RAM to handle large workloads reliably [Linen Conversation; Airflow Memory Exit -9; Heavy RAM Container]. 3. Guidance for Lower Memory Footprints • Pagination / Streaming: The knowledge sources do not explicitly mention built-in streaming or pagination flags for the CLI report generation command itself;

edr report

is designed to compile everything into a single static file [Serving Lag Context]. (Paging parameters like

limit

and

start_time_from

are available on the Elementary Cloud REST API endpoints, but not documented as streaming mechanisms for the static CLI report) [List Latest Executions; List Test Executions]. • Disable Failure Samples at the dbt Source: If failed test rows are large, you can disable them globally in

dbt_project.yml

so they are not captured in the warehousevars test_sample_row_count: 0[Test Result Samples; Linen Conversation] • Full-Refresh Elementary Models: If historical Elementary warehouse tables (such as

elementary_test_results

) have grown excessively over time, you can clean them up by running:dbt run --select elementary --full-refresh_(Note: this clears older historical test data)_ [Linen Conversation; Report Size Fix]. 👍 0 button 👎 0 button ℹ️ Tip: Mention @elementary.ai in the thread for follow-ups or DM the app for 1-to-1 help.

"--target-path" is saving temporary file in ecs container, if saving things in the s3 bucket could save place/memory? @kapa.ai

Thinking completed The knowledge sources do not explicitly mention that saving directly to S3 reduces container memory (RAM) usage. Here is what the documentation and knowledge sources cover regarding

--target-path

and S3: • What

--target-path

does: The

--target-path

option controls the local disk directory where Elementary writes local output files, such as

edr.log

and the generated HTML report (default is

./edr_target

) [CLI Options; Target Path Linen Conversation]. It manages local disk storage/cleanliness, not process RAM allocation. • S3 usage (

edr send-report

): You can upload and host reports directly on S3 using

edr send-report --s3-bucket-name <BUCKET_NAME>

[Host the report on Amazon S3]. While this saves persistent disk space on your container, users running

edr send-report

still report hitting container OOM issues when loading large datasets into memory [Memory Tuning]. The Out-Of-Memory (OOM) issue occurs because Python must query, load, and serialize all historical test metrics, executions, and samples into memory before writing or uploading the report [Linen Conversation; Heavy RAM Container]. To prevent OOM errors, the supported approach is to restrict the data loaded into memory using flags such as

--days-back

,

--executions-limit

,

--disable-passed-test-metrics true

, and

--disable-samples true

[Linen Conversation; Report Size Fix]. 👍 0 button 👎 0 button ℹ️ Tip: Mention @elementary.ai in the thread for follow-ups or DM the app for 1-to-1 help.