You’re right to be concerned—by design, Elementary will keep accumulating history unless you actively manage retention. There is no built‑in global retention / auto‑cleanup for Athena/S3 or other warehouses; you must control growth yourself. [[Results lifecycle](https://elementary-community.slack.com/archives/C02CTC89LAX/p1689146074949549); [Metadata cleanup options](https://elementary-community.slack.com/archives/C02CTC89LAX/p1687380949503199)] Below are the options that are documented. --- ## 1. How to limit historical data going forward Elementary’s models and reports use different “time windows,” but none of them automatically delete old rows; they only limit what is processed or shown: - Anomaly tests (metrics in

data_monitoring_metrics

) You can limit how much history is used by anomaly detection via

training_period

and

detection_period

in your test configs, but this does not purge older rows already stored. [[Anomaly config](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/training-period); [Detection period](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/detection-period)] - Reports (

edr report --days-back

)

--days-back

only limits what appears in the HTML/JSON report and how much data is processed when generating it; it does not control how long data is kept in warehouse tables. [[High RAM & days-back](https://www.linen.dev/s/elementary-community/t/29111594/u088wcp3b4p-my-elementary-container-is-really-heavy-in-terms)] So, to truly limit growth, you need to: 1. Decide a retention window (e.g., keep last 30/60/90 days). 2. Implement periodic cleanup SQL against Elementary tables (see next section). 3. Optionally, add S3 lifecycle rules to expire old objects if you are storing artifacts or table data in S3. [[Metadata lifecycle](https://www.linen.dev/s/elementary-community/t/29153856/u088wcp3b4p-i-have-anomaly-tests-with-the-following-configur)] There is currently no documented Elementary setting that says “only store last N days” and automatically enforces it. --- ## 2. How to clean existing Elementary data (warehouse tables) Elementary maintainers explicitly say you can treat Elementary tables like any other tables and delete old rows yourself: - Incremental tables such as: -

elementary_test_results

-

dbt_run_results

-

dbt_invocations

-

data_monitoring_metrics

grow over time and are not auto‑pruned. [[Lifecycle & growth](https://elementary-community.slack.com/archives/C02CTC89LAX/p1689146074949549); [Storage scaling](https://elementary-community.slack.com/archives/C02CTC89LAX/p1683451667062879)] - Recommended approach is to run periodic

DELETE

statements using a date column (

created_at

,

updated_at

, etc.): - For anomaly metrics:

sql     DELETE FROM data_monitoring_metrics     WHERE updated_at < <threshold_date>;

[[Cleanup metrics](https://elementary-community.slack.com/archives/C02CTC89LAX/p1766509160109659?thread_ts=1766509160.109659&cid=C02CTC89LAX)] - For general Elementary history (example pattern):

sql     DELETE FROM elementary_test_results     WHERE created_at < <threshold_date>;      DELETE FROM dbt_run_results     WHERE created_at < <threshold_date>;      DELETE FROM dbt_invocations     WHERE created_at < <threshold_date>;

[[Results lifecycle](https://elementary-community.slack.com/archives/C02CTC89LAX/p1689146074949549); [Storage scaling](https://elementary-community.slack.com/archives/C02CTC89LAX/p1683451667062879)] You’d adapt these to Athena SQL and your actual column names. - Some users instead recreate the whole Elementary schema when they want a clean slate:

bash   dbt run -s elementary -t prod -f --profile elementary

This effectively truncates and rebuilds Elementary models, but you lose all history. [[Schema reset](https://elementary-community.slack.com/archives/C02CTC89LAX/p1719390641107289?thread_ts=1719390641.107289&cid=C02CTC89LAX)] --- ## 3. Cleaning / limiting dbt artifacts (JSON, S3, etc.) From the available sources: - Elementary uploads dbt artifacts (manifest, run_results, etc.) into warehouse tables; these are part of the schema that keeps growing. [[Storage scaling](https://elementary-community.slack.com/archives/C02CTC89LAX/p1683451667062879)] - There is no general CLI cleanup for all artifacts/tables. The only documented CLI cleanup flag is

edr monitor -f true

, which cleans only the

alerts_v2

table. [[CLI cleanup scope](https://elementary-community.slack.com/archives/C02CTC89LAX/p1719390641107289?thread_ts=1719390641.107289&cid=C02CTC89LAX)] - For S3‑backed data (Athena / Hive / Iceberg), users are advised to: - Use S3 lifecycle policies to expire old objects beyond your reporting window. [[Metadata lifecycle](https://www.linen.dev/s/elementary-community/t/29153856/u088wcp3b4p-i-have-anomaly-tests-with-the-following-configur)] - Optionally partition tables (e.g., by date) and prune partitions, though the Athena partitioning re‑engineering is still an open topic. [[Athena integration issue](https://github.com/elementary-data/elementary/issues/1698)] So for “artefacts of dbt elementary” in Athena/S3, the documented options are: - Warehouse side:

DELETE

old rows from artifact tables (e.g.,

dbt_run_results

,

dbt_invocations

,

elementary_*

). - S3 side: configure bucket lifecycle rules to expire old data files if your Elementary tables or raw artifacts are stored in S3. --- ## 4. Practical next steps for your 150 TB Athena/S3 setup Based strictly on the docs and community answers: 1. Decide retention (e.g., 60–90 days of history). 2. For each large Elementary table in Athena: - Identify the date column (

created_at

,

updated_at

, etc.). - Run one‑off

DELETE

statements to remove data older than your retention threshold. - Schedule this as a recurring job (e.g., daily) so growth is bounded. 3. If your Elementary tables are in S3 (Hive/Iceberg): - Add S3 lifecycle rules to expire objects older than your retention window. 4. If you ever want a full reset: - Run

dbt run -s elementary -t <target> -f --profile elementary

to rebuild the schema from scratch. [[Schema reset](https://elementary-community.slack.com/archives/C02CTC89LAX/p1719390641107289?thread_ts=1719390641.107289&cid=C02CTC89LAX)] The knowledge sources do not describe any built‑in, configurable retention feature that would automatically keep only the last N days across all Elementary tables, so everything above is based on manual SQL / lifecycle policies.

The knowledge sources don’t give a definitive, engine‑level guarantee, but they do clarify a few important points: 1. Elementary tables can be safely pruned Elementary explicitly recommends deleting old rows from its tables (e.g.

data_monitoring_metrics

,

dbt_run_results

,

dbt_invocations

,

elementary_test_results

) using a date column like

created_at

/

updated_at

. [[Results lifecycle](https://elementary-community.slack.com/archives/C02CTC89LAX/p1689146074949549); [Metrics cleanup](https://elementary-community.slack.com/archives/C02CTC89LAX/p1766509160109659?thread_ts=1766509160.109659&cid=C02CTC89LAX)] 2. But metadata bloat in S3 is engine‑specific For Iceberg/Trino, there’s a separate issue where Iceberg metadata files keep growing even when table data is managed; Trino added specific configs like

write.metadata.delete-after-commit.enabled

and

write.metadata.previous-versions-max

to control that. [[Iceberg metadata growth](https://github.com/elementary-data/elementary/issues/1835)] That shows that simply reducing table row counts doesn’t automatically guarantee that all historical metadata files in S3 are cleaned up; it depends on how your engine (Athena + underlying table format) manages metadata. 3. What is documented as “working” today From Elementary’s side, the supported levers are: - Delete old rows from Elementary tables (this reduces logical data size and, depending on your table format/engine, may or may not immediately reclaim all S3 metadata). [[Results lifecycle](https://elementary-community.slack.com/archives/C02CTC89LAX/p1689146074949549)] - Use S3 lifecycle policies to expire old objects if you know that older data/metadata files are no longer needed. [[Metadata lifecycle](https://www.linen.dev/s/elementary-community/t/29153856/u088wcp3b4p-i-have-anomaly-tests-with-the-following-configur)] Because the sources don’t describe Athena’s exact behavior with your table format, I can’t confidently say “yes, reducing table size will automatically shrink S3 metadata TB.” What is clear is: - Reducing table history (via

DELETE

or full rebuild) is necessary to cap growth from Elementary’s perspective. - To actually reduce S3 TB when “most of the TB are metadata,” you will likely also need engine‑/format‑specific metadata cleanup (e.g., Iceberg/Trino settings, compaction, or S3 lifecycle rules), which is outside what Elementary itself controls. [[Iceberg metadata growth](https://github.com/elementary-data/elementary/issues/1835)] So: pruning Elementary tables is the right first step, but on its own it may not be sufficient to reclaim all S3 metadata space, depending on how Athena + your table format manage metadata files.