Hi team, we're running into repeated

OOMKills

issues on our datanodes, and the memory usage feels disproportionate to our actual traffic. Wondering if you could help us understand what's going on. Setup: - Version:

v1.0.0-rc.2-nightly-20260323

- Cluster: 3 datanodes, 3 frontends, 3 meta, 1 flownode - Nodes: 3x

m7g.large

(8 GiB each) on EKS, us-east-2 - Storage: S3 object storage + EBS gp3 WAL (50Gi) - No custom datanode config — defaults only (config below) - No resource requests/limits set on pods (BestEffort QoS) Datanode config (all defaults, no compaction tuning):

[logging]
  dir = "/data/greptimedb/logs"
  level = "info"
  log_format = "text"

[storage]
  type = "S3"
  bucket = "..."
  region = "us-east-2"
  root = "greptime"

[wal]
  dir = "/wal"

The problem: Each datanode consumes 2.5–4.5 GiB of memory with almost no CPU load (2–7m cores). The traffic on this cluster is very low — we're an early-stage product, not running any heavy queries. Despite this, memory keeps growing and

datanode-2

has been OOMKilled twice (exit code 137). Current snapshot: | Pod | CPU | Memory | datanode-0 | 2m | 4,558 Mi | datanode-1 | 2m | 2,760 Mi | datanode-2 | 7m | 2,460 Mi (recently restarted, growing) For context, the non-datanode components are tiny in comparison: frontends ~90–130 Mi, meta ~40 Mi, flownode ~47 Mi. What we see in the logs before the OOM: 1. Massive TWCS compaction activity on a single region

4621384810496

(table 1076), processing 40+ time windows in rapid succession 2. Health probe timeouts (

context deadline exceeded

) — 24 liveness + 24 readiness failures 3. Then the OOM kill The last logs before the kill:

2026-03-31T01:18:21Z mito2::compaction::twcs: Skipped 8 large files in append mode for region 4621384810496(1076, 0), window 1774612800, max_size: 536870912
... (dozens more compaction lines for different windows) ...
2026-03-31T01:19:03Z raft_engine::purge: purged 9 expired log files for queue Append
# silence — OOMKilled at 01:19:13

Our questions: 1. Is 3–4.5 GiB per datanode expected for a low-traffic cluster with default config? This seems very high — a standalone single-node setup would fit in less memory than one of our datanodes is using. 2. Could there be a memory leak in TWCS compaction? The memory appears to grow monotonically without being released. 3. Are there any recommended settings to cap compaction memory or limit concurrent compaction tasks? We're running entirely on defaults. 4. Any known issues related to memory?

Could you export the result of the following SQL to us?

select * from information_schema.ssts_manifest where region_id = 4621384810496 order by min_ts;

This compaction is highly suspicious, please provide more info about present SST file list by following SQL above. Maybe attach the compaction logs to this thread if possible.

Here is the export

Could you enable these options for the datanode? https://github.com/GreptimeTeam/greptimedb/blob/233e35c0c91ba9859fa514511872834f6502cbd8/config/datanode.example.toml#L449-L456 We can set it to 500MB and fail-fast as a workaround. This may get rid of the OOM kill.

[[region_engine]]
[region_engine.mito]
# ....strip....
experimental_compaction_memory_limit = "500MB"
experimental_compaction_on_exhausted = "fail"
# ....strip....

Does this table has columns that store large texts or large blobs?

Yes, the average span stores roughly 50 KB, and the largest spans store ~1.2 MB per row. We'll apply the compaction memory limit:

[region_engine.mito]
experimental_compaction_memory_limit = "500MB"
experimental_compaction_on_exhausted = "fail"

Given the row sizes, is there any additional configuration we should consider?

Thanks for sharing. The data profile is helpful.

Is 500 MB enough for the compaction tasks to succeed?

In your workload, files are usually large enough. We observe that even a single row group in the file is nearly 500MB. We usually don’t need to merge them if a compaction takes more than 500MB.

Given the row sizes, is there any additional configuration we should consider?

Currently no. We can optimize the internal batch size and row group size in the database side for this workload. Now we always use a fixed row group size which isn’t the best size in some case. cc @Ning @shuiyisong

Right now this is what it looks like for our observability dashboard: https://github.com/8monkey-ai/hebo-platform/blob/main/apps/api/src/modules/traces/service.ts

Based on the data collected during the last 7 days, this is the span size distribution