Hello 👋 I am working on memory estimation accuracy for compaction as a follow-on to PR #7305. Does this align with the v1.1 compaction optimization work or the v1.2 ARM phase

Hi Gbolahan, the resource management will be our primary work around the release of GreptimeDB v1.0. Please feel free to share your thoughts about that aspect in issue, discussion or this channel.

The estimation_compaction_bytes just sums max_row_group_uncompressed_size, which only accounts for input_buffer, ignoring output buffer and Arrow overhead. I benhmarked the actual rss delta across different schemas to see if I can find a right multiplier, but honestly, the data was too noisy to get a constant. So instead of picking a magic number, I do think we need to physically hold: 1. The input row group in memory 2. the output row group buffer 3. Arrow/parquet page buffers and intermediate dictionary structures This gives a baseline of 3x However, to make this truly adaptive, we need to be schema-aware. Basically extract schema at compaction time, and apply granular factors i.e ~2.5x for dense float64, scaling up to ~4-5x for high-cardinality string tags

@Gbolahan Thanks for sharing your findings. Did you measure memory consumption during compaction? Compaction is essentially an N-way external merge sort, so overlap among input files may also affect peak memory usage. While this isn’t a major concern, we can still refine our estimation step by step.

My benchmark actually only measured flushing a single batch, not a true N-way compaction merge. My assumption was completely way off. I think I have an idea how to go about it better now. Thanks for pointing that out