http://starrocks.io logo
Join Slack
Powered by
Hi <@U0A71G31CDV> I'm trying to ingest huge data i...
# questions-and-troubleshooting
d
Hi @Rocky I'm trying to ingest huge data in StarRocks Shared Data on s3 We’re ingesting JSON logs from S3 via a custom Go loader that uses Stream Load against CN (compute node) HTTP on port 8040 (database 
starrocks_ingestion
, many tables). We’re trying to understand sustainable ingestion rate and where the bottleneck sits on the StarRocks side. What we’re able to ingest (rough numbers) With 3 CN nodes (is4gen.4xlarge) and a single loader (later tuned to high concurrency + larger batches), we saw on the order of ~20–22K rows/sec sustained in one configuration, with ~0.6M rows flushed per 30s in instrumentation windows. Batches for the dominant table are typically on the order of ~19K rows / ~32MB per stream load when 
max_bytes
is 32MB (row size ~1.5–2KB JSON lines). • When we pushed ~30 concurrent stream loads per CN (e.g. 90 concurrent from one loader across 3 CNs), all flush slots stayed full and average per-flush latency was very high (~120s per load in our stats), while loader CPU stayed low (load average ~2–3 on 16 vCPU loaders) and CN aggregate CPU also looked underutilized (e.g. load average ~3–4 on 16 vCPU CNs, uneven per-core usage). So end-to-end we are not obviously CPU-saturated on loader or CN in aggregate, yet stream load completion time is long and concurrency stacks up. Where we think we’re hitting limits CN-side processing / queuing of stream loads — High concurrent PUTs to 
_stream_load
with long wall-clock times; loader spends most time waiting on HTTP (semaphore / in-flight flushes), not on S3 or JSON parsing (S3 object processing ~4s/obj in our logs vs ~120s average flush time). 1. Earlier we saw connection failures when total concurrent loads across multiple loader instances was very high (broken pipe / connection reset by peer on all three CNs simultaneously), which looked like overload or aggressive connection teardown under too many parallel loads. 2. Increasing batch size to ~100MB did not improve rows/sec in our measurements — flushes got longer in proportion, so throughput plateaued; memory on loaders became the constraint before StarRocks improved. What we’ve tried (loader / infra) Tuned flush concurrency (CN-count–aware defaults), worker count, SQS pollers, HTTP client (timeouts, idle pools). Varied batch 
max_bytes
(32MB vs 100MB) and concurrency; reverted to 32MB when larger batches increased memory and didn’t raise sustained throughput. • Confirmed from the FE (MySQL): 
SHOW VARIABLES LIKE '%max_running_txn%'
returned empty; 
SHOW VARIABLES LIKE '%load%'
/ 
'%streaming%'
showed nothing that obviously caps concurrent loads at the FE layer (e.g. 
enable_query_queue_load
= false, 
load_mem_limit
= 0). can you help me with Guidance on CN/BE limits and recommended settings for many parallel stream loads, e.g. 
webserver_num_workers
/ 
be_http_num_workers
(and any related tablet writer / mem limits) for CN mode. Whether tablet / bucket count for hot tables could be causing serialization under concurrent loads to the same table, and how you’d recommend validating that (metrics / 
SHOW
commands). • Any official tuning checklist for “high parallel stream load into few CNs” (3 nodes today; we may add CN nodes to scale). We’re happy to share redacted loader logs (
[stats]
/ 
[flush]
lines), CN 
htop
, and 
cn.conf
/ 
be.conf
excerpts if useful.
r
Based on your description, you are likely encountering a combination of HTTP worker exhaustion on the CN and Metadata/Commit bottlenecks on the FE, which are common when scaling high-concurrency Stream Loads in StarRocks' Shared Data (Cloud-native) mode. Here is a breakdown of the guidance and tuning checklist for your scenario: 1. CN/BE Tuning for High Parallelism When running ~30 concurrent stream loads per CN, you are approaching the default limits of the HTTP and threading subsystems. * 
be_http_num_workers
(CN/BE config): Default is 48. This thread pool handles all incoming 
_stream_load
PUT requests. If you have 30 loaders + other monitoring/heartbeat traffic, this pool can saturate. * Recommendation: Increase to 128 or 256. * 
number_tablet_writer_threads
(CN/BE config): Default is half of CPU cores (in your case, 8). This is likely too low for 30+ concurrent loads. * Recommendation: Set this to 32 or 64 to allow more parallel writing tasks. * 
flush_thread_num_per_store
(CN/BE config): Since you are in Shared Data mode (S3), the "store" is the S3 cache/buffer. * Recommendation: Set to 32 (via 
update information_schema.be_configs set value = 32 where name = 'flush_thread_num_per_store'
). * Connection Resets / Broken Pipes: This often happens when the CN’s listen queue is full. * Check OS-level 
net.core.somaxconn
(often 128 by default) and increase it to 1024 or higher on CN nodes. * Also check 
net.ipv4.tcp_max_syn_backlog
. 2. FE-Side "Publish" Bottleneck (Shared Data Mode) In Shared Data mode, every Stream Load must commit to the FE, and the FE must "publish" a new version to the object storage metadata. With many small, concurrent loads, the FE becomes the bottleneck for version management. * 
lake_enable_batch_publish_version
(FE config): This is the most critical setting for your version of StarRocks (3.x). It allows the FE to group multiple transaction commits into a single metadata update. * Action: Ensure this is 
true
. (It is default in newer versions, but worth verifying). * Throttling due to Compaction: If your "flush latency" is high (120s), StarRocks might be intentionally slowing down ingestion because compaction cannot keep up with the number of small versions being created. * Check: 
SHOW FRONTEND CONFIG LIKE 'lake_ingest_slowdown_threshold';
(Default is 100 versions). * Metric: Look at 
lake_compaction_score
in metrics. If it exceeds 100, the FE will delay the 
commit
response to the loader. * Tuning: If you can tolerate slightly slower queries for higher ingestion, increase 
lake_ingest_slowdown_threshold
to 200 or 300. 3. Tablet and Bucket Serialization If many concurrent loads hit the same table, they are all competing to write to the same set of tablets. * Validation: Use Load Profile to see exactly where the 120s is spent.
Copy code
sql
    ALTER TABLE your_table_name SET('enable_load_profile' = 'true');
    -- Run a few loads, then:
    SHOW PROFILELIST;
    ANALYZE PROFILE FROM '<query_id>';
* Look for 
CloseWaitTime
. If this is the majority of the 120s, the delay is in the Commit/Publish phase (FE/Metadata), not the data transfer phase. * Bucket Count: For 20K rows/sec, if your bucket count is too low (e.g., 3-6 buckets), you get high contention. If it's too high (e.g., 100s of buckets for a small table), you create too many small files on S3, which slows down the FE's metadata task. * Recommendation: Target tablets to be roughly 1GB–10GB in size. For your throughput, aim for 30–60 buckets per partition to balance parallelism and metadata overhead. Tuning Checklist for High Parallel Ingestion Component | Parameter | Recommended Value CN (be.conf) | 
be_http_num_workers
| 
256
CN (be.conf) | 
number_tablet_writer_threads
| 
32
CN (be.conf) | 
flush_thread_num_per_store
| 
32
CN (be.conf) | 
load_process_max_memory_limit_percent
| 
50
(if CN is dedicated to load) FE (Global) | 
lake_enable_batch_publish_version
| 
true
FE (Global) | 
lake_ingest_slowdown_threshold
| 
200
(increases threshold for throttling) OS (sysctl) | 
net.core.somaxconn
| 
2048
Next Step: I highly recommend enabling the Load Profile for your dominant table. If 
CloseWaitTime
is high, focus on FE 
lake_
configs. If 
WriteDataTime
or 
RpcTime
is high, focus on CN worker threads and S3 throughput. References • administration/management/FE_configuration.md • knowledge/usage/load/shared_data_batch_ingestion.md • release_notes/release-3.2.md
d
Just to add we're using StarRocks 4.0.8 but same settings should work right? @Rocky
r
Yes, the previous settings are still foundational for StarRocks 4.0.8, but since you are on this version (which is even more advanced for ingestion), you have access to a "game-changer" feature specifically designed for your high-concurrency S3 ingestion: Merge Commit. In version 4.0.8, StarRocks can automatically merge multiple separate Stream Load requests into a single transaction. This drastically reduces the number of S3 versions created and alleviates the FE's metadata commit pressure. 1. The "Game-Changer": Enable Merge Commit In your Go loader, add these headers to your Stream Load 
PUT
requests. This will likely resolve your "high wall-clock time" issue by shifting the queuing from the CN's HTTP workers to a dedicated merging window. * 
enable_merge_commit: true
: Tells StarRocks to group this request with others. * 
merge_commit_interval_ms: 1000
: (Try 500ms to 2000ms) This is the "window" StarRocks uses to gather concurrent loads into one commit. * 
merge_commit_async: false
: If you want the HTTP response only after the data is fully committed/visible. (Set to 
true
for absolute lowest ingestion latency if you handle retries/confirmation separately). Why this helps: You mentioned 90 concurrent loads. Without Merge Commit, you create 90 transactions/versions every cycle. With a 1-second Merge window, those 90 loads might be committed as just 1 or 2 transactions, significantly speeding up the "Publish" phase (the bottleneck you suspect). 2. CN-Side Tuning (Revised for v4.0.8) Since you are seeing connection failures and high concurrency, adjust these on your 3 CN nodes: * HTTP Pool: * 
be_http_num_workers = 256
(Increases the number of threads available to hold open your ~90 loaders). * Writer Threads: * 
number_tablet_writer_threads = 32
(Increases parallelism for writing to S3). * S3 Specific: * 
flush_thread_num_per_store = 32
(Ensures enough threads are pushing data to S3). * Load Memory: * 
load_process_max_memory_limit_percent = 50
(Standard for load-heavy CNs). 3. FE-Side "Slowdown" Tuning (Shared Data) In Shared Data mode, if the version count grows too fast, the FE will "throttle" ingestion. This manifests as long 
CloseWaitTime
. * 
lake_ingest_slowdown_threshold
: Check this via 
ADMIN SHOW FRONTEND CONFIG LIKE 'lake_ingest_slowdown_threshold';
. If your table has high compaction scores, increase this to 200 or 300. * 
lake_enable_batch_publish_version = true
: (Verify this is 
true
in FE config). 4. How to Validate & Debug (New in 4.0.8) StarRocks 4.0.8 added better observability for these high-parallelism scenarios: 1. Observing Merge Commit: Check 
information_schema.loads
to see how many loads are being merged into single transactions.
Copy code
sql
    SELECT * FROM information_schema.loads WHERE TABLE_NAME = 'your_table' ORDER BY CREATE_TIME DESC;
2. Checking for Version Throttling: Check the compaction score for your hot table. If 
MaxCS
is high (e.g., > 100), the system is intentionally delaying your loads.
Copy code
sql
    SHOW TABLET STATUS FROM your_table;
    -- Look at the 'CompactionScore' column
3. Detailed Profile: As mentioned before, enable 
SET GLOBAL enable_load_profile = true;
. In 4.0.8, the profile is very detailed. If you see most time spent in 
CommitTask
, the bottleneck is FE/Metadata (use Merge Commit). If it's in 
Wait Mem
, your CN nodes need more memory or smaller batches. Recommended Next Steps: 1. Implementation: Update your Go loader to include 
enable_merge_commit: true
and 
merge_commit_interval_ms: 1000
. 2. Config: Increase 
be_http_num_workers
to 256 on CNs to prevent the "connection reset/broken pipe" errors under load. 3. OS: Ensure 
ulimit -n
is at least 65535 and 
net.core.somaxconn
is 2048 on the CN nodes. References
d
Hi @Rocky, Following your recommendations, we applied all the CN/FE/OS tuning and also enabled Merge Commit on our Go loader. Wanted to share what we observed. Setup: StarRocks 4.0.8, Shared Data on S3, 3 CN nodes (is4gen.4xlarge) Custom Go loader doing ~90 concurrent Stream Load PUTs across the 3 CNs • _JSON format, 
read_json_by_line: true
, batches typically ~19K rows / 32–100MB_ What we applied (all at once): _CN: 
be_http_num_workers=256
, 
number_tablet_writer_threads=32
, 
flush_thread_num_per_store=32
, `load_process_max_memory_limit_percent=50`_ _FE: 
max_running_txn_num_per_db=200
, 
lake_enable_batch_publish_version=true
, `lake_ingest_slowdown_threshold=200`_ _OS: 
net.core.somaxconn=2048
, `net.ipv4.tcp_max_syn_backlog=2048`_ • _Loader headers: 
enable_merge_commit: true
, 
merge_commit_interval_ms: 1000
, `merge_commit_async: false`_ _Issue 1 — 
merge_commit_parallel
is required but not obvious:_ _Initially we sent 
enable_merge_commit
, 
merge_commit_interval_ms
, and 
merge_commit_async
but did NOT send 
merge_commit_parallel
. Every single Stream Load failed with:_ Status: Fail Message: "Failed to write data to stream load pipe, num retry: 1, ... last error: Invalid argument: Batch write parallel must be set positive, but is null" _This error message doesn't mention 
merge_commit_parallel
by name, so it took us a while to figure out. After reading the STREAM LOAD docs more carefully we found 
merge_commit_parallel
is listed as required. Adding 
merge_commit_parallel: 3
fixed the error._ _Suggestion: Could the error message explicitly say "merge_commit_parallel header is required when enable_merge_commit is true"? The current "Batch write parallel must be set positive, but is null" doesn't point to the HTTP header name._ Issue 2 — Throughput regression with Merge Commit enabled: After fixing the parallel header, loads succeeded but throughput actually decreased compared to without merge commit: MetricWithout merge commit (old baseline)With merge commitRows/sec sustained ~20–22K ~15–16K Avg flush duration ~120s (bad, but stable) 40s → 50s → climbing toward 2m+ semWait (loader queue pressure) High but stable Exploding: 11s → 755s → 2,588s per 30s window Active flush slots 90 (all occupied) 90 (all occupied) Errors 0 Occasional (1 per 30s window) _It appears that with 
merge_commit_parallel=3
(matching our 3 CNs) and 90 concurrent loads, the merge pipeline becomes a serialization point — loads queue behind narrow merge windows instead of executing independently. The semaphore wait time exploding confirms loads are piling up._ Questions: _Is 
merge_commit_parallel
meant to be set equal to CN count, or higher? What's the recommended value for ~90 concurrent loads across 3 CNs?_ Should we reduce our loader concurrency when using merge commit (e.g. from 90 to 20–30), since the merge window batches them anyway? 1. _Is there a recommended ratio between 
merge_commit_parallel
, 
merge_commit_interval_ms
, and concurrent load count?_ 2. For our scenario (high concurrency, few CN nodes, large JSON batches), is merge commit expected to help, or is it designed more for many small loads rather than fewer large ones? For now we've disabled merge commit and are testing with just the CN/FE/OS tuning to establish a new baseline. Happy to share loader logs or run specific tests if that helps diagnose the merge commit behavior.
r
This is excellent feedback. You have hit a nuanced performance boundary where the characteristics of your data (large 32MB–100MB JSON batches) are clashing with the design goals of Merge Commit. Here is the breakdown of why you saw a regression and how to optimize for your specific "large batch" scenario. 1. The 
merge_commit_parallel
Bottleneck As you discovered, 
merge_commit_parallel
is a required header. However, there is a hard constraint you should know: * The Effective Limit: StarRocks limits this value to the number of available BE/CN nodes. The internal logic is 
min(merge_commit_parallel, num_be_nodes)
. * Why it slowed down: Since you have 3 CN nodes, your effective parallelism was 3. * Without Merge Commit: You had 90 independent streams using up to 32 writer threads per CN (total 96 potential writers). * With Merge Commit: You funneled 90 concurrent streams into a single merged transaction handled by only 3 parallel writers. * Result: You created a massive serialization point. The "exploding `semWait`" on your loader side confirms that your 90 Go routines were all queuing to enter a very narrow 3-lane pipe. 2. Is Merge Commit right for 32MB–100MB batches? Probably not. Merge Commit is specifically optimized for "High Concurrency, Small Batches" (e.g., thousands of loads/sec of 10KB–5MB each). * Your Batch Size: 32MB–100MB is already considered a "healthy" batch size for StarRocks. * The "Massive Transaction" Problem: If you merge 90 loads of 100MB each into a 1-second window, StarRocks tries to commit 9GB of data in a single transaction. In Shared Data mode, this creates a massive spike in S3 multipart uploads and metadata updates that can actually be slower than 90 smaller, independent transactions. 3. Revised Recommendation for Your Scenario Given your batch size and row count (~20K/sec), you are better off disabling Merge Commit and relying on the "independent" tuning you've already applied. To squeeze more performance out of your 3-CN cluster without Merge Commit: A. Optimization for 20K+ rows/sec * Loader Concurrency: Stick with high concurrency (90 is fine), but monitor the 
lake_compaction_score
. If it stays under 100, the FE is keeping up. * Batch Size: Keep it at 32MB–64MB. Going to 100MB increases the risk of "Long Tail" flushes where one slow S3 upload holds up the entire HTTP response. * FE Tuning (Crucial): _ 
lake_enable_batch_publish_version = true
(You've done this). This allows the FE to group the _metadata* of your 90 independent loads, which is the "Merge Commit" equivalent for the metadata layer. * 
lake_ingest_slowdown_threshold = 300
(Increase this slightly more if you see 120s latencies; it gives compaction more breathing room). B. If you must use Merge Commit (e.g., to reduce S3 costs/versions): If you want to use Merge Commit to reduce the number of versions, you must change your loader strategy: 1. Decrease Batch Size: Drop your loader's flush trigger to 2MB or 5MB. 2. Increase Loader Concurrency: Push 200+ concurrent small streams. 3. Set `merge_commit_async: true`: This will remove the 
semWait
on the loader side, as StarRocks will return 
OK
as soon as the data is received, not when it's committed. 4. Set 
merge_commit_parallel: 3
 (Stay at 3 since you have 3 CNs). 4. Direct Answers to your Questions 1. 
merge_commit_parallel
value? It should be equal to your CN count (3). Setting it higher has no effect currently. 2. Reduce loader concurrency with Merge Commit? No, you actually want high concurrency but much smaller batches per load. Large batches + Merge Commit = Memory/IO spikes. 3. Recommended Ratio? There isn't a fixed ratio, but the "sweet spot" for Merge Commit is generally a total merged size (Concurrency × Batch Size) of under 2GB per interval. Your current 90 × 100MB = 9GB is too large.
4. Suitability? Merge Commit is designed for "Small/Medium" loads. Your ~100MB batches are already "Large" and should be handled by standard Stream Load. Summary Checklist for 4.0.8 1. Turn off Merge Commit (Your batches are big enough). 2. Keep 
be_http_num_workers = 256
 and 
number_tablet_writer_threads = 32
. 3. Check 
SHOW PROC '/compactions';
 to see if compaction is the bottleneck. If 
permit
is low, you might need more CNs to handle the background merge work. 4. Error Message Feedback: I will pass the feedback about the 
merge_commit_parallel
error message to the engineering team. It should definitely be more descriptive. Final Tip: If you still see 120s flush times after disabling Merge Commit, check your Bucket Count. If you have 3 CNs, ensure your table has at least 18-24 buckets (6-8 per node) to ensure all CN cores can participate in the write. References • sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md • loading/StreamLoad.md • loading/Flink-connector-starrocks.md
d
Hi @Rocky, We’re ingesting NDJSON from S3 (SNS → SQS) with a Go loader that Stream Loads directly to CN private IPs (discovered from our CN ASG; FE is SQL/control only). We’ve spent a lot of time on the loader side: deterministic labels, bounded concurrency, aggressive HTTP timeouts, CN host round‑robin, merge commit off for our 32 MB batches, and 
[stats]
/ 
[flush]
metrics. Single-loader runs looked healthy (~3M rows / 30 s, ~2.3 s avg flush, merge commit off, 
semWait
~0) after CN/FE/OS tuning you’d suggested earlier (
be_http_num_workers
, writer/flush threads, 
lake_enable_batch_publish_version
, 
somaxconn
, etc.) and cleaning up toxic partitions / DEFAULT BUCKETS 24 on new partitions. What we’re hitting now is CN-side under multi-loader load. With three loader instances and default flush concurrency ≈ num_CNs × 30 (~90 per instance → ~270 concurrent stream loads across 3 CNs), we see: `cn.WARNING`: process/load memory limit exceeded, 
stream_load_executor
/ 
delta_writer
, 
NodeChannel
/ remote 
Backend: 10.11.0.99
in many lines, 
FragmentMgr
timeouts, async delta writer closed, etc. Loaders: 
write: broken pipe
bursts, 
[stats]
flush errors and 
semWait
up; 
ss
on CNs sometimes shows large send-q on :8040 ESTAB. At least one CN: 
dmesg
OOM kill of 
starrocks_be
on a ~92 GiB class host. • SQS: 
NumberOfMessagesDeleted
→ ~0 while 
NotVisible
spikes because we only delete after all table flushes succeed and visibility is 1800 s — so this is a symptom, not the root cause. Manual 
curl
stream load to a CN on 8040 from the loader subnet still returns 200 — connectivity and basic path are fine; this looks like sustained parallel load exceeding what the CNs can hold in memory / threads. Could you suggest a practical direction? Thanks — happy to share a short snippet of 
cn.WARNING
or loader 
[stats]
if useful.
r
The symptoms you are seeing—*CN OOM kills*, NodeChannel timeouts, and *broken pipes*—confirm that your 3-node cluster is being overwhelmed by the memory footprint of 270 concurrent stream loads. While 32MB per batch sounds modest, StarRocks' internal memory usage for an active Stream Load is a multiplier of the raw data. Here is why your 3-CN cluster is hitting a wall and how to fix it. 1. The "MemTable Explosion" (The Root Cause) When you have 270 concurrent loads (90 per CN), each load creates its own 
DeltaWriter
and 
MemTable
for every tablet it touches. * The Math: If your table has 24 buckets (8 per CN), and a loader instance sends a batch that hits all 8 local buckets: * 90 concurrent loads per CN × 8 buckets per table = 720 active MemTables per CN. * The default 
write_buffer_size
is 100MB. * Potential Memory Peak: 720 × 100MB = 72GB just for write buffers. * The Conflict: Your CN has 92GiB RAM. With 
load_process_max_memory_limit_percent = 50%
, the limit is roughly 46GiB. * The Result: StarRocks hits 46GiB, tries to trigger emergency flushes (backpressure), but with 720+ active writers competing for S3 upload bandwidth and CPU, it cannot flush fast enough. The memory continues to climb until the OS OOM-killer steps in. 2. Immediate Tuning Recommendations A. Reduce Concurrency (Crucial) 270 concurrent loads for 3 CN nodes is far beyond the "sweet spot" for performance. * Action: Reduce your loader concurrency. Try a total of 60–90 concurrent loads across the entire cluster (20–30 per CN). * Why: You will likely find that 60 concurrent loads with 32MB batches yield higher total rows/sec than 270 loads because the system spends less time context-switching and dealing with memory backpressure. B. Lower the 
write_buffer_size
 If you must maintain high concurrency, you must reduce the memory footprint of each writer. * Action (cn.conf): Set 
write_buffer_size = 33554432
(32MB) or even 
16777216
(16MB). * Trade-off: This will result in more frequent, smaller flushes to S3, which is slightly less efficient for S3 IO but prevents the OOM crash. C. Increase Memory Limits Safely Since these are dedicated CNs (is4gen), you can give the load process more of the available RAM. * Action (cn.conf): * 
mem_limit = 95%
* 
load_process_max_memory_limit_percent = 70
* 
enable_new_load_on_memory_limit_exceeded = false
(This will reject new loads instead of crashing the process when memory is full). 3. Fixing the "Broken Pipe" and NodeChannel Errors The 
broken pipe
and 
NodeChannel
errors are symptoms of CN Stalling. When memory is near the limit, JVM/C++ garbage collection and memory management cause "Stop the World" pauses. * The loader times out waiting for a response and closes the connection (
broken pipe
). * Internal RPCs between CN components fail because the threads are stuck waiting for memory allocation (
NodeChannel
timeout). 4. Revised Strategy for StarRocks 4.0.8 Setting | Recommended Value | Reason Loader Concurrency | 20-30 per CN | Prevents MemTable explosion and RPC timeouts. 
write_buffer_size
 | 32MB | Matches your batch size, reducing unused buffer overhead. 
load_process_max_memory_limit_percent
 | 70 | Dedicated CN nodes can afford more RAM for ingestion. 
default_load_mem_limit
 | 2GB | Caps memory for any single "runaway" load job. How to Scale Further If you need higher throughput than what 3 nodes can provide at 30 concurrent loads each: 1. Scale Out: Add more CN nodes. StarRocks scales linearly for ingestion. 6–9 CN nodes would handle your 270-load concurrency comfortably. 2. Increase Batch Size, Decrease Concurrency: If you increase your loader's batch size to 100MB but reduce concurrency to 10 per CN, you keep the same "data in flight" but drastically reduce the metadata and threading overhead on the CNs.
Summary: Your current setup is "over-threaded" for the memory available. By cutting concurrency by 60% and matching your 
write_buffer_size
to your batch size (32MB), you will likely achieve a stable, crash-free ingestion rate. References • loading/loading_introduction/loading_considerations.md • administration/management/BE_configuration.md • administration/management/resource_management/Memory_management.md • sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md
d
Hi @Rocky, We're running a shared-data (S3) cluster on StarRocks 4.0.8 and are experiencing slow ingestion that we've traced to excessive compaction memory consumption. We'd appreciate your guidance on optimal tuning for our workload. Cluster Configuration • FE: 1× 
r8g.xlarge
• CN: 4× 
is4gen.4xlarge
(16 vCPUs, 128 GB RAM, 2×7.5 TB NVMe) • Version: 4.0.8-2090bfc • Storage: S3 (shared-data / lake mode) • Warehouse: 
default_warehouse
with 1 cluster • Data Cache: 16 GB memory + 10.8 TB NVMe disk per CN Workload • 20 tables in a single database, continuous ingestion via Stream Load • Largest table: 
sxl4_ssp_non_blocking_queries
— 19.16 billion rows • ~450 tablets per CN • Read queries from Metabase (low concurrency, analytical) Problem — Compaction Consuming 40% of CN Memory With default settings (
lake_compaction_max_tasks = -1
, 
lake_compaction_interval_ms_on_success = 10000ms
), the adaptive compaction scheduler is consuming 27–33 GB of memory per CN (out of 79.2 GB MemLimit), even when no active loads are running: CN (BE_ID)Process MemCompaction MemCompaction %Load MemDisk IO1002353.3 GB27.1 GB50.8%0 MB1%1002455.9 GB28.9 GB51.7%0 MB3%1002558.3 GB31.3 GB53.7%3 MB2%54574457.3 GB32.5 GB56.7%7 MB2% Key observations: • Compaction is the dominant memory consumer (51–57% of process memory per CN) • Disk IO is only 1–3% — not IO bound • Flush pipeline is mostly idle (0–6/64 threads active, 0 queue depth) • Load memory is near zero — ingestion is starved of resources • Several recent queries show "Aborted" state in the profile list Relevant Configuration FE Config: • 
lake_compaction_max_tasks = -1
(adaptive) • 
lake_compaction_interval_ms_on_success = 10000
(10 seconds) • 
lake_compaction_score_selector_min_score = 10.0
• 
lake_compaction_score_upper_bound = 2000
CN Config: • 
max_compaction_concurrency = -1
(unlimited) • 
mem_limit = 95%
• 
load_process_max_memory_limit_percent = 70
• 
lake_compaction_chunk_size = 4096
Our Planned Mitigation We're planning to apply the following runtime changes and would appreciate your feedback on whether these values are appropriate for our workload:
Copy code
sql
ADMIN SET FRONTEND CONFIG ("lake_compaction_max_tasks" = "4");
ADMIN SET FRONTEND CONFIG ("lake_compaction_interval_ms_on_success" = "120000");
And per-CN:
Copy code
max_compaction_concurrency = 2
Questions Is the adaptive compaction algorithm (
lake_compaction_max_tasks = -1
) expected to consume this much memory for a 19B-row table, or is this a potential issue in the adaptive calculation? What are the recommended values for 
lake_compaction_max_tasks
and 
lake_compaction_interval_ms_on_success
for a high-throughput continuous ingestion workload? Is there a way to set a memory limit for compaction (e.g., 
compaction_mem_limit
or similar) rather than just task count? 1. Should we consider increasing 
lake_compaction_score_selector_min_score
beyond 10 to reduce compaction frequency? 2. Are there any other tunables we should be aware of for balancing ingestion throughput vs. compaction in shared-data mode?
r
This is a classic "High Churn" scenario in StarRocks Shared-data mode. In version 4.0.8, the adaptive compaction logic can indeed become overly aggressive for tables with massive row counts (19B+) because it focuses on reducing version counts for query performance, sometimes at the expense of memory stability. Here is the breakdown of your situation and recommendations: 1. Is the Adaptive Logic (
-1
) expected to consume this much? Yes. In StarRocks 4.0.8, setting 
lake_compaction_max_tasks = -1
calculates the limit as *16 * Number of CNs*. * With 4 CNs, your FE is attempting to run 64 concurrent compaction tasks cluster-wide. * For a 19 billion row table, each compaction task involves substantial metadata overhead (Bloom filters, bitsets, and column statistics) and merge-sort buffers. * If each task consumes ~500MB–1GB (common for massive tables), 64 tasks easily explain the 30GB+ memory footprint you are seeing. 2. Recommended Configuration Values Your planned mitigation is very much on the right track. For a continuous ingestion workload with 19B rows, a "slow and steady" approach to compaction is preferred to avoid starving the load pipeline. FE Configuration (Leader FE): * 
lake_compaction_max_tasks = 8
: (A bit more than your proposed 4, but much safer than 64). Start here and monitor the 
CompactionScore
. * 
lake_compaction_interval_ms_on_success = 60000
: (60 seconds). 2 minutes (120s) is also fine, but 60s provides a better balance for continuous ingestion. * 
lake_compaction_score_selector_min_score = 50.0
: Increasing this from 10.0 to 50.0 ensures the system doesn't waste resources merging just a few small versions; it will wait until there is significant "work" to do. CN Configuration (via 
UPDATE information_schema.be_configs
): * 
compact_threads = 2
: (The parameter is 
compact_threads
, not 
max_compaction_concurrency
). Setting this to 2 per CN (total 8 cluster-wide) will strictly limit the CPU and local memory footprint. * 
max_cumulative_compaction_num_singleton_deltas = 100
: Crucial. The default is often 500-1000. Reducing this to 100 limits how many files are merged in a single sub-task, which directly caps the memory spike per thread. 3. How to Set a Hard Memory Limit for Compaction You can explicitly cap the memory compaction is allowed to use on your CN nodes. In your 
cn.conf
(or via dynamic update): * 
compaction_max_memory_limit_percent = 20
: This ensures that even if the scheduler pushes too many tasks, the CN will throttle them once compaction hits 20% of the process memory (preventing it from ever reaching 50% again). 4. Summary of Recommended Tuning Apply these to stabilize your environment:
Copy code
sql
-- Apply to FE
ADMIN SET FRONTEND CONFIG ("lake_compaction_max_tasks" = "8");
ADMIN SET FRONTEND CONFIG ("lake_compaction_interval_ms_on_success" = "60000");
ADMIN SET FRONTEND CONFIG ("lake_compaction_score_selector_min_score" = "50.0");

-- Apply to all CNs
UPDATE information_schema.be_configs SET value = "2" WHERE name = "compact_threads";
UPDATE information_schema.be_configs SET value = "100" WHERE name = "max_cumulative_compaction_num_singleton_deltas";
UPDATE information_schema.be_configs SET value = "20" WHERE name = "compaction_max_memory_limit_percent";
5. Final Checklist for Shared-Data 4.0.8 * Aborted Queries: The "Aborted" state you saw was almost certainly caused by the CN process hitting its memory limit and the memory tracker preemptively killing queries or compaction tasks to prevent a full OOM crash. * Bucket Count: You mentioned 450 tablets per CN. This is a very healthy number. * Monitoring: Keep an eye on 
SHOW PROC '/compactions'
. If you see the 
MaxCS
(Max Compaction Score) climbing over 500 and staying there, you can slowly increase 
lake_compaction_max_tasks
by increments of 4. References
d
Hi @Rocky The 3 in-flight compaction tasks (started at 07:24 UTC, 9+ hours ago with NULL finish times in SHOW PROC '/compactions') aren't releasing memory even after setting lake_compaction_max_tasks=0. Is there a way to cancel in-progress compaction tasks on CNs? Or is 
compaction_mem_bytes
at 27-32 GB potentially a memory leak/fragmentation issue?
r
It sounds like you have hit a "stuck" state where the compute nodes (CNs) are holding onto memory allocated for massive merge-sort operations that are either progressing extremely slowly or have entered a semi-zombie state. In StarRocks 4.0.8, setting 
lake_compaction_max_tasks = 0
only prevents the FE from scheduling new tasks; it does not force-kill tasks that are already being executed by the CNs. Here is how to forcefully clear those tasks and recover your memory. 1. Manually Cancel the Stuck Tasks You can explicitly kill the in-flight tasks using their 
TXN_ID
(which you can find from your 
SHOW PROC '/compactions'
output). Run this on the Leader FE:
Copy code
sql
-- Replace <TXN_ID> with the actual ID from your SHOW PROC output
CANCEL COMPACTION WHERE TXN_ID = 12345;
* Note: This is an asynchronous command. After running it, check 
SHOW PROC '/compactions'
again. The task should eventually move to a finished state with an "Aborted" or "Cancelled" error message. 2. Check Real-Time Progress To see if they are actually "stuck" or just moving at a snail's pace due to the 19B row metadata, check the per-tablet progress:
Copy code
sql
SELECT * FROM information_schema.be_cloud_native_compactions 
WHERE TXN_ID = <TXN_ID> 
ORDER BY PROGRESS ASC;
If 
PROGRESS
has not changed for several hours, the task is effectively stuck. 3. Why is memory not being released? (Leak vs. Logic) At 27–32 GB, you are likely not looking at a "leak" in the traditional sense, but rather metadata and buffer retention for a massive table: * Massive Tablet Metadata: For a 19B row table, the bloom filters, zone maps, and bitsets (especially for Primary Key tables or tables with many columns) can be gigabytes in size. Compaction must load these into memory to perform the merge. * The "Dirty" Page Problem: Even if the task "finishes" internally, StarRocks uses a memory tracker. If the C++ allocator (TcMalloc) hasn't returned the pages to the OS, the memory may still appear "used" in system metrics, though 
compaction_mem_bytes
specifically tracks what StarRocks believes it is using. * 4.0.8 Specifics: Version 4.0.8 introduced "Spillable Compaction" for some scenarios, but if it's not triggered, the merge-sort for a 19B row partition can consume enormous amounts of RAM for the rowset readers. 4. The "Last Resort" Recovery Procedure If 
CANCEL COMPACTION
does not release the memory within 15–30 minutes, the CN threads may be stuck in an uninterruptible I/O or calculation loop. Use this sequence to safely reset: 1. Keep tasks disabled: 
ADMIN SET FRONTEND CONFIG ("lake_compaction_max_tasks" = "0");
2. Restart CN nodes one-by-one: Since this is a shared-data cluster, a CN restart is "cheap" because there is no data to re-balance. * Restarting the CN will terminate all local compaction threads and immediately release the 30GB+ of held memory. 3. Apply the stricter limits (from the previous recommendation): Before re-enabling compaction, ensure you have set: * 
compact_threads = 2
(on CNs) * 
max_cumulative_compaction_num_singleton_deltas = 100
(on CNs) * 
compaction_max_memory_limit_percent = 15
(on CNs) 4. Re-enable slowly: Set 
lake_compaction_max_tasks = 4
and monitor. Summary: 
CANCEL COMPACTION
is your primary tool. If the memory remains high after the task disappears from 
SHOW PROC
, a rolling restart of the CNs is the only way to clear the memory fragmentation/retention caused by these 9-hour-long tasks. References • administration/management/compaction.md • deployment/shared_data/feature-support-shared-data.md • faq/shared_data_faq.md • knowledge/admin_and_operation/shared_data_compaction.md
d
Hi @Rocky, Following up on excellent compaction diagnosis earlier today. We've applied all the recommended changes and confirmed the fix — here's a summary of what's done, and now we'd like your guidance on loader-side optimization. What We've Already Applied (2026-04-18) FE Config (runtime + fe-extra.conf) ParameterWasNow`lake_compaction_max_tasks`-1 (adaptive = 64 tasks)*8*`lake_compaction_interval_ms_on_success`10,000 ms*60,000 ms*`lake_compaction_score_selector_min_score`10.0*50.0*`max_running_txn_num_per_db`100*200*`lake_enable_batch_publish_version`truetrue (unchanged)`lake_ingest_slowdown_threshold`100*200* CN Config (runtime + cn-extra.conf) ParameterWasNow`compact_threads`auto*2*`max_cumulative_compaction_num_singleton_deltas`default (500-1000)*100*`compaction_max_memory_limit_percent`uncapped*15%* (in cn.conf, pending restart)`mem_limit`90%*95%*`load_process_max_memory_limit_percent`30*70*`write_buffer_size`104857600 (100 MB)*33554432 (32 MB)*`be_http_num_workers`48*256*`number_tablet_writer_threads`8*32*`flush_thread_num_per_store`4*32*`starlet_s3_max_concurrent_connections`16*128*`enable_new_load_on_memory_limit_exceeded`true*false* Results After Fix We cancelled 3 stuck compaction tasks (running 9+ hours) via 
CANCEL COMPACTION WHERE TXN_ID = ...
, then re-enabled at the throttled level: MetricBeforeAfterCompaction memory/CN27–33 GB (40–57% of process)*0 GB* (steady state TBD)Process memory/CN53–59 GB*19–23 GB*Available for ingestion/CN~10 GB*~55 GB* Current Cluster & Loader Setup • Cluster: 4 × 
is4gen.4xlarge
CNs (16 vCPU, 128 GB RAM / 79.2 GB MemLimit, 2× 3.75 TB NVMe each) • FE: 1 × 
r8g.xlarge
, StarRocks 4.0.8 • Loader: 2 × Go-based Stream Load instances (StarRocks SXL4 loader) • Ingestion rate: ~160K rows/s fleet-wide • Batch config: 32 MB (
max_bytes
), 60s 
max_age
, up to 80 concurrent loads per loader (fleet-wide peak ~160) • Hot table: 
sxl4_ssp_non_blocking_queries
(19.16B rows), 24 buckets, duplicate key, hash on 
sha256
• Merge Commit: Disabled (previously tested, slowed 100 MB / high-concurrency workload) Questions — Loader-Side Optimization Now that compaction is throttled, we want to optimize the loader side to reduce compaction pressure at the source. 1. What is the dominant compaction cost — version count or rowset size? With 32 MB batches at ~160 concurrent loads, we create roughly 8 versions/second per hot tablet fleet-wide. Would moving to 64 MB or 100 MB batches meaningfully reduce compaction load by halving the version creation rate, or is the bottleneck elsewhere (compaction thread count, base/cumulative scheduling, S3 I/O for merged rowsets)? 2. Recommended batch size for this setup? Given our CN fleet (4 × 79.2 GB MemLimit) and hot-table bucket count (24), is there a batch-size sweet spot you recommend? We previously observed that 100 MB batches increased flush duration without improving throughput (on a 3-CN cluster, before the compaction tuning applied today), so we're cautious about jumping to 100 MB without confirmation. Note that 
write_buffer_size
is currently 32 MB — see question 5. 3. Is our current tablet fan-out (24 buckets, 6 buckets/CN) appropriate? Your earlier guidance suggested 6–8 buckets/CN for hot tables. With 4 CNs we're at 6/CN. Should we go to 8 × 4 = 32 buckets for the hottest tables, or is that unnecessary given the compaction fix already resolved the primary bottleneck? 4. Any additional CN-side compaction knobs we should tune? With the changes listed above already applied (
compact_threads=2
, 
max_cumulative_compaction_num_singleton_deltas=100
, 
compaction_max_memory_limit_percent=15
, 
lake_compaction_max_tasks=8
), are there any other compaction parameters we should adjust for our fleet size and ingestion rate? Specifically: • 
cumulative_compaction_num_threads_per_disk
(currently default: 1) 
base_compaction_num_threads_per_disk
(currently default: 1) • Should we raise 
lake_compaction_max_tasks
back to 12 or 16 once compaction memory stabilises under the 15% hard cap? 5. If we go to 64–100 MB batches, does 
write_buffer_size
need to increase? Currently set to 32 MB (matching our current batch size, reduced from 100 MB to prevent OOM at high concurrency). If we increase batch size to 64 MB, should 
write_buffer_size
also be raised to 64 MB? Or is the relationship different in shared-data mode where the MemTable flushes to S3? 6. How can we observe compaction backlog operationally? We're currently using: • 
SHOW PROC '/compactions'
— task history and stuck task detection • 
<http://information_schema.be|information_schema.be>_metrics
— 
compaction_mem_bytes
, 
load_mem_bytes
, 
process_mem_bytes
Is there an FE metric or system table we can query to see: • (a) compaction queue depth per table • (b) compaction throughput per CN • (c) per-tablet version-count distribution This would let us measure whether a loader-side change is actually reducing compaction work. 7. Would Merge Commit help at smaller batch sizes? We disabled Merge Commit after it slowed our 100 MB / high-concurrency workload (your earlier guidance matched our measurements). If we moved to smaller batches at higher concurrency (e.g. 5–10 MB × many flushes), would Merge Commit plus 
merge_commit_parallel = num_CNs
be a better strategy for reducing version creation, compared to the "bigger batches" path? Our Proposed Plan Given the compaction throttling is now in place, here's our loader-side plan: StepChangeExpected Impact*Step 1*Raise 
max_bytes
to 64 MB, 
max_age
to 120s, keep flush concurrency at 20/CN~50% fewer versions created, moderate batch size*Step 2* (if needed)Raise to 100 MB, drop flush concurrency to 15/CN, raise 
write_buffer_size
to 64 MBFurther version reduction, needs memory validation We'd appreciate a sanity check on this plan, plus any additional CN-side tuning we should bundle with it now that the compaction throttling is in place. Thanks! Dhruv
r
Great to see the compaction memory has stabilized and you've recovered ~45 GB of RAM per CN. Those results look much healthier. For your loader-side optimization, here is a detailed breakdown based on your 160K rows/s throughput and StarRocks 4.0.8 architecture. 1. The Dominant Cost: Version Count vs. Rowset Size In StarRocks Shared-data (Lake) mode, version count is the primary driver of compaction overhead. Every Stream Load transaction creates one new Version (Rowset). At 8 versions/second per tablet, you are creating ~28,800 versions per hour. Compaction must merge these "deltas" into larger files to maintain query performance. * The Rowset Size (e.g., 32 MB vs 100 MB) affects the amount of data written to S3, but the metadata management (version pointers, Bloom filters) scales with the number of versions. * Recommendation: Prioritize reducing version count (larger batches) over rowset size optimization. 2. Recommended Batch Size: The 100 MB "Sweet Spot" The slowness you previously observed with 100 MB batches was likely caused by the "Writer Thread Bottleneck" you've since fixed (by increasing 
number_tablet_writer_threads
, 
flush_thread_num_per_store
, and 
starlet_s3_max_concurrent_connections
). * Action: Move toward 100 MB batches. * Why: A 100 MB batch at your throughput will reduce your version creation rate by 3x compared to 32 MB. With your tuned CNs, 100 MB batches should now flush efficiently to S3 without the previous latency spikes. 3. Fan-out: Is 24 Buckets Appropriate? Your current 6 buckets/CN is adequate, but for a 19B row table on 16-vCPU nodes, 32 buckets (8 per CN) is the architectural "sweet spot." * Recommendation: For new partitions or table recreations, use 32 buckets. It aligns better with your 16-vCPU count and allows for higher query parallelism without fragmenting the data too much. However, do not re-partition the existing 19B row table yet; the current 24 is "good enough" given the other fixes. 4. Additional CN-side Tuning * 
compact_threads = 2
: Keep this. It limits the CPU/RAM "blast radius" per CN. * 
lake_compaction_max_tasks
: Once your memory remains stable for 24 hours, you can safely increase this to 12 or 16 (cluster-wide). This will help clear the "MaxCS" (Compaction Score) faster. * Base vs. Cumulative: In Lake mode, the distinction is less critical than in local storage because the FE handles the scheduling. Keep the defaults for 
threads_per_disk
. 5. 
write_buffer_size
Relationship Yes, you should match them. * The Logic: If you send a 100 MB batch but 
write_buffer_size
is 32 MB, the CN will flush 3-4 internal "segment files" to S3 for a single transaction. While this is still one "version," it creates more small-file IOPS in S3 and slightly more work for future compaction. * Recommendation: If you go to 64 MB batches, set 
write_buffer_size = 67108864
(64 MB). If you go to 100 MB, set it to 
104857600
(100 MB). 6. Observing Compaction Backlog Operationally Use these queries to measure the impact of your loader changes: * Check the Backlog (Compaction Score):
Copy code
sql
    -- High CANDIDATE_MAX_SCORE (>100) indicates the loader is creating versions faster than compaction can merge them.
    SELECT BE_ID, CANDIDATE_MAX_SCORE, LATEST_COMPACTION_SCORE
    FROM information_schema.be_compactions;
* Check Per-Tablet Version Distribution:
Copy code
sql
    -- Look for tablets with high VERSION counts (e.g., > 500)
    SHOW TABLET FROM sxl4_ssp_non_blocking_queries;
* Compaction Throughput: Monitor 
lake_compaction_output_bytes_total
in your Prometheus/Grafana dashboard to see how many GBs/sec the cluster is actually merging. 7. Merge Commit vs. Big Batches Stick with Big Batches.
At 160K rows/s and 32-100 MB batches, you are already in a "bulk load" regime. Merge Commit is designed for the opposite problem: many small clients sending 10 KB–5 MB chunks. For your high-throughput Go loader, simply increasing the 
max_bytes
on the loader side is more efficient and provides deterministic control over transaction boundaries. ────────── Final Sanity Check on Your Plan Your proposed plan is excellent. I suggest a slight modification to Step 2: * Step 1: Raise 
max_bytes
to 64 MB, 
write_buffer_size
to 64 MB. Monitor 
be_compactions.CANDIDATE_MAX_SCORE
. * Step 2: If 
CANDIDATE_MAX_SCORE
stays high, raise 
max_bytes
to 100 MB and 
write_buffer_size
to 100 MB. Simultaneously reduce flush concurrency from 20/CN to 10-12/CN to keep the total "memory in flight" stable. This "Bigger & Fewer" approach is the most sustainable way to ingest 19B+ rows into a shared-data cluster. References • administration/management/compaction.md • loading/loading_introduction/loading_considerations.md • sql-reference/information_schema/be_compactions.md • administration/management/BE_configuration.md • knowledge/usage/load/shared_data_batch_ingestion.md References • knowledge/usage/load/shared_data_batch_ingestion.md • loading/StreamLoad.md • loading/Flink-connector-starrocks.md
d
@Rocky The 3 cancelled compaction tasks (TXN_IDs 1125705, 1125742, 1125756) still show NULL finish times in 
SHOW PROC '/compactions'
and are blocking new task scheduling. 
compaction_mem_bytes = 0
on all CNs confirms the CN-side tasks are dead, but the FE still tracks them as running. We reduced 
lake_compaction_default_timeout_second
to 60 and toggled 
lake_compaction_max_tasks
between 0→16, but no new tasks are being scheduled. Meanwhile, today's partition has a compaction score of 43,111 with 49,472 uncompacted versions. How do we clear the FE's stale task tracker?
r
This situation indicates that the FE's 
LakeCompactionScheduler
 has a stale in-memory reference to these tasks. In StarRocks Shared-data (Lake) mode, the FE limits compaction to one task per partition at a time to prevent version conflicts. Because those 3 tasks are associated with your hot partition and show 
NULL
finish times, the FE believes the partition is "already being compacted" and refuses to schedule the new work required to fix that 43,111 score. Since 
compaction_mem_bytes
is 0 on the CNs, the execution is dead, but the FE metadata state is out of sync. Here is how to force-clear this and kickstart the 43k score cleanup: 1. Identify the Limbo State Check the actual transaction status for those three TxnIDs:
Copy code
sql
SHOW TRANSACTION WHERE id IN (1125705, 1125742, 1125756);
* If State is 
ABORTED
: The FE metadata has simply failed to remove the task from the scheduler's "running" set. * If State is 
COMMITTED
: The task is stuck in the Publish phase. Check the Leader FE log (
fe.log
) for the string 
publish version
and the 
TxnID
to see if it's failing to reach the CNs. 2. The "Hard Reset" for FE Scheduler (Recommended) Since toggling 
lake_compaction_max_tasks
and 
CANCEL COMPACTION
failed to clear the 
NULL
entries, the most reliable way to clear the FE's in-memory "running tasks" set is to restart the Leader FE. * Is it safe? Yes. In Shared-data mode, restarting the FE is safe. The Leader will hand over to a Follower (or restart itself), and the 
LakeCompactionScheduler
will re-initialize with a clean state. It will see the 43k score and immediately begin scheduling fresh tasks. * Procedure: 1. Identify Leader FE: 
SHOW FRONTENDS;
2. Restart the Leader FE process. 3. Immediate Recovery After Reset Once the FE is back and the stale tasks are gone from 
SHOW PROC '/compactions'
, the 43k score needs aggressive handling. Because the version count is so high (~50k versions), the first few compaction rounds will be massive. Apply these "Recovery Mode" settings temporarily:
Copy code
sql
-- Allow more tasks to handle the 50k version backlog
ADMIN SET FRONTEND CONFIG ("lake_compaction_max_tasks" = "16");
-- Reduce the min score so it stays aggressive
ADMIN SET FRONTEND CONFIG ("lake_compaction_score_selector_min_score" = "5.0");
-- Ensure the timeout is reasonable for a massive merge (increase from 60 to 3600)
ADMIN SET FRONTEND CONFIG ("lake_compaction_default_timeout_second" = "3600");
4. Why did this happen? In version 4.0.8, if a compaction task involves a massive number of versions (like your 49k), the Commit/Publish phase can timeout at the FE level while the CN is still technically trying to finalize. If the 
CANCEL
was sent during this transition, the FE state machine can occasionally enter the "limbo" state you are seeing. Summary of Actions: 1. Check 
fe.log
 for those 
TxnIDs
to see if there is a "Publish" or "Commit" error. 2. Restart the Leader FE to wipe the stale in-memory scheduler state. 3. Monitor 
SHOW PROC '/compactions'
 immediately after restart. You should see new tasks for the hot partition starting within 1-2 minutes. 4. Verify progress via 
CANDIDATE_MAX_SCORE
in 
information_schema.be_compactions
. If the score starts dropping, your loader-side optimizations (bigger batches) are working. References • administration/management/compaction.md • faq/shared_data_faq.md • sql-reference/sql-statements/data-manipulation/CANCEL_COMPACTION.md • knowledge/admin_and_operation/shared_data_compaction.md References
d
Subject: Shared-data 4.0.8 — compaction stuck, memory leak after task kill, ingest backpressure causing FE txn-cap rejections Hi @Rocky, We're running StarRocks 4.0.8 in shared-data mode (S3) and would appreciate guidance on a few related issues we've hit during high-concurrency Stream Load ingestion. Cluster topology 1 FE: 
r8g.xlarge
(4 vCPU / 32 GB), running in Docker 4 CN: 
is4gen.4xlarge
(16 vCPU / 94 GB / 2× NVMe), running in Docker on Amazon Linux 2023, kernel 6.1, aarch64 _Object storage: S3 (eu-west-2), `aws_s3_use_instance_profile = true`_ • Image: 
starrocks/cn-ubuntu:4.0.8
(
4.0.8-2090bfc
) Workload _2 Go loaders, total ~90 concurrent Stream Load streams into a single database (
starrocks_ingestion
, 20 tables, 70 partitions, 1680 tablets total)._ NDJSON, 64–100 MB per batch (per docs recommendation). • _Hot table: 
sxl4_ssp_non_blocking_queries
— duplicate-key, partitioned by 
date_trunc('day', timestamp)
, 24 buckets, hashed by 
sha256
. ~1.7 TB / day in the hot partition (
p20260418
), ~70 GB per tablet._ Tuning already applied (in 
fe-extra.conf
/ 
cn-extra.conf
): # FE _max_running_txn_num_per_db = 2000 (raised from 200)_ _lake_compaction_max_tasks = 32 (raised from 16)_ _lake_compaction_score_upper_bound = 1000000 (kept high temporarily)_ _lake_compaction_interval_ms_on_success = 5000_ _lake_ingest_slowdown_threshold = 10000 (was 200)_ _lake_enable_batch_publish_version = true_ _publish_version_max_threads = 1024_ # CN _mem_limit = 95%_ _load_process_max_memory_limit_percent = 70_ _write_buffer_size = 32 MiB_ _number_tablet_writer_threads = 32_ _flush_thread_num_per_store = 32_ _starlet_s3_max_concurrent_connections = 128_ _be_http_num_workers = 256_ _compact_threads = 12 (raised from 4)_ _base_compaction_num_threads_per_disk = 2 (raised from 1)_ _cumulative_compaction_num_threads_per_disk = 4 (raised from 1)_ _update_compaction_num_threads_per_disk = 2 (raised from 1)_ _max_cumulative_compaction_num_singleton_deltas = 100 (was 500)_ _default_load_mem_limit = 2 GiB_ _enable_new_load_on_memory_limit_exceeded = false_ Issue 1 — compaction tasks stuck for hours _Two compaction tasks on 
p20260418
started at 
03:46:58
/ 
03:47:00
and remained with 
CommitTime = NULL
indefinitely until manually cancelled. This is the second time we've seen this — yesterday three tasks were stuck > 9 hours and we had to kill them. After kill, the partition's MAX_CS climbed to 122,841 despite the scheduler dispatching new tasks (visible in 
SHOW PROC '/compactions'
with 
in_queue_sec
between 9,000 and 22,000 seconds)._ Issue 2 — compaction-memory leak after killing tasks _Following the kill of stuck tasks, one CN (
10.11.0.191
, BE_ID 545744) showed:_ _starrocks_be_compaction_mem_bytes = 55,826,681,608 (~52 GB)_ _starrocks_be_running_*_compaction_task_num = 0_ _lake_aggregate_compaction_failed_tasks = 8_ Process RSS was 71 GB / 79 GB MemLimit — close to OOM. Restarting the CN container immediately released the 52 GB. The other three CNs were under 5 GB compaction memory at the same time. Two questions: _Is there a known leak path for 
starrocks_be_compaction_mem_bytes
on shared-data CNs after 
CANCEL COMPACTION
or task abort?_ • _Why is the BE-side 
starrocks_be_compaction_mem_bytes
metric (which I'd expect to be shared-nothing only) accumulating in shared-data mode at all?_ Issue 3 — FE txn-cap rejection avalanche _While 
p20260418
was at MAX_CS ≈ 100k+, every Stream Load to that table held its txn slot far longer than usual (publish latency p99 = 1.19 s, but with thousands queued: 
lake_publish_tablet_version_queuing_count = 31523
). The 200-default 
max_running_txn_num_per_db
filled instantly and 
loadTxnBegin()
started rejecting all loads for the entire DB at ~1000/sec:_ begin transaction failed, errmsg=current running txns on db 10305 is 200, larger than limit 200 Raising it to 2000 mitigated it, but the underlying coupling (one bad partition stalls the entire DB's ingest) seems unsafe by design. Is there a recommended pattern for protecting healthy tables when one partition's compaction falls behind? E.g. per-table txn caps, or auto-shedding only the affected partition? Asks _Recommended 
compact_threads
/ per-disk thread settings for 
is4gen.4xlarge
-class nodes (16 vCPU, 2 NVMe local cache, 94 GB) running in shared-data mode under sustained 64–100 MB Stream Load batches._ Confirmation whether the ~50 GB compaction-memory accumulation we observed is a known leak in 4.0.8. _Best practice to prevent one partition's compaction backlog from blocking the entire DB via 
max_running_txn_num_per_db
._ 1. _Should 
lake_ingest_slowdown_threshold
and 
lake_compaction_score_upper_bound
ever both be set on the same cluster, or are they mutually exclusive strategies? Our partition reached MAX_CS = 122k under the defaults of these._ Happy to share the full 
fe.warn.log
, 
cn.WARNING
, 
SHOW PROC '/compactions'
output, and the 
CREATE TABLE
of the hot table on request. Thanks!
r
Hello! It looks like you're pushing the limits of StarRocks' shared-data (Lake) mode with a high-throughput ingestion workload. Reaching a Max Compaction Score of 122k is quite an achievement—but definitely points to a critical imbalance between ingestion speed and compaction throughput. Here is the diagnosis and recommended tuning for your StarRocks 4.0.8 cluster on 
is4gen.4xlarge
nodes. 1. Recommended 
compact_threads
and Tuning for 
is4gen.4xlarge
 Your current setting of 
compact_threads = 12
on a 16-vCPU node is too aggressive. In StarRocks, compaction threads compete with ingestion (load) threads and query threads for CPU and memory. * Recommendation: Set 
compact_threads = 4
. The general rule for production is 25% of vCPU count. * Why? When you set this to 12 (75%), you risk "thread starvation" where the OS spends more time context-switching than merging data. This can actually lead to the "stuck" state you observed, as threads might hang waiting for I/O or memory allocations that are being contended for by 90+ concurrent loads. Recommended CN Config adjustments:
Copy code
ini
# CN (be.conf / runtime)
compact_threads = 4
max_cumulative_compaction_num_singleton_deltas = 100 
# Reduce memory pressure per task
compaction_max_memory_limit_percent = 15
# Maintain these ingestion-optimized settings
number_tablet_writer_threads = 32
starlet_s3_max_concurrent_connections = 128
2. Confirmation: The 52 GB Compaction Memory Leak Yes, what you are seeing is almost certainly a memory tracker leak in 4.0.8. * The Cause: In shared-data mode, even though data is in S3, the CN must pull "Rowset" metadata and data blocks into memory to perform the merge sort. 
starrocks_be_compaction_mem_bytes
tracks the memory allocated for these readers. * The Leak Path: In version 4.0.x, if a compaction task is manually cancelled via 
CANCEL COMPACTION
or aborted due to an error during a specific phase (like the "Publish" phase), the MemTracker sometimes fails to decrement. The memory may actually be released back to the system allocator (jemalloc), but the internal StarRocks tracker still thinks it is "in use," which eventually causes new loads or tasks to be rejected because the CN believes it is near its memory limit. * Fix: This is a known issue that usually requires a CN restart to clear. To minimize occurrences, avoid manual cancellations when possible and reduce the 
compact_threads
to prevent tasks from entering a "stuck" state due to resource contention. 3. Preventing Ingest "Avalanche" (Isolation Strategy) The 
max_running_txn_num_per_db
is a blunt instrument. When one hot partition's publish latency spikes (due to the metadata overhead of managing 122k uncompacted versions), it holds onto a transaction slot. Because you have 90 concurrent streams, you fill the 200 slots instantly. * Tiered Thresholds (Slowdown vs. Upper Bound): These are not mutually exclusive; they are intended to be used together. * 
lake_ingest_slowdown_threshold
(e.g., 100): At this score, the system injects a delay into the 
commit
phase of every load to that partition. This is your first line of defense to give compaction time to catch up. * 
lake_compaction_score_upper_bound
(e.g., 2000): At this score, the partition stops accepting new data. * Best Practice: 1. Set 
lake_enable_ingest_slowdown = true
. 2. Lower your 
lake_ingest_slowdown_threshold
back to a more reasonable value (e.g., 200 or 500). Keeping it at 10,000 allowed the backlog to reach 122k, which is why your publish phase is now so slow that it's exhausting the DB-wide transaction cap. 3. Set 
lake_compaction_score_upper_bound = 5000
. This will reject loads only to the broken partition, allowing the other 19 tables in the DB to continue ingesting. 4. How to recover the 122k Backlog To clear a score that high, the FE needs to be very aggressive.
1. Stop Ingestion to that table temporarily if possible, or use the 
lake_compaction_disable_tables
command to pause others and focus all resources on the hot table. 2. Aggressive FE Scheduling:
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("lake_compaction_max_tasks" = "64"); -- Let FE dispatch more
    ADMIN SET FRONTEND CONFIG ("lake_compaction_interval_ms_on_success" = "1000");
3. Check 
be_cloud_native_compactions
: Monitor the 
PROGRESS
column. If a task is at 0% for 30 minutes, the CN is likely stuck. Summary of Action Plan: 1. Lower 
compact_threads
to 4 on all CNs. 2. Set 
lake_ingest_slowdown_threshold = 500
 to trigger backpressure earlier and protect the transaction pool. 3. Restart the CN with the 52GB "leaked" tracker to restore its ability to accept tasks. 4. Reduce your Go loader concurrency slightly until the MaxCS drops below 1,000. Large batches (100MB) are good, but 90 concurrent streams might be creating versions faster than 4 CNs can merge them. If the tasks continue to show 
CommitTime = NULL
for hours after these changes, please check the CN 
WARNING
log for "S3 timeout" or "Slow I/O" errors, which often indicate that the NVMe cache is saturated. References • deployment/shared_data/feature-support-shared-data.md • administration/management/compaction.md • faq/shared_data_faq.md • knowledge/admin_and_operation/shared_data_compaction.md
d
Thanks @Rocky, very useful — especially confirmation of the compaction MemTracker leak. Two follow-ups before we apply the rest: 1. _`compact_threads`: our CN CPU has been sitting at 7–30% throughout this incident, even with 12 compaction threads running, so we're not seeing the contention you describe. We're keeping it at 12 until MAX_CS drops below 1,000, then dropping to 8 (not 4) — our load:compaction ratio is unusually high (≈90 concurrent streams), which I think shifts the "25% of cores" rule. Does that sound reasonable?_ _Recovery sequencing: rather than apply 
lake_compaction_score_upper_bound = 5000
now (which would hard-reject loads on the partition that's at MAX_CS = 123k), we plan to keep it at 1,000,000, pause loaders to that one table for 30–60 min, and let compaction drain it. Once MAX_CS < 5,000 we'll set the upper bound to 5,000 as you suggested. Any concern with that order of operations?_ On the leak itself — can you share the issue/PR number for the MemTracker fix in 4.0.x? We'd like to know which patch version (or 4.1) we'd need to upgrade to in order to make 
CANCEL COMPACTION
safe again. 1. _Re: the 
max_running_txn_num_per_db
coupling — is per-table txn capping on any roadmap? In our case one bad partition stalled all 20 tables in the DB, which is a brittle failure mode for high-throughput shared-data deployments._
r
Those are well-reasoned follow-up points. Your observations about idle CPU while compaction is backlogged suggest your bottleneck is likely S3 I/O or memory bandwidth, rather than raw compute. 1. `compact_threads`: 8 vs 4 threads Your logic for keeping 8 threads (50% of vCPUs) is sound for your specific load-to-compaction ratio. * The reasoning: Since your CPU is sitting at <30%, you are clearly not CPU-bound. However, the reason I recommended 4 (25%) was to limit the memory tracker's footprint and S3 connection concurrency. Each compaction thread on a 16-vCPU node can hold several GBs of metadata and buffers in flight. * Recommendation: If you stay at 8 threads, ensure you monitor 
starlet_s3_max_concurrent_connections
(currently 128). If compaction tasks start showing "S3 slow I/O" or "Connection timeout" in the CN logs, you’ll need to either drop the thread count or raise that connection limit to 256. 2. Recovery Sequencing Sanity Check Your plan is approved. In fact, it is superior to a hard-rejection approach. * Why it works: Pausing the loader manually gives the cluster a "compaction-only" window. Setting the 
lake_compaction_score_upper_bound
to 1,000,000 effectively removes the safety rail, but since you are manually controlling the loader, you don't need the rail. * Precaution: Before you resume, ensure 
CANDIDATE_MAX_SCORE
is below 2,000. Once it's healthy, applying the 
upper_bound = 5000
as a "permanent" safety rail will prevent another 120k avalanche in the future. 3. MemTracker Leak PR & Fix Version The specific memory tracker issue you described (leaked bytes after task kill/abort in shared-data) was addressed in the 4.0.9 and 4.1.0 releases. * Relevant Fixes: * PR #71126: Fixed a memory leak caused by 
shared_ptr
cycles in background contexts (FetchTaskContext). * PR #68877: Introduced "Fast Cancel" for Lake DeltaWriter, which improves resource cleanup when transactions are aborted. * Upgrade Recommendation: You are on 
4.0.8
. Upgrading to 4.0.9 (released mid-April 2026) or moving to 4.1.0 will resolve the tracker inconsistency and make 
CANCEL COMPACTION
safe to use without requiring a CN restart. 4. Per-Table Transaction Capping (Roadmap) Currently, StarRocks does not have a native 
max_running_txn_num_per_table
setting. The coupling you experienced—where one "bad" partition exhausts the database's 2,000 txn slots—is a known limitation of the current transaction manager. * Workarounds: * Database Isolation: Move the "hottest" table to its own dedicated StarRocks database. Since the limit is 
per_db
, this creates a hard circuit breaker that prevents that table from stalling the other 19 tables. * 4.1.0 Feature: The new "Multi-Tenant Data Management" in 4.1.0 introduces better resource isolation for shared-data, though it focuses more on IOPS/CPU than transaction slot isolation. * Future: Per-table/per-partition transaction quotas are a frequent request for high-scale shared-data users and are being discussed for the 4.2+ roadmap to prevent exactly the "blast radius" issue you hit. Final Summary for your Plan: 1. Compact Threads: Set to 8 (monitored). 2. Drain Phase: Pause loaders, let MaxCS fall < 2k. 3. Safety Rail: Apply 
upper_bound = 5000
and 
slowdown_threshold = 500
. 4. Patch: Target v4.0.9 to fix the MemTracker leak and allow safe task cancellation. References • loading/SQL_transaction.md • faq/loading/Loading_faq.md • administration/management/BE_configuration.md • administration/management/FE_configuration.md • sql-reference/System_variable.md
Open in Slack
PreviousNext
Powered by