@Rocky We're running a StarRocks shared-nothing cluster, 3 BE pods, 2 CPU cores / pod limit on Kubernetes). Current version: 3.5.7-d2da301. We're seeing two recurring problems: 1. BE pods exceed their CPU limit (100–170% of a 2-core limit) during ~2h windows, caused by routine load ingestion (~2,000–3,500 rows/sec) running concurrently with background compaction 2. High routine load error rows accumulated on the FE Please recommend config options (BE and FE) we should tune for: - Controlling compaction concurrency and CPU ceiling so compaction doesn't starve ingestion threads - Throttling or prioritizing routine load ingestion under CPU pressure - Reducing or surfacing routine load error row buildup (max error tolerance, retry behavior) - Any memory/thread pool settings relevant to a 3-BE shared-nothing topology under sustained row-level ingestion Include the config key name, recommended value range, and the tradeoff for each.

Running a StarRocks BE with only 2 CPU cores is significantly below the recommended minimum (8+ cores). In this resource-constrained environment, the default background thread counts will cause severe CPU oversubscription when ingestion and compaction run simultaneously. To stabilize your 3-BE shared-nothing cluster on version 3.5.7, apply the following configurations: 1. Control Compaction Concurrency Compaction is CPU-intensive. By default, StarRocks may spawn more compaction threads than you have physical cores, starving the ingestion process. Config Key | Location | Recommended Value | Tradeoff

compact_threads

| BE |

1

or

2

| Tradeoff: Slower merging of data version (tablets). If set too low, you may see "too many versions" errors if ingestion is high.

max_compaction_concurrency

| BE |

2

| Tradeoff: Limits total parallel compaction tasks. Prevents CPU spikes but can lead to a backlog of uncompacted data.

update_compaction_num_threads_per_disk

| BE |

1

| Tradeoff: Specifically throttles compaction for Primary Key tables. Essential if your routine load targets PK tables. 2. Throttle Routine Load Ingestion The default concurrency for Routine Load assumes a much larger BE. You must reduce the task pressure to match your 2-core limit. Config Key | Location | Recommended Value | Tradeoff

max_routine_load_task_num_per_be

| FE |

2

to

4

| Tradeoff: Lowering this reduces CPU usage but limits total ingestion throughput. At 3.5k rows/sec, 4 tasks should suffice.

max_routine_load_task_concurrent_num

| FE |

2

| Tradeoff: Limits how many tasks a single job can spawn. Helps prevent one high-traffic topic from hogging all BE threads.

routine_load_task_consume_second

| FE/Job |

20

-

30

| Tradeoff: Longer tasks reduce the frequency of "commit" overhead, but increase the latency before data is visible in queries. 3. Manage Routine Load Error Rows "Error row buildup" usually means your

max_error_number

is high enough that the job keeps running despite bad data, or you are not seeing why rows are failing. * Surface Errors: Set

log_rejected_record_num

to a positive value (e.g.,

100

) in your BE config. This allows you to see the actual failed rows via

information_schema.loads

. * Job Tuning: In your

CREATE ROUTINE LOAD

statement, use these properties: * `"max_error_number" = "100"`: If more than 100 errors occur in a window, the job pauses. This prevents the FE from being flooded with error metadata. _ `"max_batch_rows" = "200000"`: This defines the error window. Errors are calculated as a ratio within

10 _ max_batch_rows

. 4. Optimize for 2-Core K8s Environment StarRocks' default thread pools are too large for a 2-core pod. Manually cap these to prevent the BE from thrashing. Config Key | Location | Recommended Value | Why?

num_cores

| BE |

2

| Explicitly tells the BE to optimize internal scheduling for 2 cores.

scanner_thread_pool_thread_num

| BE |

8

(Default 48) | Reduces context switching during data scans.

pipeline_prepare_thread_pool_thread_num

| BE |

2

| Matches physical core count for query preparation.

fragment_pool_thread_num_min

| BE |

2

(Default 64) | Prevents the BE from pre-allocating dozens of idle threads. Summary Recommendation 1. Lower Compaction: Set

compact_threads = 1

and

max_compaction_concurrency = 2

on all BEs. 2. Lower Ingestion Concurrency: Run

ADMIN SET FRONTEND CONFIG ("max_routine_load_task_num_per_be" = "4");

on the FE. 3. Monitor Versions: If you see

OLAP_ERR_TOO_MANY_VERIONS

after these changes, you must either increase CPU cores per pod or reduce the Kafka ingestion frequency by increasing

max_batch_interval

intervals. References • administration/management/BE_configuration.md • faq/Others.md

@Kailas Unni I am self hosting 3 shared nothing clusters on 3.5. I would like to address if there are any elephants in the room. 1. What is the purpose you're running the Starrocks cluster on lower specs ? Is it a QA/Test env ? 2. Ideally on production, we run on 4vCPU and 8Gi Mem for FE Nodes and 8 vCPU and 32Gi Mem. 3. While tuning the compaction values might seem compelling, I would advise against it. We did the same and started getting too many versions during Stream load whenever batch size exceeded 200. This number might be different depending on your data volume and cardinality. Happy to chat more, and understand your usecase.