@Rocky

2026-06-01 11:34:42.185	
{
  "level": "W",
  "tid": "139624492025408",
  "source_file": "kv_store.cpp",
  "source_line": "348",
  "log": "rocksdb iterate timeout, try to compact",
  "filepath": "/var/log/app/be.WARNING"
}
	2026-06-01 11:34:42.161	
{
  "level": "W",
  "tid": "139624492025408",
  "source_file": "kv_store.cpp",
  "source_line": "327",
  "log": "rocksdb iterate timeout: 60400, limit: 60000",
  "filepath": "/var/log/app/be.WARNING"
}
	2026-06-01 11:33:40.855	
{
  "level": "W",
  "tid": "139624936848768",
  "source_file": "memory_lock.cpp",
  "source_line": "46",
  "log": "mlock failed for  147415040-357412864 (209997824 bytes): Cannot allocate memory",
  "filepath": "/var/log/app/be.WARNING"
}

The logs you provided indicate two related issues on your StarRocks BE (Backend) nodes: memory configuration limits and performance bottlenecks in the internal metadata storage (RocksDB). 1.

mlock failed: Cannot allocate memory

This error in

memory_lock.cpp

occurs when the StarRocks process attempts to lock a memory region into physical RAM (to prevent it from being swapped out) but is blocked by the Operating System. * Cause: The most common reason is that the Linux

ulimit

for "locked-in-memory size" (

memlock

) is set too low. * Impact: StarRocks uses

mlock

for critical performance components like the Primary Key (PK) Index. If memory locking fails, these components may be swapped to disk, leading to severe performance degradation. * Solution: Check your current limit with

ulimit -l

. It should be set to

unlimited

. You can set this permanently by adding the following lines to

/etc/security/limits.conf

(and restarting the BE):

bash
    * soft memlock unlimited
    * hard memlock unlimited

2.

rocksdb iterate timeout: 60400, limit: 60000

This warning in

kv_store.cpp

indicates that an internal scan of the RocksDB metadata (often used for Primary Key indices or tablet metadata) took longer than the 60-second internal threshold. * Cause: This is typically caused by "compaction debt." If there are too many deleted or updated records (tombstones) in the index that haven't been cleaned up yet, iterating through the index becomes very slow. * Automatic Action: As seen in your logs (

try to compact

), StarRocks detects this and automatically triggers a RocksDB compaction to try and optimize the index structure. * Potential Trigger: The

mlock

failure mentioned above often triggers this. If the index cannot stay in RAM, every "iterate" operation results in slow disk I/O, causing it to exceed the 60s limit. Recommended Actions 1. Fix Ulimits: Ensure

memlock

,

nofile

, and

stack

are set to

unlimited

or the recommended high values as per the StarRocks Environment Configuration. 2. Check Memory Usage: Verify if the BE process is reaching its

mem_limit

(default is 90% of RAM). If the host is under extreme memory pressure,

mlock

will fail even if ulimits are correct. 3. Monitor Disk I/O: If these timeouts persist after fixing memory limits, check the disk latency on the drives where your BE metadata is stored (usually the first path in

storage_root_path

). NVMe SSDs are highly recommended for Primary Key tables. 4. Version Check: If you are on an early version of 3.x or 4.x, ensure you are on the latest patch release, as there have been specific bug fixes regarding memory management in

memory_lock.cpp

recently. References • deployment/environment_configurations.md • developers/build-starrocks/Build_in_docker.md • faq/operation_maintenance_faq.md • faq/Sql_faq.md • administration/management/BE_configuration.md