@Rocky Can you provide any guidance on how to remedy any issues that might be found by running the script 'starrocks-diagnostics.py' found here: https://github.com/StarRocks/starrocks/tree/main/tools/starrocks-diagnostics

The

starrocks-diagnostics.py

script is a comprehensive tool designed to verify that your environment and StarRocks cluster settings align with best practices. When you run this script, it typically flags issues with a

[FAIL]

or

[WARN]

status. Below is a guide on how to remedy the most common issues identified by this script: 1. Operating System & Kernel Settings These are the most frequent issues found during diagnostics, as they directly impact stability and performance. * Transparent Huge Pages (THP): * Issue: THP can cause high system CPU usage and memory allocation delays. * Remedy: Disable it temporarily with

echo never | sudo tee /sys/kernel/mm/transparent_hugepage/enabled

and

echo never | sudo tee /sys/kernel/mm/transparent_hugepage/defrag

. To make it permanent, add

transparent_hugepage=never

to your boot loader (grub) configuration. * Virtual Memory Settings (

max_map_count

): * Issue: StarRocks BE requires a high number of memory map areas. * Remedy: Set

vm.max_map_count

to at least

262144

.

bash
        sudo sysctl -w vm.max_map_count=262144
        # To make permanent, add "vm.max_map_count=262144" to /etc/sysctl.conf

* Swap and Swappiness: * Issue: Swapping memory to disk severely degrades performance. * Remedy: Disable swap using

sudo swapoff -a

. Alternatively, set

vm.swappiness

to

0

or

1

in

/etc/sysctl.conf

. * Overcommit Memory: * Issue: Strict overcommit settings can cause memory allocation failures. * Remedy: Set

vm.overcommit_memory

to

1

.

bash
        sudo sysctl -w vm.overcommit_memory=1

2. Resource Limits (

ulimit

) * File Descriptors: * Issue: StarRocks opens many files and connections. * Remedy: Ensure the

nofile

limit is set to at least

65535

. Check with

ulimit -n

. You can update this in

/etc/security/limits.conf

. 3. Storage and Disk Configuration * Disk Scheduler: * Issue: Default schedulers (like

cfq

) may not be optimal for SSDs/NVMe. * Remedy: * For SATA/HDD: Use

mq-deadline

. * For SSD/NVMe: Use

none

or

kyber

.

bash
        echo none | sudo tee /sys/block/<device_name>/queue/scheduler

* Storage Space: * Issue: Low disk space on BE data nodes or FE metadata paths. * Remedy: Clear old logs in the

log/

directory or add more storage capacity. Monitor

<http://be.INFO|be.INFO>

for "Low disk space" warnings. 4. StarRocks Configuration (FE/BE) * Memory Limit (

mem_limit

): * Issue: BE process is crashing due to OOM or the limit is set too low for the available RAM. * Remedy: Adjust

mem_limit

in

be.conf

. By default, it is 90% of total memory. If other processes run on the same node, lower this value. * Java Heap Size (

JAVA_OPTS

): * Issue: FE metadata operations are slow or failing with

OutOfMemoryError

. * Remedy: Increase the

-Xmx

and

-Xms

values in

fe.conf

. 5. Network and Connectivity * Port Conflicts: * Issue: Required ports (e.g., 9030, 8030, 9050, 8040) are occupied. * Remedy: Check for port usage with

netstat -tulpn

and ensure no other services are using StarRocks ports. * Clock Skew: * Issue: Time difference between nodes is too large (important for consistency and logs). * Remedy: Synchronize clocks across all nodes using

ntp

or

chrony

. Next Steps If the script highlights specific Log Errors (like "Too many versions" or "Tablet is corrupted"): 1. Too many versions: This usually indicates a high frequency of small loads. Remedy this by increasing the data volume per load or adjusting compaction settings (

cumulative_compaction_num_threads_per_disk

). 2. Tablet corrupted: Run

SHOW BACKENDS

and

SHOW TABLET <id>

to identify the affected node and consider rebalancing or repairing the tablet.