< Rocky> I see sporadic query timeouts on cluster of version StarRocks #questions-and-troubleshooting

<@U0A71G31CDV> I see sporadic query timeouts on cl...

Andrei Harbunou

06/18/2026, 7:23 PM

@Rocky I see sporadic query timeouts on cluster of version 4.0.0. I noticed that when it happens it affect all queries for short period of time. CPU and memory metrics don't show anomalies and stays low. group concurrency overlflow metrics also stays the same. what other metrics can be useful here? any suggestions on what can be root cause?

Rocky

06/18/2026, 7:24 PM

Based on your description—sporadic, cluster-wide timeouts with low CPU/memory and no concurrency overflow—the issue is likely a Front End (FE) stall. In version 4.0.0, StarRocks has several new observability tools that can help pin this down. 1. Root Cause Hypothesis * FE Lock Contention: A specific query (e.g., a complex metadata operation or a DDL) might be holding a global or database-level lock, blocking all other queries from beginning the planning phase. * FE Full GC: Even if memory looks "low" on average, a "Stop-the-World" Garbage Collection pause in the JVM will hang the FE. If the pause exceeds a few seconds, queries will timeout. * Metadata Stalling: If you are using external catalogs (like Hive or Iceberg), a slow response from the remote metastore or NameNode can block FE threads during the file-listing phase. 2. Key Metrics & Diagnostic Queries Since standard resource metrics are healthy, you should look at these specific areas: A. Metadata Locks (The "Smoking Gun") In version 4.0.0, you can query the system view to see if a specific thread is blocking others:

Copy code

sql
-- Check for locks held for more than 5 seconds
SELECT lock_object, lock_mode, hold_time_ms, thread_info, waiter_list
FROM sys.fe_locks 
WHERE hold_time_ms > 5000;

If you see a thread with a long

hold_time_ms

and a non-empty

waiter_list

, that thread is the root cause. B. JVM Garbage Collection Check the FE's GC logs or Prometheus metrics. * Log: Look at

${STARROCKS_HOME}/log/fe.gc.log

. Search for "Full GC" or pauses longer than 1s. * Metrics: Monitor

jvm_gc_pause_seconds_sum

and

jvm_gc_pause_seconds_count

. * Memory Profiles: In 4.0.0, check

fe/log/proc_profile

. Recent versions automatically generate flame graphs here when memory spikes or anomalies are detected. C. FE Audit Log (Pending Time) Examine the

fe.audit.log

for queries that timed out. Look at the

PendingTimeMs

field: * If

PendingTimeMs

is high (e.g., several seconds), the query was stuck inside the FE (likely waiting for a lock or a thread) before it even started executing on the BE. 3. Suggested Fixes / Workarounds * Enable Table-Level Locking: If

lock_manager_enabled

false

in your FE config, the cluster uses coarse database-level locks. Ensure this is

true

to reduce contention. * JVM Tuning: If GC is the culprit, increase the FE heap size (

-Xmx

) in

fe.conf

. For large metadata clusters, 16GB or 32GB is often necessary. * Limit Large Metadata Scans: If the stall happens during queries on external tables with many partitions, consider setting

enable_collect_remote_files_async = true

(if available in your sub-version) or optimizing the external metastore. * Analyze Big Queries: Use

SHOW PROC '/current_queries';

during the stall to see if a single massive query is currently in the "Planning" state. If the issue persists and

sys.fe_locks

is empty, check for network TCP retransmissions or RPC timeouts between the FE and BE, as these can also cause cluster-wide "hiccups" without spiking CPU. References • best_practices/query_tuning/query_profile_operator_metrics.md • faq/operation_maintenance_faq.md • administration/management/resource_management/query_queues.md

Andrei Harbunou

06/18/2026, 7:30 PM

@Rocky for reference I only run MVs that are based on Iceberg table

Rocky

06/18/2026, 7:30 PM

Since you are running Materialized Views (MVs) on Iceberg tables and experiencing sporadic, cluster-wide timeouts while CPU/Memory remain low, the root cause is almost certainly FE (Front End) Metadata Lock Contention. In StarRocks, query planning for Iceberg requires fetching manifest files from remote storage (S3/HDFS). If an MV refresh or a query triggers a metadata update while holding a metadata lock, it can block all other queries trying to access the same database or table. 1. Diagnostic Metrics & Views In version 4.0.0, you have access to specific tools to confirm lock contention: * sys.fe_locks
(Real-time): When the timeout occurs, immediately run:

Copy code

sql
    SELECT lock_object, lock_mode, hold_time_ms, thread_info, waiter_list
    FROM sys.fe_locks
    WHERE hold_time_ms > 2000;

If you see a thread with a long

hold_time_ms

and a large

waiter_list

, that thread (likely an MV refresh or an Iceberg metadata scan) is the blocker. * Prometheus Metrics: * `starrocks_fe_slow_lock_wait_time_ms`: Tracks how long queries are waiting for metadata locks. * `starrocks_fe_slow_lock_held_time_ms`: Tracks how long locks are being held. * Audit Log `PendingTimeMs`: Check your

fe.audit.log

. If the timed-out queries have a high

PendingTimeMs

(e.g., 5000ms+) but low

PlanTime

and

ExecuteTime

, it confirms they were stuck waiting for a lock before execution even started. 2. Potential Root Causes for Iceberg MVs * Database-level Locking: By default, StarRocks might use a global database lock. If one MV is refreshing, it might block all other queries in that DB. * Iceberg Manifest Fetching: Planning Iceberg queries involves reading manifest files. If your Iceberg table has many small files or snapshots, reading these manifests from S3/HDFS can be slow. This "io-bound" work is done inside the FE while holding a lock. * JVM Safepoints (GC): Even if total memory is low, a specific metadata-heavy operation can trigger a "Young GC" or "Full GC" pause, freezing the FE. 3. Suggestions for Root Cause & Fix 1. Enable Table-Level Locking: Ensure your FE is using fine-grained locking to prevent one table's refresh from blocking the whole database. * Check config:

ADMIN SHOW FRONTEND CONFIG LIKE 'lock_manager_enabled';

* If

false

, set

lock_manager_enabled = true

fe.conf

(requires restart). 2. Optimize Iceberg Metadata Cache: Increase the FE memory allocated to Iceberg metadata to reduce remote I/O during planning. *

iceberg_metadata_memory_cache_capacity

(Default is 512MB, consider increasing to 2GB or more). * Enable disk cache for Iceberg metadata:

enable_iceberg_metadata_disk_cache = true

. 3. Check
fe.log
for `DeadlockChecker`: Search for "DeadlockChecker" or "Slow lock" in your logs. StarRocks 4.0.0 logs JSON stacks when it detects locks held longer than

slow_lock_threshold_ms

(default 3s). 4. Reduce Refresh Concurrency: If multiple Iceberg MVs refresh simultaneously, they may compete for the same metadata threads. Check

max_refresh_concurrency

in your MV settings. Summary The "all queries affected" symptom points to a FE lock. Focus on the

sys.fe_locks

view and the

starrocks_fe_slow_lock_wait_time_ms

metric. If you find the lock is held by a "PartitionBasedMvRefreshProcessor" thread, you should look into optimizing the Iceberg manifest size or increasing metadata cache settings mentioned in that thread's stack. References • sql-reference/sys/fe_locks.md • data_source/catalog/iceberg/iceberg_catalog.md • administration/management/monitoring/metrics.md • data_source/datalake_faq.md References

Rocky

06/18/2026, 7:30 PM

• administration/management/monitoring/metrics.md • sql-reference/sys/fe_locks.md

Andrei Harbunou

06/18/2026, 7:42 PM

@Rocky I see this in logs

Copy code

WARN (autovacuum-pool1-t3|58457) [LockManager.logSlowLockTrace():425] LockManager detects slow lock : {"owners":[{"id":91890,"name":"starrocks-taskrun-pool-300","type":"READ","heldFor":4895,"waitTime":0,"stack":["java.base@17.0.16/java.util.Collections$SetFromMap.add(Collections.java:5686)","java.base@17.0.16/java.lang.Throwable.printStackTrace(Throwable.java:666)","java.base@17.0.16/java.lang.Throwable.printStackTrace(Throwable.java:736)","app//org.apache.logging.log4j.core.pattern.ThrowablePatternConverter.formatOption(ThrowablePatternConverter.java:211)","app//org.apache.logging.log4j.core.pattern.ThrowablePatternConverter.format(ThrowablePatternConverter.java:137)","app//org.apache.logging.log4j.core.layout.PatternLayout$NoFormatPatternSerializer.toSerializable(PatternLayout.java:342)","app//org.apache.logging.log4j.core.layout.PatternLayout.toText(PatternLayout.java:240)","app//org.apache.logging.log4j.core.layout.PatternLayout.encode(PatternLayout.java:225)","app//org.apache.logging.log4j.core.layout.PatternLayout.encode(PatternLayout.java:59)","app//org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(AbstractOutputStreamAppender.java:215)","app//org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutputStreamAppender.java:208)","app//org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputStreamAppender.java:199)","app//org.apache.logging.log4j.core.appender.RollingFileAppender.append(RollingFileAppender.java:301)","app//org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:161)","app//org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:134)","app//org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:125)","app//org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:89)","app//org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:683)","app//org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:641)","app//org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:624)","app//org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:560)","app//org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:82)","app//org.apache.logging.log4j.core.Logger.log(Logger.java:162)","app//org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2205)","app//org.apache.logging.log4j.spi.AbstractLogger.logMessageTrackRecursion(AbstractLogger.java:2159)","app//org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2142)","app//org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:2052)","app//org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1923)","app//org.apache.logging.log4j.spi.AbstractLogger.error(AbstractLogger.java:881)","app//com.starrocks.connector.iceberg.IcebergCatalog.getPartitions(IcebergCatalog.java:338)","app//com.starrocks.connector.iceberg.IcebergCatalog.getPartitionsByNames(IcebergCatalog.java:376)","app//com.starrocks.connector.iceberg.IcebergMetadata.getPartitions(IcebergMetadata.java:572)","app//com.starrocks.connector.CatalogConnectorMetadata.getPartitions(CatalogConnectorMetadata.java:197)","app//com.starrocks.server.MetadataMgr.getPartitions(MetadataMgr.java:804)","app//com.starrocks.connector.partitiontraits.IcebergPartitionTraits.getPartitions(IcebergPartitionTraits.java:61)","app//com.starrocks.connector.partitiontraits.DefaultTraits.getPartitionNameWithPartitionInfo(DefaultTraits.java:119)","app//com.starrocks.connector.partitiontraits.DefaultTraits.getUpdatedPartitionNames(DefaultTraits.java:148)","app//com.starrocks.catalog.MaterializedView.getUpdatedPartitionNamesOfExternalTable(MaterializedView.java:961)","app//com.starrocks.catalog.MvRefreshArbiter.getMvBaseTableUpdateInfo(MvRefreshArbiter.java:195)","app//com.starrocks.scheduler.mv.MVPCTRefreshPartitioner.getMvPartitionNamesToRefresh(MVPCTRefreshPartitioner.java:251)","app//com.starrocks.scheduler.mv.MVPCTRefreshListPartitioner.getMVPartitionsToRefresh(MVPCTRefreshListPartitioner.java:364)","app//com.starrocks.scheduler.PartitionBasedMvRefreshProcessor.getPartitionsToRefreshForMaterializedView(PartitionBasedMvRefreshProcessor.java:1055)","app//com.starrocks.scheduler.PartitionBasedMvRefreshProcessor.checkMvToRefreshedPartitions(PartitionBasedMvRefreshProcessor.java:320)","app//com.starrocks.scheduler.PartitionBasedMvRefreshProcessor.doRefreshMaterializedView(PartitionBasedMvRefreshProcessor.java:478)","app//com.starrocks.scheduler.PartitionBasedMvRefreshProcessor.doRefreshMaterializedViewWithRetry(PartitionBasedMvRefreshProcessor.java:430)","app//com.starrocks.scheduler.PartitionBasedMvRefreshProcessor.doMvRefresh(PartitionBasedMvRefreshProcessor.java:382)","app//com.starrocks.scheduler.PartitionBasedMvRefreshProcessor.processTaskRun(PartitionBasedMvRefreshProcessor.java:215)","app//com.starrocks.scheduler.TaskRun.executeTaskRun(TaskRun.java:352)","app//com.starrocks.scheduler.TaskRunExecutor.lambda$executeTaskRun$0(TaskRunExecutor.java:60)","app//com.starrocks.scheduler.TaskRunExecutor$$Lambda$3896/0x0000000802306da8.get(Unknown Source)","java.base@17.0.16/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)","java.base@17.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)","java.base@17.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)","java.base@17.0.16/java.lang.Thread.run(Thread.java:840)"]}],"waiter":[{"id":58458,"name":"autovacuum-pool1-t4","type":"WRITE","waitTime":3002},{"id":58457,"name":"autovacuum-pool1-t3","type":"WRITE","waitTime":3001},{"id":56,"name":"autovacuum","type":"READ","waitTime":994}]}

Rocky

06/18/2026, 7:42 PM

This log snippet provides a clear explanation of why your cluster is stalling. You are experiencing Lock Starvation triggered by an expensive logging operation in the Front End (FE). 1. The Mechanism of the "Stall" Even though the MV refresh thread is only holding a READ
lock, it is blocking the whole cluster because of how the lock queue works: 1. The Blocker: An MV refresh thread (

starrocks-taskrun-pool-300

) is trying to get partitions for an Iceberg table. It encountered an error and is currently stuck in

log.error()

. 2. The Bottleneck: The stack trace shows the thread is spending 4.9 seconds inside

java.lang.Throwable.printStackTrace

and

log4j

formatting logic. While it is "busy" printing this error to the log file, it continues to hold a

READ

lock on the metadata. 3. The Queue: While that thread is stuck,

autovacuum

threads (IDs 58458, 58457) arrived and requested WRITE
locks. 4. The Stall: In StarRocks, once a

WRITE

lock is waiting in the queue, all new

READ

requests (i.e., your incoming user queries) are blocked to prevent the "writer" from being starved forever. Result: Your entire cluster's metadata access is paused for ~5 seconds while one thread finishes writing a massive stack trace to a log file. 2. Root Cause: Iceberg Metadata Error The thread is stuck because it's trying to log an error from:

com.starrocks.connector.iceberg.IcebergCatalog.getPartitions(IcebergCatalog.java:338)

This typically happens when: * Iceberg Metadata is Stale: StarRocks is trying to access a manifest or metadata file in S3/HDFS that was deleted by a concurrent Spark/Flink job (a common issue in early 4.0.x versions). * Catalog Connectivity: There is a timeout or "Connection Reset" when talking to the Iceberg REST catalog or Hive Metastore. * Deep Stack Traces: If the exception chain is very long,

Throwable.printStackTrace

can become computationally expensive and slow down the FE. 3. Recommended Actions Immediate Mitigation * Identify the Error: Look in your

fe.log

for the actual error message generated by

IcebergCatalog:338

. It will appear immediately after the "slow lock" warning. This will tell you if it's a "File Not Found," "Timeout," or "Permission Denied" issue. * Disable Slow Stack Printing (If possible): If you have modified

log4j2.xml

, ensure you aren't using overly verbose layout patterns for exceptions. Standard StarRocks configs are usually fine, but high I/O wait on the FE's disk can make log writing synchronous and slow. Optimization for Iceberg MVs * Upgrade to 4.0.1+: Version 4.0.0 had several early-release issues regarding Iceberg metadata caching and lock granularity. Newer patches (4.0.1, 4.0.2) include fixes that reduce lock hold times during partition fetching. * Increase FE Heap: If the FE is hitting "near-full" memory,

Throwable.printStackTrace

and string manipulation become extremely slow due to GC pressure. Ensure the FE has at least 16GB-32GB of RAM. * Table-Level Locking: Ensure

lock_manager_enabled = true

is set in your

fe.conf

. This prevents a lock on one Iceberg table from blocking the entire database. Diagnostic Query Run this during the next stall to see exactly which Iceberg tables are involved:

Copy code

sql
SELECT * FROM information_schema.task_runs 
WHERE state = 'RUNNING' 
ORDER BY create_time DESC;

This will show you which MV was refreshing when the lock occurred. You should-lock stall occurred. References • sql-reference/sys/fe_locks.md • data_source/catalog/iceberg/iceberg_catalog.md • administration/management/monitoring/metrics.md

Rocky

06/18/2026, 7:42 PM

• administration/management/resource_management/query_queues.md • faq/operation_maintenance_faq.md • data_source/datalake_faq.md • best_practices/query_tuning/query_profile_operator_metrics.md • sql-reference/sql-statements/data-definition/CREATE_MATERIALIZED_VIEW.md -2) -asynchronous)

Andrei Harbunou

06/18/2026, 7:57 PM

@Rocky I also see that query timeouts at like > 60 sec, if lock is for 5 sec only why it took more?

Rocky

06/18/2026, 8:01 PM

The discrepancy between the 5-second lock warning and the 60-second query timeout is a common diagnostic trap. Here is why this happens and what is causing your cluster-wide stall: 1. The "Snapshot" Illusion The log entry you see is a warning trigger, not a post-mortem summary. * StarRocks has a background checker (usually set to 3 seconds via

slow_lock_threshold_ms

). * At exactly ~4.9 seconds of the lock being held, the checker woke up, saw the thread was still active, and dumped the stack trace. * The lock was likely held much longer than 5 seconds. The thread was stuck in a "Blocking Logging" state (see below), and it could have remained stuck for the full 60 seconds or more. 2. Head-of-Line Blocking (The "Write" Barrier) In your log, notice the waiters: * `owner`: 1 thread with a

READ

lock (ID 91890, the MV refresh). * `waiter`: 2 threads wanting a

WRITE

lock (ID 58458, 58457,

autovacuum

). StarRocks uses a fair locking policy to prevent "writer starvation." Once a

WRITE

lock request (like

autovacuum

) enters the queue, all new
READ
requests (your user queries) are blocked until the current

READ

owner finishes AND the

WRITE

waiters finish. Even if the

READ

owner eventually finishes, the

autovacuum

(WRITE) task must then run. If you have many Iceberg MVs, these metadata "write" tasks can stack up, effectively freezing all incoming query planning for the duration of the entire queue. 3. The Root Cause: Expensive Logging in 4.0.0 Your stack trace shows the thread is stuck here:

java.lang.Throwable.printStackTrace

log4j...ThrowablePatternConverter.format

This is a known performance issue when an Iceberg metadata error occurs in StarRocks 4.0.0: 1. Iceberg Metadata Error: Something went wrong in

IcebergCatalog.getPartitions

(often a

NullPointerException

or "File Not Found" due to a stale snapshot bug in 4.0.0). 2. Synchronous Logging: StarRocks attempts to log the error. If the stack trace is deep (common with Iceberg/Thrift), the process of converting that stack trace into a string for

log4j

is CPU-intensive and synchronous. 3. Lock Contention: Because this logging happens while the thread holds the metadata lock, the entire cluster's metadata lock is held hostage by the FE's disk I/O and string serialization. Recommended Fixes 1. Upgrade to 4.0.2 or later: Version 4.0.0 has a specific bug (Issue #64424/65381) where Iceberg metadata fetching can fail on evolved or stale partitions, triggering the exact stack trace you see. Later patches significantly optimize metadata handling and lock granularity. 2. Disable Heavy Stack Tracing: You can tell the Lock Manager not to capture full stacks, which reduces the overhead during these events:

Copy code

sql
    ADMIN SET FRONTEND CONFIG ("slow_lock_print_stack" = "false");

3. Enable Table-Level Locking: Ensure your FE is not using global database locks: * In

fe.conf

, set

lock_manager_enabled = true

and restart. 4. Increase FE Memory: If the FE is near its JVM heap limit, the

printStackTrace

and

log4j

formatting can trigger massive GC overhead, turning a 5-second stall into a 60-second one. Immediate Next Step: Look in your

fe.log

for the actual Iceberg error that follows that "Slow Lock" warning. It will likely be a

NullPointerException

IcebergCatalog.java:338

. Fixing that specific metadata inconsistency (e.g., by running

REFRESH EXTERNAL TABLE ...

) may stop the immediate timeouts. References • sql-reference/sys/fe_locks.md • administration/management/FE_configuration.md • data_source/catalog/iceberg/iceberg_catalog.md • administration/management/monitoring/metrics.md

Rocky

06/18/2026, 8:01 PM

Rocky

06/18/2026, 8:01 PM

Rocky

06/18/2026, 8:01 PM

Rocky

06/18/2026, 8:01 PM

Rocky

06/18/2026, 8:01 PM

Rocky

06/18/2026, 8:01 PM

Rocky

06/18/2026, 8:01 PM

Rocky

06/18/2026, 8:01 PM

Rocky

06/18/2026, 8:01 PM

2 Views

Open in Slack

Previous Next