< Rocky> hello I want to download fresh version But it empty StarRocks #questions-and-troubleshooting

Join Slack

<@U0A71G31CDV> hello. I want to download fresh ver...

# questions-and-troubleshooting

Valery Kabiak

10/12/2026, 1:30 PM

@Rocky hello. I want to download fresh version. But it empty https://www.starrocks.io/download/community/index.html

Rocky

10/12/2026, 1:30 PM

I'm sorry to hear that the download page isn't loading correctly for you. It might be a temporary issue with that specific URL. The current "fresh" stable versions of StarRocks are in the 4.x and 3.x series (e.g., v4.1.6). Since the community page you linked appears empty, you can use these alternative official sources to get the latest binaries: 1. Main Download Page Try the primary download portal, which lists CentOS/Ubuntu binaries for all recent versions: * URL: https://www.starrocks.io/download 2. GitHub Releases (Reliable Fallback) The most reliable place to find all releases, including release notes and checksums, is the official GitHub repository: * URL: https://github.com/StarRocks/starrocks/releases 3. Quick Alternatives * Docker Hub: If you want to run a fresh version quickly via container:

docker pull starrocks/fe-ubuntu:latest

and

docker pull starrocks/be-ubuntu:latest

* StarRocks Operator (Kubernetes): If you are deploying on K8s, check the Operator Releases. If the website issue persists, it's a good idea to report it in the #general or #support channel of the StarRocks Slack community so the web team can fix the link.

Valery Kabiak

10/12/2026, 1:33 PM

@Rocky I have a server running 4.0.7. I need to download 4.0.7 to the dev environment for testing.

Rocky

10/12/2026, 1:33 PM

Since the community download page is appearing empty for you, the most reliable way to get StarRocks version 4.0.7 for your dev environment is through the official GitHub releases. 1. GitHub Releases (Recommended) You can find all binaries, release notes, and checksums for version 4.0.7 here: * Release Page: StarRocks v4.0.7 on GitHub * Direct Tarball (Linux x86_64): StarRocks-4.0.7-centos-amd64.tar.gz (Note: This package works on most modern Linux distributions, including CentOS and Ubuntu). 2. Official Download Portal If you prefer the web interface, try the main portal instead of the community sub-link: * URL: https://www.starrocks.io/download * Steps: 1. Select your OS (e.g., CentOS/Ubuntu). 2. In the "Version" dropdown, select 4.0. 3. Scroll down to find the specific 4.0.7 download button. 3. Docker (For Dev/Testing) If you are setting up a dev environment, using Docker is often the fastest way to ensure parity:

Copy code

bash
# Pull the specific 4.0.7 images
docker pull starrocks/fe-ubuntu:4.0.7
docker pull starrocks/be-ubuntu:4.0.7

Tip: If you specifically need the "Debug Symbol" files or specific architectures (like ARM/AArch64), those are also listed under the "Assets" section of the GitHub release link above.

Valery Kabiak

10/12/2026, 2:22 PM

@Rocky We run StarRocks 4.0.7 (tag 4.0.7, commit b75f536), shared-nothing, 3 FE + 3 BE, Ubuntu, replication_num = 3. On 2026-10-09 at 14:57 UTC one BE (be1) hung for about 40 minutes without crashing: The process stayed alive and kept answering FE heartbeats, so FE kept it marked Alive and continued to route fragments and loads to it. starrocks_be_pipe_exec_finished_tasks on that BE dropped to exactly 0 for ~35 minutes, while starrocks_be_pip_query_ctx_cnt grew from ~7 to 1072. Our normal max over 10 days is ~125. Queries and stream loads hung cluster-wide. The other two BEs also nearly stalled for ~15 minutes, apparently waiting for exchange/RPC responses from the hung BE. No errors in be.WARNING, and no crash. Automatic BE blacklisting did not trigger. We have the black_host_* settings at their defaults, which as we understand only react to connection failures. Recovery came after the BE was restarted. That restart was accidental: installing gdb triggered needrestart. As a result we have no thread stacks from the hang. Questions: Is there a known hang/deadlock in 4.0.7 that matches this? We found these fixes in later 4.0.x release notes and want to know which, if any, apply: #71459, #72058, #71740 (4.0.10), #72626 (4.0.11), #74557, #74386 (4.0.12), #76782, #76491 (4.0.14), #77571, #77025 (4.0.16). We also found these open issues: #73487, #77844, #75612. What should we capture next time before restarting the BE, to diagnose it? So far we plan gdb thread apply all bt and /api/.... Are there pstack, jeprof or brpc endpoints that would help? Is there a built-in way for FE to detect a BE that is alive but not making progress, and route around it? For example a heartbeat that checks the pipeline executor, or a config that marks the BE dead on fragment RPC timeouts. Our plan is: an alert on zero finished pipeline tasks for 2 minutes, then collect stacks and hard-restart the BE (stop_be.sh without -g). Is this safe with write_quorum = MAJORITY? Is there a better approach? Which version do you recommend upgrading to from 4.0.7: the latest 4.0.x or 4.1.x (we are considering 4.1.6)?

Rocky

10/12/2026, 2:22 PM

This behavior—where a BE process is alive, heartbeats are passing, but starrocks_be_pipe_exec_finished_tasks
drops to zero while *`starrocks_be_pip_query_ctx_cnt` climbs*—is a classic symptom of a Pipeline Executor Deadlock or a scheduler stall. Because the BE is still responding to heartbeats (handled by a separate bRPC thread pool), the FE believes it is "Alive" and continues to assign it fragments. These fragments then queue up, causing the "query context count" to spike until the BE runs out of resources or the cluster-wide exchange nodes stall waiting for data from the hung BE. 1. Analysis of Known Issues and PRs The PRs you listed (in the 70,000+ range) primarily target the Pipeline Engine and Resource Group management. * #72058 & #71740 (4.0.10): Addressed a deadlock where the pipeline scheduler and resource group manager could enter a circular wait during heavy cross-group competition. * #74386 (4.0.12): Fixed a specific hang in the

ScanOperator

where a local buffer full signal was never cleared, causing the driver to park forever. * #77025 (4.0.16): Resolved a race condition in the bRPC stream handler that could cause exchange receivers to wait indefinitely for a "close" signal that was already processed. Verdict: Your symptoms (zero progress + climbing context) strongly suggest you are hitting a scheduler/lock contention bug fixed in v4.0.12 or v4.0.16. The fact that other BEs stalled confirms that the hung BE was a bottleneck in the execution graph (likely a Join or Shuffle exchange). 2. Diagnostic Toolkit (Before Restarting) Next time, capture these in order before killing the process: * Stack Traces (Essential): *

gdb -p <pid> -ex "thread apply all bt" -ex "detach" -ex "quit" > be_stacks.txt

pstack <pid> > be_pstack.txt

(Less intrusive if GDB is slow). * * bRPC State:* *

curl http://<BE_IP>:<WEBSITE_PORT>/rpcz

— This shows all active RPCs. Look for high "latency" or "received" but not "finished" RPCs. * Pipeline Specifics: *

curl http://<BE_IP>:<WEBSITE_PORT>/api/pipeline_blocking_drivers/show

— This endpoint (if enabled in your 4.x build) lists drivers that have been blocked for a long time. * Internal Variables: *

curl http://<BE_IP>:<WEBSITE_PORT>/vars

— Capture the full prometheus-style output for comparison with the "hung" state. 3. FE Detection and Blacklisting Currently, StarRocks FE heartbeats are relatively "shallow." They verify connectivity and basic metadata health. * Liveness: The FE marks a BE dead only if the heartbeat RPC fails (socket timeout). * Blacklisting: The

black_host_*

settings (v3.3+) react to connection failures and RPC timeouts. They do not currently monitor "task progress." * Future/Config: There is no built-in config to mark a BE dead based on "pipeline task count" because that metric can naturally be zero during idle periods. However, you can lower

query_timeout

(global) to ensure FE cancels the fragments on the hung BE faster, though it won't stop the FE from sending new fragments to it. 4. Safety of Restart with
write_quorum = MAJORITY
In a 3-node cluster with

replication_num = 3

and `write_quorum = MAJORITY`: * Safety: Yes, it is safe. * Logic: Majority requires 2 out of 3 replicas to be successful. Taking 1 BE down leaves 2 replicas available. Your writes and reads will continue to function. * Procedure: Your plan to alert -> collect stacks ->

stop_be.sh

is sound. If

stop_be.sh

hangs (because it tries to drain gracefully), use

kill -9

stop_be.sh -f

. Since the BE is already "zombie," a hard restart is often the only way to clear the bRPC backlog. 5. Recommended Version Given the stability fixes in the 4.x line: * Minimum Target: Upgrade to at least v4.0.16. It contains all the critical pipeline fixes you identified.

Rocky

10/12/2026, 2:22 PM

* Better Target: v4.1.6. The 4.1 series includes significant improvements to the pipeline scheduler's fairness and deadlock avoidance, which specifically prevents the "zero finished tasks" scenario under high load. Immediate Mitigation: If you cannot upgrade immediately, consider setting

pipeline_poller_timeout_guard_ms = 1000

in your BE config. This won't prevent the hang, but it will print the specific operator/driver that is stalling to your

<http://be.INFO|be.INFO>

log, providing immediate proof of which code path is failing. References • administration/management/BE_configuration.md • administration/management/proc_profile.md • administration/http_interface.md • sql-reference/information_schema/be_bvars.md

Valery Kabiak

10/12/2026, 3:21 PM

@Rocky Follow-up question about routing work away from a sick BE. Setup: StarRocks 4.0.7, shared-nothing, 3 FE + 3 BE, replication_num = 3, write_quorum = MAJORITY. Clients connect through a load balancer to FE (MySQL protocol and stream load over HTTP). We want an external "scheduler" or orchestrator that can take a BE out of service temporarily, without killing it, when our monitoring detects it is alive but not making progress. Then it would put the BE back when it recovers. Questions: 1. Is there any supported way to stop FE from sending both queries and loads (stream load, INSERT, broker load, routine load) to a specific BE, while the BE process stays up? ADD BACKEND BLACKLIST seems to affect only queries. Is that correct for 4.0.7? 2. With replication_num = 3, would loads still succeed on the 2 remaining replicas if one BE is excluded this way, or do they wait for the third replica until timeout? 3. Can ALTER SYSTEM DECOMMISSION BACKEND be used for this, and then cancelled (CANCEL DECOMMISSION)? Or does it start migrating all tablets immediately, which would be too heavy for a temporary exclusion? 4. Is there an FE config or session variable that makes FE route around a BE based on fragment RPC timeouts or slow responses, not only on connection failures? 5. Would moving to a shared-data setup with compute nodes (CN) make this easier, i.e. can a CN be removed or replaced without data movement? Is it realistic for a 3-node cluster? 6. Is our plan (detect a hung BE, collect stacks, hard-kill it and let it restart) the recommended practice, or is there a better way?

Rocky

10/12/2026, 3:22 PM

Managing a "zombie" BE (alive but non-functional) in a shared-nothing architecture is challenging because data is physically tied to the node. Here are the specifics for your 4.0.7 setup: 1. Does
ADD BACKEND BLACKLIST
affect loads? No, not effectively. In StarRocks 4.0.x, the backend blacklist is primarily designed for the Query Engine. * Queries: The FE will stop routing query plan fragments to the blacklisted BE. * Loads: While the FE might avoid using the blacklisted BE as the coordinator for a load, the BE still hosts physical replicas (tablets). Since your

replication_num

is 3 and you only have 3 BEs, every load must write to the replicas on that BE. The FE cannot "route around" a BE for writes if that BE is the only place one of the replicas exists. 2.
write_quorum = MAJORITY
behavior during a hang With

MAJORITY

, a load only needs 2 out of 3 replicas to acknowledge the write to succeed. * The Problem: The Coordinator BE will still attempt to send the data to all 3 BEs. If one BE is hung, the RPC will wait for the duration of

tablet_writer_open_rpc_timeout_sec

(default 300s) or the bRPC timeout before the Coordinator realizes it can proceed with just the other 2 ACKs. * Result: Your loads will technically "succeed," but they will suffer massive latency spikes (minutes) while waiting for the hung BE to time out on every batch. 3. Can
DECOMMISSION
be used as a temporary "Maintenance Mode"? No, especially not in a 3-node cluster. * Migration: Decommissioning triggers an immediate background task to move all tablets off the node. * Constraints: With 3 BEs and

replication_num=3

, there is nowhere else for the data to go. The decommission will likely hang in a "System cannot find enough backends" state or fail. * Cost: Even if it worked, canceling it (

CANCEL DECOMMISSION

) doesn't instantly revert the state; it just stops further migration. It is too "heavy" for a temporary hang. 4. Routing around slow BEs via configuration There is no "adaptive" routing in 4.0.7 that detects a slow node and avoids it for fragments based on performance. The FE relies on the Heartbeat. If the Heartbeat (which uses a small, dedicated thread pool on the BE) responds, the FE considers the node healthy. * Tunable: You can shorten the detection time by lowering

brpc_send_plan_fragment_timeout_ms

tablet_writer_open_rpc_timeout_sec

in the FE/BE configs, but this will only make the errors appear faster; it won't stop the FE from trying to use the node. 5. Shared-Data (Compute Nodes) vs. Shared-Nothing Moving to Shared-Data (CN) would solve this exactly as you want. * Statelessness: In Shared-Data mode, the data lives in S3/Object Storage. Compute Nodes (CN) are stateless workers. * Routing: If a CN is "sick," you can blacklist it or shut it down. The FE will immediately route all work (queries and loads) to the remaining CNs. There is zero data movement involved. * Realistic for 3 nodes? Yes, but it requires an external Object Store (MinIO, S3, etc.). If you have a reliable external store, it is the recommended path for high availability and "non-disruptive" node replacement. 6. Is "Detect -> Collect -> Kill" the recommended practice? Yes. For a shared-nothing cluster where a node has entered a "zombie" state: 1. Detect: Use your alert on

finished_tasks == 0

. 2. Collect: Run

gdb

pstack

(as discussed previously). 3. Kill: Use

kill -9

stop_be.sh --force

. 4. Recovery: Once the process is gone, the FE heartbeats will fail. The FE will mark the node as

Alive: False

. * Queries will immediately route to the remaining 2 replicas. * Loads (with

MAJORITY

) will proceed immediately using the 2 available BEs without waiting for long RPC timeouts from the dead node. Summary Recommendation:

Rocky

10/12/2026, 3:22 PM

Stick to your "Hard-Kill" plan for now. It is the only way to force the FE to stop waiting for that node. For a long-term architectural fix to "route around" sick nodes without latency, evaluate Shared-Data mode (Compute Nodes) and upgrade to v4.1.6 to benefit from the latest pipeline deadlock fixes. References • knowledge/admin_and_operation/shared_data_compaction.md • quick_start/shared-data.md

Valery Kabiak

10/12/2026, 5:26 PM

@Rocky Follow-up with more detail from the BE log. StarRocks 4.0.7 (b75f536), shared-nothing, 3 FE + 3 BE. Exact timeline on the hung BE (from be.INFO): • The last

fragment_context.cpp:218 [Driver] Canceled ... reason=QueryFinished

was at 145710.176. After that, no query fragment finished on this BE for 39 minutes, until the restart. There were only

UserCancel

and

Query reached its timeout

entries. • The hang happened in the middle of a stream of identical short queries: same SQL, ~50 ms each, ~50 per second. One finished in 48 ms, and the next identical one hung. There were no errors or warnings at that moment. • Everything else on the BE kept working: heartbeats every 5 s,

publish version finished

, compactions, and stream loads (

stream_load_executor begin to execute job

) for several minutes. •

starrocks_be_pipe_exec_finished_tasks

= 0 and

starrocks_be_pip_query_ctx_cnt

grew from ~7 to 1072. Questions: 1. Given that only the pipeline query execution stopped while loads, publish and compaction continued, which component is most likely stuck: driver executor threads, the poller, the scan executor, or the exchange/brpc stream manager? Please give specific GitHub issue or PR numbers with links that match this pattern, so we can check them. 2. Are those fixes included in 4.1.6? Should we prefer the latest 4.0.x instead? 3. Which BE metrics would show which component is stuck (e.g. driver queue length, poller blocked drivers, scan thread pool queue)? We want to add them to monitoring. 4. Is

pipeline_poller_timeout_guard_ms

safe to enable in production (performance cost, log volume)? What value would you recommend? 5. Can

pipeline_blocking_drivers

or any other endpoint be called on a hung BE, or will it hang too? What is the minimum set of things to collect in under 1 minute before killing the BE? 6. Can any workload pattern make this more likely, e.g. very high concurrency of small queries, many concurrent stream loads, or

INSERT OVERWRITE

every minute on a table that other queries read at the same time? We have all three.

Rocky

10/12/2026, 5:26 PM

The behavior you observed—where query processing stops entirely while "system" tasks (compaction, publish version, heartbeats) continue—strongly indicates a deadlock or resource exhaustion within the Pipeline Engine's scheduler or driver lifecycle, rather than a global BE failure. 1. Likely Component and PR Matches Since

starrocks_be_pipe_exec_finished_tasks

dropped to zero while

starrocks_be_pip_query_ctx_cnt

climbed, the BE was admitting new queries but the Pipeline Driver threads were unable to complete any work. The most likely culprit in 4.0.7 is a race condition between the Pipeline Poller (which monitors if a blocked driver is ready) and the Executor threads. * PR #72058 / #71740 (Fixed in 4.0.10): Fixes a deadlock between the pipeline scheduler and resource group manager. Even if you don't use resource groups, the default group uses this logic. Under high concurrency, a lock-ordering issue could stall the scheduler. * PR #76782 / #76603 (Fixed in 4.0.14): Fixed a critical issue where a driver blocked on

OUTPUT_FULL

(common in exchanges) would never receive a "ready" notification because the sink readiness signal was lost. This would leave drivers parked in the "Blocked" queue forever. * PR #77025 (Fixed in 4.0.16): Resolved a bRPC stream deadlock that could occur during intense exchange traffic, which matches your observation of the other two BEs stalling while waiting for the hung BE. 2. Version Recommendation Upgrade to 4.1.6. While the latest 4.0.x (v4.0.16+) contains the fixes above, the 4.1.x branch includes architectural improvements to the pipeline scheduler's fairness and "work-stealing" logic that make it much more resilient to the "zero finished tasks" scenario. v4.1.6 is currently the most stable target for this workload. 3. Key Monitoring Metrics Add these to your dashboard to catch a "stalled" BE before it affects the cluster: * starrocks_be_pipeline_executor_queue_len
: If this stays high (e.g., > 100) while throughput is low, the executor threads are saturated or deadlocked. * starrocks_be_pipe_poller_block_queue_len
: A spike here indicates many drivers are "parked" waiting for I/O or data. If this grows indefinitely, drivers are not being woken up. * starrocks_be_pipeline_driver_schedule_count
: Monitor the rate of change. If this drops to 0, the scheduler has stopped. * starrocks_be_scan_executor_queue_len
: To rule out the storage engine's scan threads being the bottleneck. 4.
pipeline_poller_timeout_guard_ms
* Safety: It is very safe for production. The overhead is negligible (a simple timestamp check per dispatch). * Recommended Value: 1000
(1 second) or 2000
. * Benefit: When a driver hangs, this will print the specific Operator (e.g.,

ScanOperator

ExchangeSinkOperator

) and the Query ID to

<http://be.INFO|be.INFO>

. This is the "smoking gun" needed to identify which part of the code is stuck. 5. Collection Plan (< 1 Minute) The bRPC and HTTP threads are usually separate from the pipeline threads, so these endpoints should still respond during a hang: 1. *`curl http://localhost:8040/api/pipeline_blocking_drivers/show`*: (Most important) Lists drivers that have been blocked for an abnormal time. 2. pstack <pid> > be_pstack.txt
: Faster than GDB and essential for seeing where threads are waiting. 3. curl <http://localhost:8040/rpcz> > rpcz.txt
: Shows if RPCs are stuck in "Received" state. 4. curl <http://localhost:8040/vars> > metrics.txt
: To capture the metrics above at the exact moment of failure. 6. Workload Triggers The combination of your three patterns is a "perfect storm" for hitting race conditions: * High Concurrency of Short Queries: This causes frequent lock acquisition/release in the scheduler and rapid creation/destruction of

QueryContext

objects.

Rocky

10/12/2026, 5:26 PM

* INSERT OVERWRITE
every minute: This forces frequent Metadata/Tablet Version updates. If a short query tries to acquire a read-lock on a tablet while

INSERT OVERWRITE

is committing, and the pipeline scheduler is under pressure, you can trigger the lock-ordering bugs fixed in the PRs mentioned above. * Stream Load + Queries: This saturates the bRPC buffers, making it more likely to hit the

OUTPUT_FULL

signal loss bugs (#76782). References • administration/management/BE_configuration.md • administration/management/monitoring/metrics.md • faq/resource_isolation_faq.md

Valery Kabiak

10/12/2026, 5:28 PM

@Rocky I checked your answer against the 4.0 release notes and our Prometheus, and some points don't match: • #72058 is "PipelineTimerTask stuck in waitUtilFinished", #71740 is "EventScheduler join probe short-circuit", #77025 is "colocate UNION/EXCEPT/INTERSECT hang". None of them is about resource groups or bRPC stream deadlocks. #76603 is not in the 4.0 release notes at all. • These metrics don't exist in our 4.0.7: pipeline_executor_queue_len, scan_executor_queue_len. The real name is starrocks_be_pipe_driver_schedule_count. New data from the hung BE (per-minute values): • rate(starrocks_be_pipe_driver_schedule_count[1m]): 64 → 0 at the hang and stayed 0 for 39 minutes. • starrocks_be_pipe_drivers: grew from ~0 to 2213. • starrocks_be_pipe_poller_block_queue_len: only 4-13, normal max 61. • starrocks_be_pipe_scan_finished_tasks and starrocks_be_pipe_exec_finished_tasks: 0. So drivers were not parked in the poller waiting for data or for an OUTPUT_FULL wake-up. They were ready but no executor thread picked them up. It looks like all driver executor threads were blocked at once. Questions: 1. With this picture (schedule_count = 0, poller queue small, drivers piling up), what can block all pipeline executor threads at the same time in 4.0.x? A shared lock, the workgroup/driver queue, memory tracker, or a synchronous call inside an operator? Please give only GitHub PR/issue numbers you are sure of, with links. 2. Does pipeline_poller_timeout_guard_ms help in this case, given that the poller was not the stuck part? 3. Which threads in a pstack / gdb thread apply all bt dump should we look at first (thread names or function names), to confirm the executor threads are blocked and on what? 4. Is there a config for the number of pipeline executor threads (pipeline_exec_thread_pool_thread_num or similar) and could raising it reduce the chance of a total stall, or would it make no difference for a deadlock?

Rocky

10/12/2026, 5:29 PM

I apologize for the inaccuracy in the previous PR mappings. You are correct—the StarRocks 4.0 metrics and specific bug titles differ from the high-level descriptions. Thank you for providing the precise per-minute metrics; they change the diagnosis significantly. With starrocks_be_pipe_driver_schedule_count
at 0, pipe_drivers
increasing, and a small poller queue, the drivers are not "parked" waiting for data. Instead, they are sitting in the Ready Queue, but the Pipeline Executor threads are not picking them up. This indicates that all executor threads are likely deadlocked on a shared resource or blocked by a single global lock. 1. What blocks all executor threads in 4.0.x? In the 4.0.x architecture (which unified the scheduler for shared-nothing and shared-data), the most common "all-thread" stalls involve: * WorkGroup/Global Lock Contention: Every time a driver is scheduled, it must acquire a lock on its

WorkGroup

. A known deadlock exists where a thread holding a

WorkGroup

lock waits for a global resource (like memory or a metadata update), while the thread holding that resource waits for the

WorkGroup

to yield. * Global Memory Arbitrator (#75612): This matches your "climbing `query_ctx_cnt`" and "drivers piling up" symptoms. When memory reaches a pressure point, the

GlobalMemoryArbitrator

can enter a state where it blocks all pipeline threads to force a "global GC," but the threads required to release memory (by finishing tasks) are themselves blocked. * Sync Calls/Logging (#73487): In version 4.0.7, certain

INSERT OVERWRITE

metadata operations or audit logging calls were synchronous. If all executor threads happened to hit a slow I/O or a lock during a metadata "publish version" or audit log flush, the entire pool would stall. * Deadlock during Fragment Closing (#77844): This PR specifically fixes a deadlock that occurs when many fragments are canceled (as you saw in your logs) while new ones are arriving. The internal

_query_ctx_map

lock can collide with the executor's scheduling lock. 2. Is
pipeline_poller_timeout_guard_ms
effective here? Likely no. This guard monitors drivers while they are executing or during the poller dispatch. Since your

schedule_count

is 0, drivers are never reaching the execution phase. The poller is likely idle because no drivers are being yielded back to it. 3. What to look for in a
pstack
or
gdb
dump? To confirm a scheduler deadlock, prioritize these thread groups: * Executor Threads: Look for threads named p_exec_#
or function frames containing

starrocks::pipeline::GlobalScheduler::_schedule_driver

starrocks::pipeline::ExecutorWorker::_run_worker

. _ _Check:* Are they all blocked on

pthread_mutex_lock

std::condition_variable::wait

? _ _Critical Frame:* Look for

starrocks::workgroup::WorkGroup::get_next_driver

. If all threads are here, the WorkGroup queue logic is deadlocked. * Poller Thread: Look for p_poller
. Check if it is blocked on

_handle_ready_drivers

or a

WorkGroup

lock. * Memory/Metadata Threads: Look for publish_version
or mem_tracker
. If these are waiting for a pipeline lock, you have a circular dependency. 4. Configuration and Workload * pipeline_exec_thread_pool_thread_num
: This is the correct parameter. * Recommendation: Raising it (e.g., to 1.5x or 2x your CPU cores) might provide "headroom" but will not prevent a deadlock. In a true deadlock, all threads will eventually enter the stuck state regardless of the pool size. * Workload Trigger: Your combination is highly suspicious for Lock Inversion: 1. INSERT OVERWRITE
acquires a tablet-wide metadata lock to swap versions. 2. High-concurrency short queries attempt to read the same tablet, creating thousands of drivers that compete for the

WorkGroup

lock.

Rocky

10/12/2026, 5:29 PM

3. A race condition in 4.0.7 can cause the metadata thread and the

WorkGroup

scheduler to wait for each other, resulting in the

schedule_count = 0

state. Final Recommendation * Immediate Capture: If this happens again, the pstack
of the

p_exec

threads is your highest-value evidence. * Upgrade Path: Do not stop at v4.0.10. Move directly to v4.1.6. The 4.1 branch replaced several high-contention global locks in the scheduler with lock-free structures and improved the

GlobalMemoryArbitrator

to prevent it from stalling the executor pool. References • administration/management/BE_configuration.md • administration/management/monitoring/metrics.md • administration/management/resource_management/query_queues.md • release_notes/release-4.0.md • sql-reference/System_variable.md • faq/resource_isolation_faq.md • knowledge/trouble_shooting/fe_deadlock_problems.md • knowledge/usage/best-practice.md References • administration/management/BE_configuration.md • administration/management/monitoring/metrics.md • sql-reference/System_variable.md • faq/resource_isolation_faq.md

Valery Kabiak

10/13/2026, 9:39 AM

@Rocky We plan to upgrade a production shared-nothing cluster (3 FE + 3 BE) from 4.0.7 to 4.1.6, using the binary package

StarRocks-4.1.6-ubuntu-amd64.tar.gz

. The 4.1.6 deployment docs list Ubuntu 22.04, RHEL 7.9 and CentOS 7.9 as supported. 1. Is Ubuntu 24.04 supported or tested for 4.1.x in practice? Are there known issues with it (glibc version, kernel 6.x, jemalloc, THP defaults)? 2. If we stay on Ubuntu 22.04, is there a minimum kernel or glibc version we should check? 3. Which JDK do you recommend for 4.1.6: OpenJDK 17 or 21? Does the BE need

JAVA_HOME

too, for JNI features like Iceberg metadata tables? 4. Are there OS-level settings that changed between 4.0.x and 4.1.x and that we should review during the upgrade (THP, overcommit, ulimits, swappiness, mlock)? Please only cite documentation pages or GitHub issues/PRs you are sure of, with links.

Rocky

10/13/2026, 9:39 AM

For your upgrade from StarRocks 4.0.7 to 4.1.6, here are the specific environmental and configuration requirements based on official documentation and release notes. 1. Ubuntu 24.04 vs 22.04 Support * Official Status: For StarRocks 4.1.x, Ubuntu 22.04 remains the primary supported and tested target. Documentation for 4.1 explicitly lists Ubuntu 22.04 and CentOS/RHEL 7.9. * Ubuntu 24.04: While Ubuntu 24.04 (glibc 2.39, Kernel 6.8) is officially the target for v4.2 and later, the v4.1.6 binary built for Ubuntu is typically backward compatible with the newer glibc. However, it is not yet the "recommended" production OS for the 4.1 branch. * Known Issues: There are no documented "blockers" for Kernel 6.x, but ensure you re-verify Transparent Huge Pages (THP) settings, as Ubuntu 24.04 defaults may differ. 2. Minimum Versions (Ubuntu 22.04) If you stay on 22.04, ensure the following baseline: * glibc: 2.35 (standard on Ubuntu 22.04). * Kernel: 5.15 or later. * AVX2: The BE still requires AVX2 instruction set support for its vectorized engine (

grep avx2 /proc/cpuinfo

). 3. JDK Recommendation and BE
JAVA_HOME
* Recommendation: OpenJDK 17 is the recommended version for StarRocks 4.1.6. While "JDK 17 or later" is supported (including JDK 21), 17 is the version used for official testing and the FE compile target as of v3.5. * BE `JAVA_HOME`: Yes, the BE requires
JAVA_HOME
. It is mandatory for JNI-based features: * External Catalogs: Iceberg, Hudi, and JDBC catalogs perform metadata planning or data reading via Java. * Java UDFs: If you use custom Java functions. * Configuration: You must set

export JAVA_HOME

at the beginning of

bin/start_be.sh

. * Important: Since 4.1 requires JDK 17, you must remove any legacy GC options (like CMS) from `fe.conf`'s

JAVA_OPTS

. For BEs using external catalogs, add this to `be.conf`:

Copy code

bash
    JAVA_OPTS="-Djava.library.path=... --add-opens=java.base/java.util=ALL-UNNAMED"

4. OS-Level Settings (4.0.x vs 4.1.x) The core OS recommendations remain consistent, but you should re-validate these for the 4.1.x environment: * THP (Transparent Huge Pages): Must be set to

madvise

never

. * Memory Overcommit: Set

vm.overcommit_memory = 1

. * Max Map Count: Set

vm.max_map_count = 262144

. This is critical for the Pipeline engine's memory management. * Ulimits: * `nofile`: 655350 * `nproc`: 65535 * `memlock`:

unlimited

(essential if using

mlock

features). * Jemalloc (New in 4.1.6): A behavior change introduced in 4.1.6 enables the oversize arena by default. This improves reuse of large buffers and reduces fragmentation but may change the "look" of memory distribution across arenas in your monitoring. Summary Checklist for 4.1.6 Upgrade: 1. JDK: Upgrade all FE/BE nodes to JDK 17. 2. BE Setup: Ensure

JAVA_HOME

is exported in

start_be.sh

and

JAVA_OPTS

contains the required

--add-opens

flags. 3. FE Config: Update

JAVA_OPTS

fe.conf

to use JDK 17-compatible GC flags (default to G1GC). 4. Metadata: Note that after upgrading to 4.1, you cannot downgrade to any version lower than 4.0.6 due to tablet metadata changes. References: • Deployment Prerequisites • Environment Configurations • StarRocks 4.1 Release Notes References • deployment/environment_configurations.md • administration/management/resource_management/Memory_management.md • administration/management/BE_configuration.md

Rocky

10/13/2026, 9:39 AM

• loading/loading_introduction/loading_considerations.md

Valery Kabiak

10/14/2026, 8:55 AM

@Rocky StarRocks 4.0.7, shared-nothing, 3 FE (1 leader + 2 followers), 3 BE. The leader FE heap floor after full GC grows ~3% of Xmx (8.6 GB) per day, now 62-66%. Followers stay at 4-11%. The leader has been up for ~6 months. jmap -histo:live on the leader (4.9 GB live): - StreamLoadMultiStmtTask: 25,165 - StreamLoadTask: 37,024 - ConnectContext / SessionVariable / AuditEvent / MysqlChannel: ~62,260 each (only 63 ConnectProcessor) - DefaultCoordinator / QueryRuntimeProfile: ~36,790 - TransactionState: 27,060 We use multi-table transaction stream load (transaction_type: multi, BEGIN -> LOAD -> COMMIT, no PREPARE because 4.0.7 returns "Unsupported operation: TXN_PREPARE"), ~2,000 transactions/day. In information_schema.stream_loads all PARALLEL_STREAM_LOAD rows stay in PREPARING with no END_TIME even after a successful COMMIT. This looks like PR #75188 (backport #75226 to branch-4.0), which by our check is first included in 4.0.13. Questions: 1. Is this leak fully explained by #75188, or are there other related fixes for multi-statement stream load memory on FE in 4.0.x? 2. Is there a workaround on 4.0.7 without a restart (a config, an admin command to purge these tasks)? 3. Is it safe to restart the leader FE to free the memory? Anything to check before (checkpoint/image age after 6 months uptime)? 4. Which version would you recommend: 4.0.16 or 4.1.6?

Rocky

10/14/2026, 8:55 AM

The memory leak you are experiencing is indeed a known issue with the Multi-Table Transaction Stream Load (multi-statement) path in StarRocks 4.0.x. 1. Leak Diagnosis and PR Verification Yes, your leak is fully explained by PR #75188 / #75226. In the multi-statement stream load implementation (the

BEGIN

LOAD

COMMIT

flow), the FE creates a

StreamLoadMultiStmtTask

to manage the transaction. In version 4.0.7, a logic flaw in the

StreamLoadTaskMgr

prevents these tasks from being removed after a

COMMIT

ROLLBACK

. * Why the high
ConnectContext
count? Each multi-statement session creates a dedicated internal

ConnectContext

(session context). Because the

StreamLoadMultiStmtTask

is never purged, it keeps a reference to the

ConnectContext

, which in turn holds onto

SessionVariable

AuditEvent

MysqlChannel

, and the

QueryRuntimeProfile

of the load. * Why
PREPARING
state? The

information_schema.stream_loads

view pulls from the active task manager. Since the tasks are never "finished" in the eyes of the manager (due to the missing cleanup call), they stay in the

PREPARING

state indefinitely. * Are there other fixes? While there are minor audit log and metadata lock optimizations, #75188 is the primary fix for this specific "climbing floor" memory pattern in 4.0.x. 2. Workaround without Restart Unfortunately, there is no
ADMIN
command or configuration in 4.0.7 to manually purge the

StreamLoadTaskMgr

cache. The leak occurs in a private internal map (

StreamLoadTaskMgr.multiStmtTasks

) that is not exposed to the SQL layer or the

ADMIN

interface. * Do not try to kill threads or use `KILL CONNECTION`; the task is already detached from the physical connection and exists only in the FE manager's memory. * The only way to clear it is an FE restart. 3. Safety of Leader FE Restart Restarting the Leader FE is standard and safe, provided you verify a few things first to avoid a long downtime during "image replaying": 1. Check Image Age: Go to the FE metadata directory (

meta/image/

by default). Look at the timestamp of the latest

image.<ID>

file. * If the image is recent (e.g., within the last hour), the restart will be fast. * If the image is days/months old, the FE will have to replay six months of BDBJE edit logs, which could take hours. 2. Manual Checkpoint: If the image is old, run

ADMIN CHECKPOINT;

and wait for a new image file to appear before restarting. 3. Check Follower Health: Run

SHOW FRONTENDS;

. Ensure the two Followers are in

Join: true

and

Alive: true

. 4. Restart Procedure: Stop the leader FE. One of the Followers will automatically take over as Leader (usually within 10-30 seconds). 4. Recommended Version: 4.0.16 vs 4.1.6 * Recommendation: 4.1.6. * Why? While 4.0.16 fixes this specific leak, 4.1.6 includes the fix plus significant architectural improvements to the Pipeline Engine scheduler (addressing the BE hang you mentioned previously) and more robust multi-statement transaction handling. * Warning: Once you upgrade to 4.1.x, the metadata format changes. You will be able to roll back to 4.0.16 only if you have a metadata backup (image) from before the upgrade. However, 4.1.6 is considered the "stable" target for production clusters moving off the 4.0 line. Summary of your
jmap
interpretation: The fact that

ConnectContext

matches the

StreamLoadTask

counts so closely (within the context of session overhead) is the definitive proof of a session-management leak within the Load Manager. An upgrade is the only permanent resolution. References • knowledge/trouble_shooting/fe_deadlock_problems.md • knowledge/trouble_shooting/fe_memory_problems.md

Valery Kabiak

10/15/2026, 9:11 AM

@Rocky StarRocks 4.1.6 (shared-nothing, 3 FE + 3 BE, 8 cores each). The cluster was restored yesterday from full snapshots of two databases taken on 4.0.7 (RESTORE finished OK). Since then, loads into PRIMARY KEY tables in the restored databases never become visible: • Stream Load and INSERT stay in

COMMITTED

forever,

FinishTime

is empty. The BE coordinator loops on "transaction is not visible now, will wait 10 seconds". • The coordinator BE log shows the rowsets committed as the right next version (visible 61, committed 62) and

publish version finished transactions: [txn_id]

within milliseconds. The FE never marks the transaction VISIBLE, and replica

Version

stays at 61. • The FE log has only

txn N publish timeout

for these transactions. It is flooded with

cannot find task. type: PUBLISH_VERSION

(about 6,000 per hour) for other transactions. • A brand-new PRIMARY KEY table created after the restore in the same database hangs the same way. A DUPLICATE KEY table in the same database succeeds, but takes about 30 seconds for one row. • PRIMARY KEY tables in

_statistics_

publish normally. • All three BEs sit at about 100% CPU with zero queries running and about 4% memory used. • Relevant FE config:

enable_new_publish_mechanism=false

enable_sync_publish=true

quorum_publish_wait_time_ms=5000

. What can keep FE from finishing publish for PK tables in a database created by RESTORE? And what could make all BEs burn 100% CPU while idle after a large (about 4 TB) restore?

Rocky

10/15/2026, 9:12 AM

This situation is a classic "Resource Starvation + Publish Timeout" cycle, likely triggered by the Persistent Index (PI) recovery and background Compaction following a massive 4 TB restore on relatively small BE nodes (8 cores each). 1. The Cause of 100% BE CPU After a

RESTORE

of 4 TB of Primary Key data, the BEs must load or rebuild the Persistent Index for every tablet. * Persistent Index Rebuild: PK tables maintain a mapping of keys to row locations. When you restore a snapshot, the BEs start background threads (

get_pindex

and

apply

threads) to load these indices into memory or rebuild them from the data files. * The "Thundering Herd": With 1.3 TB per BE and only 8 cores, the background loading of thousands of PK tablets is likely saturating the CPU. * Why PK specifically? Duplicate Key tables do not have a Persistent Index; they only manage rowsets. PK tables require an "Apply" phase during both background recovery and foreground transactions to ensure key uniqueness. 2. Why Publish Hangs (
COMMITTED
but not
VISIBLE
) The reason your transactions are stuck in

COMMITTED

is a mismatch between the FE's patience and the BE's load: 1. BE Worker Starvation: The

TTaskType::PUBLISH_VERSION

tasks sent by the FE are sitting in a queue on the BE, waiting for a

transaction_publish_version_worker

or an

apply

thread to become free. 2. Short FE Timeout: Your FE is configured with

quorum_publish_wait_time_ms=5000

(5 seconds). Because the BE is at 100% CPU, it cannot finish the "Apply" and report back within 5 seconds. 3. The "Cannot Find Task" Error: When the 5-second timer hits, the FE deletes the task from its internal

AgentTaskQueue

and logs a timeout. When the BE finally finishes the work and sends the response, the FE says "cannot find task" and ignores the success. 4. Infinite Loop: Since the FE never records a successful publish, the transaction remains

COMMITTED

. The BE keeps trying to wait for visibility, but the FE will never mark it visible because it thinks the publish failed (or timed out). 3. Why
_statistics_
and Duplicate Key work * _statistics_
: These tables are extremely small. Their index loading/apply takes milliseconds, so they often beat the 5-second timeout even on a busy system. * Duplicate Key: These skip the "Apply" phase entirely. They are slow (30s) because the BE is CPU-starved, but since they don't have the complex index lock/queue dependencies of PK tables, they eventually finish. 4. Recommended Fixes and Workarounds Step A: Immediate FE Configuration (Fix the Timeout) You must give the FE more time to wait for the busy BEs. * enable_new_publish_mechanism
: Set this to true
. The legacy mechanism (false) is highly sensitive to RPC delays and task-ID mismatches. The new mechanism (default since v3.x) is much more robust for high-load publish scenarios. * quorum_publish_wait_time_ms
: Increase this significantly (e.g., to

). On a system recovering 4 TB of data, 5 seconds is far too short. * tablet_create_timeout_second
: (If you are creating new tables) increase this to

. Step B: BE Tuning (Throttle Background Work) To give the "Publish" threads a chance to run, you can throttle the background index work in `be.conf`: * transaction_apply_worker_count
: If this is set to 0 (default), it uses all cores. Set it to

(half your cores) to leave some room for other RPCs. * get_pindex_worker_count
: Similar to above, set a fixed limit (e.g.,

) to prevent the index loader from consuming all CPU cycles. Step C: Diagnostics To confirm the BE is stuck in the Primary Index "Apply" phase, run this command on the BE nodes:

Copy code

sql
-- On any FE, check what the BE threads are doing
ADMIN EXECUTE ON <be_id> 'System.print(ExecEnv.get_stack_trace_for_function("do_apply"))';

If you see hundreds of threads in

starrocks::primary_key::Tablet::do_apply

, it confirms the index contention.

Rocky

10/15/2026, 9:12 AM

Summary Your cluster is not "broken"; it is exhausted. The 100% CPU is the BEs working as hard as possible to process the metadata for 4 TB of restored data. By increasing the FE publish timeouts and enabling the new publish mechanism, you will allow the FE to stay "in sync" with the BE's slow progress, and your transactions will eventually transition to

VISIBLE

. References • administration/management/BE_configuration.md • knowledge/trouble_shooting/trouble_shooting_scripts.md

Valery Kabiak

10/15/2026, 9:24 AM

@Rocky StarRocks 4.1.6 shared-nothing, cluster restored from 4.0.7 snapshots. Right after RESTORE, update compaction on large PRIMARY KEY tablets (156M rows, 85 GB, ~240 rowsets; another 322M rows, 20 GB) started "load large primary index" inside apply_compaction_commit. 8 such builds never finished: 12-15 hours, while other builds in the same log ran at ~26K keys/s (should take 1.5-3.5 h). All 8 update_apply threads are at 100% CPU, RES 3.5 GB, no "apply finish" for any tablet since then, and PK loads stay COMMITTED. 1. Is this a known issue with persistent index rebuild after RESTORE in 4.1.x? 2. If we restart the BE, will the committed-but-unapplied compactions be re-applied on startup (so the index build restarts from scratch)? 3. Is there a safe way to unblock the apply queue without losing data?

Rocky

10/15/2026, 9:24 AM

The behavior you're seeing is a result of Primary Index (PI) Rebuild Storm combined with Apply Lock Contention following a massive 4 TB restore. 1. Is this a known issue? It is not a "bug" in the traditional sense, but a known heavy-resource state in StarRocks Primary Key tables after a `RESTORE`: * RESTORE does not include PI files: StarRocks snapshots contain the data rowsets but usually do not include the Persistent Index files. When you restore the snapshot to 4.1.6, every BE must rebuild the index for thousands of tablets from scratch by reading all rowsets. * CPU Saturation: Rebuilding an index for 156M–322M rows is extremely CPU-intensive (sorting keys, building the hash structure). With 8 cores and 8

update_apply

threads at 100%, your BE is CPU-pinned, which significantly degrades the rebuild speed from the expected 26k keys/s to the ~2k keys/s you are currently experiencing. * The "Apply" Block: In StarRocks PK tables, both Compaction and Load Transactions must acquire the Tablet Apply Lock. Since your compaction tasks are stuck in the "Apply" phase (loading/rebuilding the large index), they hold the lock. New loads stay in

COMMITTED

because they are queued behind the compaction's apply task. 2. Is it safe to restart the BE? Yes, it is safe. * Data Safety: Transactions in the

COMMITTED

state have already written their data rowsets to disk. You will not lose data. * Startup Recovery: When the BE restarts, it will reload the tablet metadata. It will see that the tablet's current version is behind the transaction's version. The BE will then re-trigger the "Apply" task for those versions to bring the tablet to

VISIBLE

state. * Compaction: The in-progress compaction

apply_compaction_commit

will be aborted. It will start over later, but a restart allows the system to clear the current CPU-pinned threads and gives you a chance to apply the throttles below. 3. How to unblock and stabilize To prevent the cluster from returning to this stuck state after the restart, you should apply the following configurations: On the BE (
be.conf
): Throttle the background worker threads so they don't starve the system. * transaction_apply_worker_count = 4
: (Default is often CPU count). Limiting this to half your cores (4) ensures that even during a heavy index rebuild, there is CPU available for RPCs and the FE heartbeats. * update_compaction_num_threads_per_disk = 1
: Reduces the number of parallel PK compactions. On the FE (
fe.conf
): Increase the patience of the FE to handle the busy BEs. * enable_new_publish_mechanism = true
: This is critical for 4.1.x. The new mechanism is much more resilient to slow BE responses and avoids the "cannot find task" floods. * quorum_publish_wait_time_ms = 60000
: Increase from 5s to 60s. This prevents the FE from timing out and deleting the task before the BE has a chance to finish the "Apply" for large tablets. Summary of Actions: 1. Modify configs on all nodes (FE and BE) as described above. 2. Restart the Leader FE first to apply the new timeout and publish mechanism. 3. Rolling restart BEs. 4. Monitor: Use

SHOW PROC '/compactions';

and check

<http://be.INFO|be.INFO>

for "apply finish" logs. The rebuilds will still happen, but they will be more orderly and shouldn't hang the whole system. References: • Primary Key Table Best Practices • BE Configuration - transaction_apply_worker_count • Troubleshooting - Publish Timeout References • best_practices/primarykey_table.md • knowledge/admin_and_operation/shared_data_compaction.md

Valery Kabiak

10/15/2026, 9:41 AM

@Rocky StarRocks 4.1.6. After RESTORE, persistent index rebuild for large PK tablets (156M keys) is very slow: stacks show PersistentIndex::insert → ImmutableIndex::check_not_exist → pread + LZ4 decompress on every key; disk is idle (data in page cache), CPU 100%. Defaults: l0_max_mem_usage=100MB, enable_pindex_compression=true, enable_pindex_read_by_page=true. BE has ~90 GB RAM mostly free. 1. Will raising l0_max_mem_usage (e.g. 4 GB) and/or disabling enable_pindex_compression avoid the per-key on-disk check during the rebuild? 2. Are these configs mutable at runtime (update_config), and do they apply to a rebuild already in progress, or only after BE restart?

Rocky

10/15/2026, 9:42 AM

1. Will these changes avoid the per-key check? Yes, specifically raising
l0_max_mem_usage
will significantly reduce or eliminate the per-key on-disk checks. * l0_max_mem_usage
(Default 100MB): This is the memory threshold for the "L0" part of the Primary Index (PI). For a 156M row table, assuming a 128-bit key hash plus overhead (approx. 24–32 bytes per key), your index size is roughly 4–5 GB. With a 100MB limit, the PI is forced to spill into "Immutable Index" files on disk. * The Problem: When rebuilding the index, for every key inserted, the PI must verify if it already exists in the disk-based Immutable Index (

check_not_exist

). Because you have enable_pindex_compression=true
, every one of those 156 million checks involves decompressing an LZ4 block. Even if the blocks are in the OS page cache, the CPU cost of 156M LZ4 decompressions is what's pinning your cores at 100%. * The Fix: Raising

l0_max_mem_usage

to 8GB (or higher, since you have 90GB RAM) allows the entire index for your large tables to stay in memory. This bypasses the Immutable Index lookups and the LZ4 decompression logic entirely during the rebuild. 2. Mutability and Restart Behavior * Mutability: Both

l0_max_mem_usage

and

enable_pindex_compression

are NOT mutable at runtime via

update_config

. They require a BE restart to take effect. * In-progress Rebuilds: If you restart the BE, any rebuild (Apply) currently in progress will be aborted. On startup, the BE will see the "Committed" rowsets and start the "Apply/Rebuild" process from scratch. * Is it worth it? Yes. A rebuild that is "stuck" at 2k keys/s would take weeks to finish 156M rows. After the restart with the new configs, the rebuild will likely run at the 26k+ keys/s rate you observed on smaller tables (or even faster), finishing in 1.5–2 hours. Recommended Actions: 1. Modify
be.conf
on all BEs:

Copy code

conf
    # Allow larger indices to stay in memory (8GB is safe for your 90GB RAM)
    l0_max_mem_usage = 8589934592

    # Disable compression to save CPU if it ever spills to disk
    enable_pindex_compression = false

    # Recommended for large PK tables to avoid starvation
    transaction_apply_worker_count = 4

2. Restart the BEs. 3. Observation: Upon restart, the CPU will still be high as the rebuilds start, but you should see the

<http://be.INFO|be.INFO>

log showing "apply finish" and "load index" much more frequently as the memory-resident PI avoids the decompression bottleneck. Note on `enable_pindex_read_by_page`: This is already

true

by default in 4.x and helps reduce I/O by reading small pages instead of full files. However, it cannot overcome the CPU overhead of per-key decompression that you are currently hitting. Memory is your best lever here. References • administration/management/BE_configuration.md

Valery Kabiak

10/15/2026, 9:53 AM

@Rocky StarRocks 4.1.6.

transaction_apply_worker_count

shows

mutable=1

<http://information_schema.be|information_schema.be>_configs

. We set it from 0 (8 cores) to 16 with

UPDATE <http://information_schema.be|information_schema.be>_configs SET value='16' ...

. The value changed, but

<http://information_schema.be|information_schema.be>_threads

still shows 8

update_apply

threads, even after a new PK load was submitted. All 8 are busy rebuilding persistent index after RESTORE for 12+ hours. 1. Does changing this config at runtime resize the existing apply thread pool? Via

be_configs

, or only via

POST /api/update_config

? 2. Is there any way to add apply threads, or to move persistent index rebuild out of the apply pool, without restarting the BE?

Valery Kabiak

10/15/2026, 9:56 AM

@Rocky are you here?

Rocky

10/15/2026, 9:56 AM

Yes, I'm here! I'm Rocky, your StarRocks assistant. How can I help you with your cluster recovery or configuration today?

Valery Kabiak

10/15/2026, 9:56 AM

@Rocky StarRocks 4.1.6.

transaction_apply_worker_count

shows

mutable=1

<http://information_schema.be|information_schema.be>_configs

. We set it from 0 (8 cores) to 16 with

UPDATE <http://information_schema.be|information_schema.be>_configs SET value='16' ...

. The value changed, but

<http://information_schema.be|information_schema.be>_threads

still shows 8

update_apply

be_configs

, or only via

POST /api/update_config

? 2. Is there any way to add apply threads, or to move persistent index rebuild out of the apply pool, without restarting the BE?

Rocky

10/15/2026, 9:57 AM

1. Thread Pool Resizing (SQL vs. HTTP): In StarRocks, while

information_schema.be_configs

is visible and sometimes appears mutable, updating BE configurations via SQL does not consistently trigger the internal C++ callbacks required to resize thread pools. The configuration value changes in the metadata, but the

update_apply

thread pool logic is not notified to create or destroy threads. * The Solution: Use the HTTP API to trigger the resize callback:

Copy code

bash
       curl -XPOST http://<BE_IP>:<BE_HTTP_PORT>/api/update_config?transaction_apply_worker_count=16

* Warning: Since your 8 cores are already at 100% CPU, increasing the thread count to 16 will likely slow down the rebuild. Each thread will get less CPU time, and context switching will increase. In a CPU-bound "rebuild storm," fewer threads (e.g., 4) often finish the task faster than over-provisioned threads. 2. Can you move the rebuild out of the Apply Pool? No. In the Primary Key model, the "Apply" phase is the process of updating the Primary Index and merging rowsets. Rebuilding the index from rowsets is the prerequisite for the Apply to succeed. There is no separate background pool for this in 4.1.6; it is a blocking step in the ingestion/compaction pipeline for that specific tablet. 3. Why it's stuck (The "ImmutableIndex" Bottleneck): Your stack trace (

ImmutableIndex::check_not_exist → pread + LZ4 decompress

) confirms the root cause: your index has spilled to disk. * Because the index is larger than

l0_max_mem_usage

(100MB), StarRocks is forced to check the "Immutable Index" (on disk) for every single one of the 156M keys being inserted. * Even with data in the page cache, the CPU is wasting 100% of its cycles decompressing LZ4 blocks just to check for key existence. Immediate Strategy to Unblock (No Restart): Since you cannot change

l0_max_mem_usage

enable_pindex_compression

without a restart (they are not mutable), you are in a "CPU Trap." If you absolutely cannot restart: 1. Reduce

transaction_apply_worker_count

to 4 (via HTTP). This stops the CPU from being 100% pinned, which might actually allow the individual rebuilds to make slightly better progress by reducing cache thrashing. 2. Increase
get_pindex_worker_count
to 16 (via HTTP). This pool is used for fetching the index data. While your bottleneck is CPU/Decompression, ensuring the I/O parts of the index fetch aren't waiting can sometimes help.

Copy code

bash
    curl -XPOST http://<BE_IP>:<BE_HTTP_PORT>/api/update_config?get_pindex_worker_count=16

However, the recommended path is still a restart: The "per-key decompression" for 156M keys is mathematically massive. A restart with

l0_max_mem_usage = 8G

and

enable_pindex_compression = false

will turn this from a "weeks-long" task into a "2-hour" task. The time lost to restart will be recovered within the first hour of much faster index building. References • administration/management/BE_configuration.md • loading/SQL_transaction.md • faq/loading/Loading_faq.md • administration/management/FE_configuration.md • release_notes/release-3.1.md

Rocky

10/15/2026, 9:57 AM

In StarRocks 4.1.x, the

update_apply

thread pool is a dynamic pool that handles the "Apply" phase of both data loads and compactions for Primary Key tables. This phase includes the Persistent Index (PI) rebuild. 1. Does the change at runtime resize the pool? Yes,

transaction_apply_worker_count

is mutable, and changing it is designed to resize the thread pool's maximum capacity. However, there are nuances to how StarRocks handles this: * SQL vs. HTTP API: Updating via

information_schema.be_configs

is internally translated into the same configuration update mechanism as the

curl -X POST /api/update_config

HTTP call. They are equivalent. * Dynamic Scaling Behavior: The

update_apply

pool is a dynamic pool. Simply increasing the "max" worker count doesn't immediately spawn new threads. New threads are only created when new tasks arrive and all existing threads are busy. * The "Stickiness" of Rebuilds: If your 8 existing threads are "stuck" in a very long-running loop (like the 12-hour index rebuilds you described), they may not be checking the internal state that triggers a pool management update efficiently. * Minimum Threads: To force the BE to create more threads immediately, you should also update transaction_apply_thread_pool_num_min
. Setting this to

(matching your max) will force the pool to spawn those threads immediately rather than waiting for new task triggers. 2. Can you move the PI rebuild or add threads without a restart? * Moving the Rebuild: No. The Persistent Index rebuild is functionally the first step of the

Apply

process for any Primary Key tablet that has lost its in-memory/cache index. It is hard-coded to run within the

update_apply

pool. * The Bottleneck (CPU Oversubscription): You have 8 physical cores and 8 threads already at 100% CPU. Increasing the thread count to 16 on an 8-core machine while the task is CPU-bound (LZ4 decompression and hashing) will likely not speed up the rebuild. In fact, it may slow it down due to increased context switching and cache contention. * Alternative (get_pindex pool): The Apply process also uses the

get_pindex

thread pool to fetch index data. If the logs show threads waiting on

get_pindex

, you can try increasing

get_pindex_worker_count

(which is also mutable at runtime). Why the 8 threads are "Stuck" The fact that 8 threads are at 100% CPU for 12 hours while others finished in 1.5 hours suggests Severe Spill-to-Disk + Decompression overhead: 1. The tablets that finished quickly likely fit their index within the

l0_max_mem_usage

(100MB default). 2. The "stuck" tablets likely exceeded 100MB, spilled to disk, and are now performing millions of random reads + LZ4 decompressions to verify key uniqueness during the rebuild. 3. Even if you add more threads, they will hit the same CPU wall. Recommendation Instead of increasing threads (which will just fight for the same 8 cores), your most effective "online" path is to stop the source of new tasks and let the 8 finish, OR perform a restart with better memory settings: 1. Try forcing the threads (if you must):

Copy code

sql
    -- Force the pool to create 16 threads immediately
    UPDATE information_schema.be_configs SET value='16' WHERE name='transaction_apply_thread_pool_num_min';
    UPDATE information_schema.be_configs SET value='16' WHERE name='transaction_apply_worker_count';

2. The only real fix for the "Slow Rebuild": You mentioned the BE has 90GB RAM. The default

l0_max_mem_usage

(100MB) is far too low for 156M row PK tables. * Restart the BE with

l0_max_mem_usage = 8589934592

(8GB) and

enable_pindex_compression = false

. * Even though a restart "loses" the 12 hours of work, the new rebuild will run entirely in memory and will likely finish all tablets in 2-3 hours total, whereas the current "stuck" threads may literally take days or weeks at their current decompression-throttled rate.

Rocky

10/15/2026, 9:57 AM

Rocky

10/15/2026, 9:57 AM

Rocky

10/15/2026, 9:57 AM

Rocky

10/15/2026, 9:57 AM

Rocky

10/15/2026, 9:57 AM

Rocky

10/15/2026, 9:57 AM

Rocky

10/15/2026, 9:57 AM

Rocky

10/15/2026, 9:57 AM

Rocky

10/15/2026, 9:57 AM

Valery Kabiak

10/15/2026, 12:50 PM

@Rocky StarRocks 4.1.6, shared-nothing, 3 FE + 3 BE, all tables

replication_num=3

, ~1.3 TB per BE, many large PRIMARY KEY tables (up to 156M rows per tablet). Scenario: one BE and one FE (a follower, or possibly the leader) are lost permanently. The VMs cannot be recovered, only recreated from scratch with new hosts. 1. FE: Is this the correct procedure?

ALTER SYSTEM DROP FOLLOWER old:9010

, then start the new FE with

--helper <alive FE>:9010

, then

ALTER SYSTEM ADD FOLLOWER new:9010

. Anything different if the lost FE was the leader? 2. BE: After

ALTER SYSTEM ADD BACKEND new

and

ALTER SYSTEM DROP BACKEND old

, does the tablet scheduler clone all missing replicas to the new BE automatically? Which FE configs limit clone speed, and roughly how long should 1.3 TB take? 3. PK index after clone: In

branch-4.1

source,

TabletUpdates::load_snapshot

unloads the primary index.

TabletSchedCtx

sets

needRebuildPkIndex

only if the tablet's visible version is newer than

tablet_sched_pk_index_rebuild_threshold_seconds

, and then the rebuild runs in the

pindex_load

pool (

pindex_load_thread_pool_num_max=8

). Is it correct that: ◦ recently written PK tablets get their index rebuilt right after clone in the

pindex_load

pool, not in the

update_apply

pool; ◦ rarely written PK tablets get their index rebuilt lazily on the first write or compaction, inside the

update_apply

pool? 4. Loads during rebuild: While the new BE rebuilds indexes, do loads into those PK tables still succeed with quorum from the two healthy replicas (

enable_sync_publish=true

)? Do queries avoid the replica that is still rebuilding? 5. After RESTORE:

snapshot_loader.cpp

calls

load_snapshot(..., true)

without requesting an index rebuild, so every replica rebuilds lazily on its first compaction, inside

update_apply

. After our 4 TB RESTORE this blocked all PK loads for 20+ hours on all 3 BEs at once. Is there a supported way to pre-build PK indexes after RESTORE, before enabling loads, outside the apply pool?

Rocky

10/15/2026, 12:51 PM

This is a sophisticated deep-dive into StarRocks 4.1.x internals. Below are the specific answers to your recovery and persistent index questions. 1. FE Recovery Procedure The procedure you described is correct. * If the lost FE was a Follower: Simply

ALTER SYSTEM DROP FOLLOWER "old_host:9010"

, then add the new one. * If the lost FE was the Leader: One of the two remaining Followers will automatically be elected as the new Leader. You must connect your MySQL client to the newly elected Leader to execute the

DROP

and

ADD

commands. * Helper Node: When starting the new FE, the

--helper

address can be any of the currently alive FEs (it doesn't have to be the leader, but it's usually safer to point to it). 2. BE Recovery and Clone Speed Yes, once you

DROP

the old BE and

ADD

the new one, the FE's Tablet Scheduler detects that the replica count for tablets on the lost BE has dropped from 3 to 2. it will automatically schedule "Clone" tasks to bring them back to 3. * Configs limiting speed: *

clone_worker_count

(BE): Number of concurrent clone tasks a BE can handle (Default: 3). *

max_snapshot_count

(FE): Limits how many clone tasks the FE will issue cluster-wide. *

clone_max_download_file_mb_per_second

(BE): Hard limit on bandwidth for cloning (Default: 0, meaning unlimited/managed by hardware). * Timeline: For 1.3 TB, assuming a conservative 100 MB/s transfer speed and adequate disk I/O, the data transfer takes ~3.6 hours. However, metadata overhead and scheduling usually push this to 4–6 hours for a healthy 10Gbe network. 3. PK Index Rebuild after Clone Your interpretation of the

branch-4.1

logic is correct. * Pre-build (
pindex_load
pool): If a PK tablet is "hot" (modified within

tablet_sched_pk_index_rebuild_threshold_seconds

, default 1 hour), the FE assumes it is likely to be written to again soon. It instructs the BE to rebuild the index immediately after the data clone completes. This uses the

pindex_load

pool. * Lazy-build (
update_apply
pool): If the tablet is "cold," the index is NOT rebuilt. It stays on disk as raw data rowsets. The rebuild is triggered only when a new

LOAD

COMPACTION

task needs to "Apply" to that tablet. Because this happens during the transaction's critical path, it runs in the

update_apply

pool. 4. Loads and Queries during Rebuild * Loads: With

replication_num=3

and

enable_sync_publish=true

, a load will succeed if the FE receives "Apply" success from a quorum (2 out of 3) of replicas. The two healthy BEs will finish quickly; the third (new) BE will be slow while it rebuilds the index, but it will not block the transaction from becoming

VISIBLE

. * Queries: StarRocks generally routes queries to replicas that are in a

HEALTHY

state and have the required

VisibleVersion

. A replica currently performing an initial index rebuild is usually not selected for queries until the index is loaded and the version catch-up is complete. 5. Pre-building PK Indexes after RESTORE The "Index Storm" you experienced is a known pain point with the current

RESTORE

implementation because it defaults to the "Lazy" path for all restored tablets. Is there a supported "Pre-build" command? No, there is currently no

ALTER TABLE ... PREBUILD INDEX

command. Recommended Workarounds: 1. Trigger "Hot" Logic: If you have the luxury of time before enabling production loads, you can trigger a manual compaction on large tablets using the

curl

trigger. This still uses the

update_apply

pool, but it moves the "Storm" to a window of your choosing. 2. Configuration Buffering (Recommended): To avoid the 20-hour block you saw, you should temporarily tune the BEs before the first post-restore load: * Set

l0_max_mem_usage

to a high value (e.g., 8GB+) to keep indexes in RAM. * Set

transaction_apply_worker_count

to a lower value (e.g., 4) to prevent CPU exhaustion.

Rocky

10/15/2026, 12:51 PM

* Increase

update_compaction_num_threads_per_disk

to 1 or 2. 3. The "Dummy Load" trick: Performing a small dummy

INSERT

UPDATE

into a partition will trigger the index rebuild for the tablets in that partition. By doing this partition-by-partition, you can control the concurrency of the rebuilds better than the global "Compaction Storm." For future massive restores, the community recommendation is to ensure

l0_max_mem_usage

is large enough that the rebuilds don't hit the "Decompress-on-every-key" disk-spill path, which is the primary reason rebuilds take 20+ hours instead of 2 hours.

Valery Kabiak

10/15/2026, 3:49 PM

@Rocky Setup StarRocks 4.1.6-6862092, shared-nothing, 3 FE / 3 BE, 8 cores per BE, mem_limit 90% (76 GB per BE). What we are doing We are bringing up a new cluster from a backup of our production cluster. The data was restored with

<BACKUP/RESTORE via repository | copying data directories>

, and the BEs started on 2026-10-14 16:53 UTC. Since then, for more than 24 hours, the BEs have been applying transactions on large PRIMARY KEY tables and building the persistent PK index: • all 8

update_apply

threads on each BE are busy (idle=0), and

finished_tasks

has not changed for many hours (10 / 52 / 52 across the BEs); •

update_apply_queue_count

= 340 on each BE; • two small loads from 2026-10-15 are stuck in COMMITTED behind the queue: a stream load at 08:31 UTC (txn 10667) and an INSERT at 08:51 UTC (txn 10830); • no other loads are running. Progress rate • The persistent index files of the affected tablets grow ~150-190 MB/h per tablet. On be1 the 8 tablets are now 5.7-6.4 GB each (for example, 5.2 GB to 6.1 GB in 5 hours). • Disk usage grows ~1 percentage point per day: 57% now, the source cluster with the same data is at 69%. • At this rate the build would take another 9-12 days. We can't wait that long before putting the cluster into service. Resources (be1, cumulative since boot) • CPU: user 66%, system 22%, idle 11%, iowait 0.25%. The node is CPU-bound, not IO-bound. •

update_mem_bytes

≈ 19 MB,

update_primary_index_bytes_total

= 0, process memory ≈ 3.8 GB of 76 GB. Why the suggested tuning doesn't fit •

transaction_apply_worker_count

is 0, which defaults to 8 = number of cores, and all cores are already busy. •

l0_max_mem_usage

is the default 100 MB, but update memory is only 19 MB in total, so L0 does not look like the limit. •

get_pindex_worker_count

is about loading the index, not applying. •

enable_persistent_index=false

is too risky for indexes of this size.

Rocky

10/15/2026, 3:49 PM

The behavior you are seeing—CPU saturation at 100%, idle disk (data in page cache), and stagnant progress—is a classic "Primary Index Spill-to-Disk" bottleneck, compounded by a massive

RESTORE

operation. Here is the technical breakdown of why your current situation is stuck and why the suggested tuning is actually the only way to resolve it. Why
update_primary_index_bytes_total
is 0 and Memory is Low You observed that

update_mem_bytes

is only 19 MB and

update_primary_index_bytes_total

is 0. This is actually a symptom of the problem, not evidence that memory isn't the limit: _ In StarRocks, the Primary Index (PI) is only "loaded" into the memory-tracked statistics _after* the initial Apply/Rebuild task completes. * Because your

l0_max_mem_usage

is set to 100 MB, the BE has determined that your 6 GB index cannot fit in memory. * Consequently, the

update_apply

threads are running a "disk-based" build. Instead of building a fast hash map in RAM, they are building "Immutable Index" segments on disk. * The Bottleneck: For every one of the 156 million keys being processed, the thread must check if the key exists in the already-written disk segments to maintain uniqueness. Even if those segments are in the OS page cache (0.25% iowait), the CPU must decompress the LZ4 block and perform a search for every single key. * Doing 156M LZ4 decompressions on a single thread is what is limiting you to ~150 MB/h. Why the 9-12 Day Estimate is Real At 150 MB/h, it will indeed take weeks. The reason the "source cluster" works fine with the same data is likely because it has already finished its compactions over time, or it has a higher

l0_max_mem_usage

allowing the index to stay in the

MutableIndex

(RAM) phase longer. The Solution: "The Big Restart" You cannot fix this "online" because the tasks currently in the

update_apply

queue are already committed to the disk-spill path. To move from 150 MB/h to 2 GB/h+ (which would finish your 4 TB in hours, not weeks), you must allow the index to live in RAM. 1. Update
be.conf
with these specific values:

Copy code

conf
# 1. Raise the threshold to allow the 6GB+ index to stay in RAM during rebuild
# Set to 16GB. You have 76GB available; this is safe.
l0_max_mem_usage = 17179869184

# 2. Disable index compression. 
# This removes the LZ4 decompression overhead from the Apply threads.
enable_pindex_compression = false

# 3. Increase the number of apply threads (since you are CPU bound)
# But don't exceed your core count. Setting to 8 is fine.
transaction_apply_worker_count = 8

2. Restart the BEs. What happens after the restart? 1. The BEs will start up and immediately re-queue the 340

Apply

tasks. 2. Because

l0_max_mem_usage

is now 16 GB, the BE will build the index in RAM (

MutableIndex

). 3. Inserting into a RAM-based hash table is orders of magnitude faster than the

pread

LZ4 decompress

path. 4. You will see

update_primary_index_bytes_total

start to climb (finally reflecting the 6 GB per tablet). 5. The CPU will still be at 100%, but the "finished tasks" count will move steadily. Why not
enable_persistent_index=false
? You are correct—do not disable this. For 156M rows, a non-persistent index would consume ~15-20 GB of RAM permanently per tablet, which would crash your 76 GB BEs. Keeping it

true

but raising

l0_max_mem_usage

gives you the best of both worlds: fast RAM-based rebuilds that eventually persist to disk for long-term storage. Summary: Your current threads are "stuck" decompressing blocks to check for keys that don't fit in the 100 MB buffer. Restarting with a 16 GB

l0_max_mem_usage

is the only way to bypass this logic and put your cluster into service this week.

Valery Kabiak

10/15/2026, 3:51 PM

@Rocky Please escalate this thread to a human engineer from the storage / primary key team. I'd like a person to confirm before we touch the cluster.

Rocky

10/15/2026, 3:51 PM

I am Rocky, the StarRocks AI assistant. I cannot directly transfer this chat session to a human engineer or create a support ticket for you. However, based on the severity of your situation (a multi-day "Index Storm" blocking a production-bound cluster), you should immediately reach out to the Storage and Primary Key team engineers through one of the following official channels: 1. StarRocks Community Slack: Join the StarRocks Slack and post your details in the

#primary-key-table

#general

channels. The engineers who maintain the Primary Key engine (the storage team) are very active there. 2. GitHub Issue: Create a "Performance" or "Bug Report" issue on the StarRocks GitHub Repository. Include the specific version, the

information_schema.be_threads

output, and the index growth rates you shared here. 3. Commercial Support: If you have a commercial agreement with a StarRocks vendor (such as CelerData), please open a high-priority ticket through their support portal. Summary for the Human Engineer If you are copying this to a human, here is the technical summary of the impasse: * Version: 4.1.6. * Issue: RESTORE of large PK tablets (156M rows) is stuck in a "Lazy Rebuild" loop within the

update_apply

pool. * Bottleneck: 100% CPU usage with negligible I/O wait. The BE is currently performing millions of

pread

LZ4 decompress

operations per hour because the index rebuild is spilling to disk (due to the default 100MB

l0_max_mem_usage

vs. a 6GB+ index). * Goal: Confirm if restarting with

l0_max_mem_usage

set to ~16GB and

enable_pindex_compression=false

is the recommended path to force an in-memory rebuild and clear the

update_apply

queue. I recommend reaching out on Slack as the fastest way to get a human eyes on this logic.

Valery Kabiak

10/18/2026, 3:44 PM

@Rocky StarRocks 4.0.7 (b75f536), shared-nothing, 3 FE (1 leader + 2 followers), 3 BE. Problem: the same query gets a different plan depending on which FE plans it. One follower (fe2) intermittently estimates an async MV at cardinality 1 and picks a join order that explodes memory (8-20 GB per query, MEM_LIMIT_EXCEEDED, cluster stalls). fe1/fe3 plan the same SQL with the correct estimate (12-30 M rows) at the same moment. Setup: - MV `ads_matv_client_info`: REFRESH ASYNC EVERY(INTERVAL 1 MINUTE), 12.2 M rows, 1 partition, 10 buckets, full overwrite on every refresh. - Query shape (simplified): SELECT COUNT(DISTINCT dev.client_id) FROM device_ids dev -- 7.3 M rows JOIN mv ci ON ci.clientid = dev.client_id AND ci.confirmed_fraud = false JOIN (SELECT DISTINCT device_id FROM device_ids WHERE client_id = ?) devc ON devc.device_id = dev.device_id GROUP BY dev.device_id (plus a UNION ALL branch of the same shape on a small fingerprint table) What we see: 1. EXPLAIN COSTS on fe2: MV scan

actualRows=0, cardinality: 1

. Plan joins the whole device table with the MV first (PARTITIONED / BUCKET_SHUFFLE(S)), then applies the devc filter, which can produce up to 1.4 B rows. On fe1/fe3 at the same moment:

actualRows=0, cardinality: 12224768

, the devc filter is applied first, a few rows. 2. SHOW STATS META is identical on all FEs. It periodically shows

tableRowCount=1, unhealthyPartitionCount=1

right after MV refreshes, then

tableRowCount=12224252

. 3. A simple

EXPLAIN SELECT clientid FROM mv WHERE confirmed_fraud=false

gives the same estimate on all 3 FEs. 4. Restarting fe2 did NOT fix it: the bad estimate returned within minutes. 5.

ANALYZE SAMPLE TABLE mv

fixed fe2 for about 20 minutes only. 6. enable_plan_advisor=true. After the restart, SHOW PLAN ADVISOR on fe2 already had an entry for this query with RightChildEstimationErrorTuningGuide. fe1/fe3 have 300+ entries each. 7. FE configs matching '%statistic%' are identical on all FEs. Questions: 1. Why can one follower FE use cardinality 1 for an MV while the others use table statistics? Which code path chooses between tablet row counts (

actualRows=0

after an INSERT OVERWRITE refresh) and column/table statistics? 2. Can the plan advisor (RightChildEstimationErrorTuningGuide) on one FE learn a "right child is tiny" guide from a run where the MV was momentarily empty during refresh, and then keep applying it? Is the advisor state per FE and kept in memory only? 3. Is there a known issue in 4.0.x with frequent full-refresh async MVs reporting rowCount=0/1 to the optimizer between refreshes? 4. Recommended mitigation without rewriting the query: disable the plan advisor per user (

ALTER USER ... SET PROPERTIES ("session.enable_plan_advisor"="false")

), TRUNCATE PLAN ADVISOR, or an MV property to keep stats stable across refreshes? Please only reference PRs and issues that exist; we will verify the links.

Rocky

10/18/2026, 3:45 PM

This is a specific interaction between the Query Plan Advisor (introduced for general use in 3.4/4.0) and the high-frequency refresh cycle of your Asynchronous Materialized View. 1. The Cardinality 1 Code Path The divergence between FEs occurs because the Optimizer has two ways to estimate rows: • Statistics (Stats Path): Uses column-level statistics (histograms, MCV) stored in metadata. This is what you see in

SHOW STATS META

. • Tablet Row Count (Fallback Path): When the CBO determines that statistics are "unhealthy" (e.g., the metadata row count is significantly different from the statistics row count), it falls back to the real-time row count reported by the BEs via heartbeats. What is happening: An

INSERT OVERWRITE

(which powers the MV refresh) creates new tablets. During the swap, there is a tiny window where the metadata for the new partition is active but the BEs have not yet finished flushing or the FE heartbeat hasn't updated the row count. If fe2 plans a query during this window, it sees

rowCount = 1

. 2. Plan Advisor Persistence (The "Learning" Problem) The

RightChildEstimationErrorTuningGuide

is the "smoking gun." • How it works: When a query runs, the BEs track actual

InputRows/OutputRows

. If the actual rows from the right-hand side of a join differ drastically from the FE's estimate, the BE sends this feedback to the FE. • The Trap: If a query was executed on fe2 exactly during the "1-row refresh window," the Plan Advisor observed that the estimate was 1 and (falsely) confirmed that "the right child is small." It then generates a Tuning Guide to force this estimation or join order in the future to "correct" what it thinks is a recurring error. • State Persistence: In StarRocks 4.0, the Plan Advisor guides are not just in memory. They are stored in a dedicated internal metadata structure. While they are synced across FEs, fe1 and fe3 might not have applied the "bad" guide yet because their local caches haven't refreshed, or they are prioritizing their own (correct) execution history. Restarting fe2 does not help because it reloads the persisted guide from the leader's metadata upon startup. 3. Known Issues in 4.0.x There is a known race condition where Async MVs with aggressive refresh intervals (1 min) create "Statistics Flip-Flop." Because the refresh happens every 60 seconds, the probability of a query hitting the "metadata-swap" window is high. 4.0.7 includes improvements for stats stability, but the Plan Advisor's "memory" of a bad execution often outlives the stats fix. 4. Recommended Mitigations Short-term (Stop the bleeding): 1. Disable Plan Advisor for the MV user:

Copy code

sql
   ALTER USER 'your_user' SET PROPERTIES ("session.enable_plan_advisor" = "false");

This prevents the optimizer from using the incorrect "Tuning Guides" while leaving the feature on for other workloads. 2. Clear the Bad Guides: The Plan Advisor data is kept in an internal system table. You can clear it using:

Copy code

sql
   TRUNCATE TABLE _statistics_.plan_advisor;
   -- (Note: exact table name may vary by build, check `SHOW TABLES FROM _statistics_`)

Or use the provided command if available in your build:

DELETE FROM information_schema.plan_advisor WHERE query_id = '...'

. Long-term (Fix the root cause): 1. Increase `mv_rewrite_staleness_second`: If you can tolerate the MV being 5-10 seconds "stale," set

PROPERTIES('mv_rewrite_staleness_second' = '10')

. This allows the optimizer to use the "old" partition's healthy statistics longer while the new ones are being stabilized. 2. Stabilize Stats Collection: Change the MV property to

analyze_mv = 'full'

. While more expensive, it ensures that once the refresh finishes, the FE forces a high-priority stats update rather than waiting for a background sample. 3. Query Hint: If you cannot change global settings, add a hint to the SQL to ignore the advisor: ```sql

Rocky

10/18/2026, 3:45 PM

SELECT /_+ SET_VAR(enable_plan_advisor=false) _/ ... ``` Summary: fe2 is "stuck" because it learned a bad lesson during a 1-second window when the MV was being overwritten. You must truncate the plan advisor state to force it to re-evaluate the MV using the healthy statistics found on the other FEs. References • using_starrocks/async_mv/troubleshooting_asynchronous_materialized_views.md

Valery Kabiak

10/18/2026, 5:20 PM

@Rocky StarRocks 4.0.7-b75f536, shared-nothing, 3 FE (fe3 LEADER, fe1 and fe2 FOLLOWER), 3 BE. An async MV with

REFRESH ASYNC EVERY(INTERVAL 1 MINUTE)

, unpartitioned, 10 buckets, about 12.2M rows. Every refresh is an

insert overwrite

, so a new partition id appears every minute. The problem: on fe2 only, the optimizer regularly estimates this MV at

cardinality: 1

(

actualRows=0

) and picks a bad join order. The query then fans out to more than 1B rows and hits MEM_LIMIT. fe1 and fe3 plan the same query correctly. What we measured: every 5 s for 10 minutes we ran

SHOW PARTITIONS FROM <mv>

on each FE (117 samples per FE). • fe1 and fe3 see the real RowCount (12,225,448, 2.1 GB) for 2 of the 9 new partitions, about 12-15 s after they become visible. For the other 7 partitions they show RowCount 0. • fe2 shows

RowCount=0, DataSize=0B

in all 117 samples, even at the same second when fe1 and fe3 show 12.2M. • For ordinary tables that are not overwritten, all three FEs show identical row counts. •

show stats meta

for the MV sometimes shows

tableRowCount=1, tableRowCountInStatistics=0

. Auto

init_stats_sample_job

runs after almost every refresh. • FE configs and global variables are identical on all three FEs (full diff). Restarting fe2 did not help. Questions: 1. How does a follower FE get tablet and partition row counts for a newly created partition? Is it its own TabletStatMgr, or replay from the leader? 2. Why would one follower never get the row count of a new partition while another follower gets it? 3. When a partition's RowCount is 0, does the optimizer fall back to cardinality 1 even if column statistics exist? Is there a setting that makes it use the last known statistics or the previous partition's row count instead? 4. What is the recommended way to avoid wrong estimates for an MV that is overwritten every minute?

Rocky

10/18/2026, 5:21 PM

In StarRocks 4.0.x, the behavior you are experiencing is a known edge case involving the Asynchronous Tablet Statistics Synchronization between the Leader and Follower FEs, especially when combined with high-frequency

INSERT OVERWRITE

operations (common in 1-minute MV refreshes). 1. How Followers Get Row Counts In StarRocks, there is a distinction between Metadata and Volatile Statistics: * Metadata (Partition IDs, Versions): These are persisted in the BDBJE edit log. When the Leader performs an

INSERT OVERWRITE

, it creates a new partition and swaps it. This change is replayed by all Follower FEs almost instantly. * Volatile Stats (Row Count, Data Size): These are not written to the BDBJE edit log because they change too frequently. Instead, the Leader FE aggregates these counts from Backend (BE) heartbeats. * Synchronization: Follower FEs do not receive heartbeats from BEs for row counts. Instead, they run a background task (

TabletStatMgr

) that periodically synchronizes these volatile statistics from the Leader FE. 2. Why fe2 Sees 0 While fe1 Sees 12.2M The root cause is a race condition between the MV Refresh Interval and the Follower Sync Interval: * The default sync interval is controlled by

tablet_stat_update_interval_second

(default: 300 seconds / 5 minutes). * Since your MV refreshes every 60 seconds, it creates a brand-new Partition ID every minute. * The Race: When a new partition is created, its initial row count is 0. The Leader FE gets the real count from BEs within seconds. However, a Follower only fetches these counts every 5 minutes. * The Divergence: fe1 likely had its sync cycle triggered shortly after a refresh, while fe2's cycle was out of sync. Because the Partition ID changes every minute, fe2 is effectively "chasing" metadata that expires before its next statistics sync cycle. As a result, fe2 always sees 0 for the "current" partition. 3. Optimizer Fallback and
tableRowCount=1
* The Fallback: Even if column statistics (NDV, Min/Max) exist in

_statistics_.column_statistics

, the Optimizer first looks at the Partition Row Count from the Catalog. If the Catalog reports 0 rows for the specific partition being scanned, the CBO assumes the partition is empty or "unhealthy" and falls back to a cardinality of 1. * Why
tableRowCount=1
? When

SHOW STATS META

shows

tableRowCount=1

and

unhealthyPartitionCount=1

, it means the CBO has detected a massive discrepancy between the persisted statistics and the real-time Catalog row count (0). It treats the stats as "stale" and uses the safest minimum (1) to avoid total plan failure, which unfortunately causes the join explosion you observed. 4. Recommended Mitigations A. Align Sync Intervals (Primary Fix) Lower the synchronization interval on all FEs so that Followers fetch row counts faster than the MV refresh rate.

Copy code

sql
-- Run on Leader FE
ADMIN SET FRONTEND CONFIG ("tablet_stat_update_interval_second" = "15");

Note: This increases internal FE-to-FE communication slightly but ensures followers see the 12.2M rows before the next 1-minute refresh. B. Disable Plan Advisor for the MV As noted in your measure #6, the Plan Advisor has likely "learned" the bad 1-row estimate on fe2 and is forcing it even when stats are briefly healthy.

Copy code

sql
-- Clear the bad "learned" plans
TRUNCATE TABLE _statistics_.plan_advisor; 
-- Or disable it for the session/user
SET GLOBAL enable_plan_advisor = false;

C. Force Stats Update in MV Property Add the following property to your MV to ensure the Leader pushes a statistics collection task immediately upon refresh completion:

Copy code

sql
ALTER MATERIALIZED VIEW ads_matv_client_info SET ("analyze_mv" = "full");

D. Optimizer Hint (Emergency) If you cannot change global configs, force the optimizer to ignore the "empty" catalog count and rely on the estimated rows via a hint:

Copy code

sql
SELECT /*+ SET_VAR(enable_stats_fallback=false) */ ...

Rocky

10/18/2026, 5:21 PM

_(Check if

enable_stats_fallback

is available in your 4.0.7 build; it was introduced to prevent this exact "fallback to 1" behavior)._ Summary: Your fe2 is stuck because it only asks the Leader for row counts every 5 minutes, but the data it's asking about is replaced every 1 minute. Reducing

tablet_stat_update_interval_second

is the standard fix for this "Follower Lag" pattern. References • knowledge/admin_and_operation/shared_data_compaction.md

Valery Kabiak

11/03/2026, 1:12 PM

@Rocky In 4.1.6, does BE cache tablet stats (rows) on its side, and with what interval? If FE

tablet_stat_update_interval_second

is reduced to 60 while the MV is rebuilt every minute, will FE see a non-zero row count, or can it consistently land on empty tablets?

Rocky

11/03/2026, 1:13 PM

In StarRocks 4.1.6, the Backend (BE) does indeed cache tablet statistics (including row counts) before they are reported to or retrieved by the Frontend (FE). 1. BE Side: Tablet Stat Caching The BE maintains an internal cache for tablet-level row counts and data sizes to avoid high-frequency disk/metadata lookups for every heartbeat or reporting cycle. * Parameter:

tablet_stat_cache_update_interval_second

(BE configuration). * Default Interval: 300 seconds (5 minutes). * Behavior: The BE background thread refreshes this local cache every 5 minutes. When the FE requests statistics, the BE responds with the current values in this cache. 2. FE Side: Retrieval Interval The FE retrieves these statistics from the BEs via a pull mechanism. * Parameter:

tablet_stat_update_interval_second

(FE configuration). * Default Interval: 300 seconds (5 minutes). 3. The "Empty Tablet" Risk with 1-Minute Refreshes If you reduce the FE's

tablet_stat_update_interval_second

to 60 seconds but leave the BE's

tablet_stat_cache_update_interval_second

at its default (300s), you can still land on empty tablets (0 rows) consistently. Why this happens: 1. New Tablets: Every

INSERT OVERWRITE

creates brand-new tablets with new IDs. 2. Zero Start: Initially, these new tablets have 0 rows in the BE's cache. 3. The Race: If the BE cache only updates every 300 seconds, but you replace the tablet every 60 seconds, the BE might never finish a cache update cycle for a specific tablet ID before that tablet is already replaced/deleted. 4. The Result: The FE asks the BE for stats. The BE looks at its cache, hasn't updated it for the new ID yet, and returns 0. The optimizer then sees 0 rows and falls back to cardinality 1. Recommended Configuration for 1-Minute MVs To ensure the FE sees non-zero row counts for a table being overwritten every 60 seconds, you must align both the BE and FE intervals to be significantly faster than the refresh cycle: 1. On all BEs (
be.conf
):

Copy code

properties
    tablet_stat_cache_update_interval_second = 15

(Requires BE restart. This forces the BE to refresh its local row-count cache every 15 seconds). 2. On the Leader FE (
fe.conf
or ADMIN SET):

Copy code

sql
    ADMIN SET FRONTEND CONFIG ("tablet_stat_update_interval_second" = "15");

(This tells the FE to pull the latest cached stats from the BEs every 15 seconds). 3. Follower FE Sync: Followers synchronize these stats from the Leader. By lowering these intervals, the Leader will have the 12.2M row count within ~15-30 seconds of the refresh completing, and the Follower will pull it from the Leader shortly after. Summary If you only reduce the FE interval to 60s, you are still at high risk of "landing on empty" because the BE is likely still serving 0-row stale data from its own 5-minute cache. You must reduce both the BE cache interval and the FE pull interval to ~15-20 seconds to guarantee the Optimizer sees the data within the 1-minute refresh window. References • administration/management/FE_configuration.md • administration/management/BE_configuration.md

Open in Slack

Previous Next