@Rocky StarRocks bug (3.3.21) - The BE reads table data with only 2 threads, because the pod has 2 cores. - Both threads are stuck in StarRocks' late-materialization page read (fetch_values_by_rowid -> seek_to_ordinal -> page cache) and never finish. Same place in two stack dumps 2 min apart, and both busy on CPU (top). - With both threads taken, no other query can read any data. Every query waits until the 300 s timeout (MV refreshes: 3600 s), and cancelling a query does not free a stuck thread. can you help me with this ? how can I debug further ?

This behavior (100% CPU in

seek_to_ordinal

during

fetch_values_by_rowid

) is characteristic of Late Materialization (LM). In this mode, StarRocks first filters rows based on a subset of columns (the ones in the

WHERE

clause) and then performs a second pass to "fetch" the remaining columns only for the matching row IDs. When a thread is stuck in

seek_to_ordinal

, it is often because it is iterating through many data pages or compressed blocks to find specific row IDs, which can be computationally expensive if the selectivity is low or the data distribution is fragmented. 1. Immediate Mitigation Since you only have 2 cores, the 2 scan threads are essentially your entire execution capacity. If they get stuck, the BE cannot process any other requests. * Disable Late Materialization: Test if the issue disappears by forcing the engine to read all columns in a single pass (Early Materialization). * Session level:

SET enable_late_materialization = false;

* BE configuration: Set

late_materialization_ratio = 0

in

be.conf

and restart (or use the

curl

command to update it dynamically if the BE is responsive). * Increase Resources: StarRocks is designed for multi-core environments. A 2-core limit is extremely tight for a BE; increasing this to at least 4 or 8 cores would provide more thread-pool "headroom" so that a single heavy query doesn't starve the entire node. 2. Debugging Steps To identify why these threads are spinning: * Analyze the Query Profile: Run the problematic query with

SET is_report_success=true;

and check the profile. Look specifically for: * `LateMaterializeRowidReadTime`: High values here confirm that LM is the bottleneck. *

ScanRows

vs. `ReadRows`: If

ScanRows

is very high but

ReadRows

is low, LM is trying to jump between thousands of non-contiguous rows. * Check Column Types: Are you querying columns with complex types (

ARRAY

,

JSON

,

MAP

)? Late materialization of complex types has had known performance regressions in some 3.3.x patches (fixed in later versions like 3.5, e.g., issue #75861). * Check for Page Cache Thrashing: Since your stack trace mentions Page Cache, check if your

storage_page_cache_limit

is too small, causing constant cache evictions and re-reads during the

seek_to_ordinal

phase. 3. Known Issues in 3.3.x While StarRocks 3.3.21 is a stable release, there have been fixes in subsequent versions (and the 3.5 branch) related to: * Redundant

seek_to_ordinal

calls in array-column read paths. * Infinite loops in specific

partial_update

scenarios. * Cancellations not being checked frequently enough in certain low-level I/O loops. Recommendation: If disabling late materialization solves the CPU spike, you may want to keep it disabled for that specific workload or consider upgrading to the latest 3.3.x patch (or 3.4/3.5) where several late materialization optimizations have been backported. If the issue persists even after disabling LM, please verify if the threads are still in the same stack trace, as that would suggest a lower-level I/O or page cache bug rather than an LM-specific logic issue. References • knowledge/trouble_shooting/fe_deadlock_problems.md