Hey @Rocky We have a big problem in production env. I've got this error in one be pod and I don't know how to fix this.

What's the root cause of this error and how can we fix this?

@          0x96ffebd starrocks::TabletUpdates::compaction_for_size_tiered(starrocks::MemTracker*)
    @          0x97033be starrocks::TabletUpdates::compaction(starrocks::MemTracker*)
    @          0x967c975 starrocks::StorageEngine::_perform_update_compaction(starrocks::DataDir*)
    @          0xa83c2ab starrocks::StorageEngine::_update_compaction_thread_callback(void*, starrocks::DataDir*)
    @         0x14111a04 execute_native_thread_routine
    @     0x7fc1ff4d1ac3 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x94ac2)
    @     0x7fc1ff5638d0 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x1268cf)
/opt/starrocks/be

_entrypoint.sh: line 165:    28 Floating point exception(core dumped) $STARROCKS_HOME/bin/start_be.sh $addition_args

I20260614 15:21:29.957203 140435706082880 persistent_index_tablet_loader.cpp:71] load large primary index start tablet:32031187 version:194 #rowset:7 #segment:11 #row:29817727 -0=29817727 bytes:4451732357

The error

Floating point exception (core dumped)

in the

starrocks::TabletUpdates::compaction_for_size_tiered

function indicates a division by zero occurring in the logic that selects rowsets for Primary Key (PK) table compaction. This is typically a bug in how StarRocks calculates size-tiered levels when encountering specific tablet states (e.g., empty rowsets or unusual data distributions). Root Cause Analysis 1. Strategy: The crash is happening in the Size-tiered Compaction strategy specifically for Primary Key tables. 2. Trigger: The log indicates the BE was attempting to load the primary index for tablet 32031187 just before the crash. The division by zero likely occurs when the compaction scheduler tries to calculate the ratio or level for the 7 rowsets mentioned (

#rowset:7

). 3. Configuration: This issue can be triggered if internal parameters (like the number of levels or size ratios) encounter a zero value or if a rowset has zero rows/segments in a way the code doesn't handle. Immediate Fix & Workaround To stop the BE from crash-looping in production, you should disable the size-tiered compaction strategy for PK tables. This will force StarRocks to use the older, more stable compaction logic for these tables. 1. Modify

be.conf

: Add or update the following parameter in the

be.conf

file of all BE nodes:

properties
    enable_pk_size_tiered_compaction_strategy = false

2. Restart the BE: Restart the crashed BE pod. It should now skip the problematic logic and start successfully. Long-term Resolution 1. Identify the problematic table: Run the following SQL in your StarRocks cluster to find which table and partition the failing tablet belongs to:

sql
    SHOW TABLET 32031187;

2. Check for unusual data: Check the rowsets of that tablet. If you see many rowsets with 0 rows or very small sizes, it might be the trigger. 3. Upgrade StarRocks: This is likely a known bug in specific versions of 3.1.x or 3.2.x. Check the release notes for your current version and consider upgrading to the latest patch release (e.g., if you are on 3.2.5, upgrade to the latest 3.2.x or 3.3.x). 4. Check Configs: Ensure you haven't manually set

size_tiered_level_num

or other related parameters to

0

in your configuration. If the problem persists after disabling the strategy or if you cannot upgrade, please provide your StarRocks version so we can check for specific known issues. References • administration/management/BE_configuration.md

@Rocky What will happen if we disable Compaction flag as u said? What are the pros and cons and trade offs

Disabling the

enable_pk_size_tiered_compaction_strategy

flag switches the Primary Key table from the newer Size-tiered Compaction strategy back to the Legacy (sequential/cumulative) strategy. Here is the breakdown of the impact and tradeoffs: 1. What happens immediately? * Stability: The BE process will stop crashing. The "Floating point exception" is specific to the math used in the size-tiered selection logic. By disabling it, you bypass the buggy code path entirely. * Compaction Style: Instead of grouping rowsets of similar sizes to merge (e.g., merging four 10MB files into one 40MB file), the system will revert to a more traditional strategy that typically merges files based on version order and simple thresholds. 2. The Trade-offs Feature | Size-tiered (Current/Crashing) | Legacy (Workaround) Write Amplification | Lower. It avoids merging small new data into large existing files repeatedly. | Higher. It may frequently merge small incoming rowsets into large existing ones, consuming more I/O. Write Throughput | Higher. Better at handling high-frequency, small-batch writes (common in CDC). | Lower. Can become a bottleneck if the ingestion rate is very high, potentially leading to "Too many versions" errors. Read Performance | Slightly Lower. May result in more "layers" of data, requiring the engine to check more files per query. | Higher. Aggressively reduces the number of data versions, leading to a "flatter" data structure. Disk Usage | Higher. Temporary space usage (space amplification) is higher because multiple tiers of files exist simultaneously. | Lower. Data is consolidated more quickly, reclaiming space from deleted/updated rows faster. 3. Pros and Cons of Disabling It Pros: * Production Stability: It fixes the immediate "Floating Point Exception" crash without requiring an emergency upgrade. * Predictable Reads: If your workload is read-heavy and ingestion is moderate, the legacy strategy can actually improve query latency by keeping the number of rowsets very low. Cons: * I/O Pressure: You will likely see an increase in Disk I/O and CPU usage from the

update_compaction

threads as the system works harder to merge files. * Ingestion Bottlenecks: If you are streaming data via Flink CDC or frequent Routine Loads, you might encounter

Failed to load ... too many versions

errors if the legacy compaction cannot keep up with the write speed. Recommendation 1. Disable it now to restore service. 2. Monitor "Compaction Score" via

SHOW PROC '/compactions';

or your Grafana dashboard. If the score for your Primary Key tables stays below 100, the legacy strategy is handling your load fine. 3. Plan an upgrade to a patched version (e.g., v3.2.10+ or v3.3.3+). Once upgraded, you can re-enable the size-tiered strategy to regain its I/O efficiency. References • administration/management/BE_configuration.md • best_practices/primarykey_table.md

@Rocky Our currently version is 3.5.16

Since you are on version 3.5.16, encountering a

Floating point exception

in

compaction_for_size_tiered

suggests that a division-by-zero bug persists (or has regressed) in the size-tiered compaction logic even in this relatively recent patch. Why this happens in 3.5.16 The error occurs when the BE is calculating how to group rowsets into "tiers" for compaction. It likely encountered a scenario where the divisor became zero. This can be triggered by: 1. Configuration Issues: If parameters like

size_tiered_level_multiple

or

pk_index_size_tiered_level_multiplier

were manually set to

0

or

1

in

be.conf

. 2. Metadata Anomalies: A rowset with

0

bytes or

0

rows being processed in a way the math doesn't handle, or a tablet with an unusual distribution of rowset sizes. 3. A Code Bug: Specifically in the logic that handles Primary Key (PK) tables' update compaction. Recommended Actions 1. Immediate Workaround (Stop the Crash) The most reliable way to stabilize your production environment immediately is to disable the size-tiered strategy for Primary Key tables. This forces the BE to use the legacy compaction logic, which does not use the problematic code path. * Edit

be.conf

on all BE nodes:

conf
    enable_pk_size_tiered_compaction_strategy = false

* Restart the BE. * Impact: Your BEs will stop crashing. Compaction will still happen, but it will use a different (sequential) algorithm. You may see a slight increase in Disk I/O, but the system will remain functional. 2. Check for Misconfigurations Verify your

be.conf

to ensure no one has accidentally set the following to invalid values: * `size_tiered_level_multiple`: Ensure it is at least

2

(default is

5

). * `size_tiered_min_level_size`: Ensure it is not

0

. * `size_tiered_level_num`: Ensure it is not

0

. 3. Inspect the Tablet The log shows the crash happened while dealing with Tablet 32031187. You can check if this tablet has corrupted metadata or an excessive number of empty versions:

sql
-- Check which table this belongs to
SHOW TABLET 32031187;

-- Check the compaction status of that tablet
-- (Replace <be_ip> and <be_http_port> with actual values)
curl http://<be_ip>:<be_http_port>/api/compaction/show?tablet_id=32031187

4. Report the Issue Since 3.5.16 is a very recent version, this should be reported to the StarRocks community or via a GitHub Issue. Provide the full stack trace and the version number. This helps the maintainers identify if a new edge case in the 3.5.x branch is causing the division-by-zero. Summary: Disabling

enable_pk_size_tiered_compaction_strategy

is your "emergency brake" to stop the production crashes. Once the environment is stable, you can investigate the specific tablet or plan a future patch for a future patch. patch update. References • administration/management/BE_configuration.md • administration/management/compaction.md • administration/management/FE_configuration.md • administration/Meta_recovery.md