Hey team and @Rocky How can i fix the following error ? BE pods crash randomly. And i got this: W20260221 121036.633342 140430751995456 kv_store.cpp:378] Corruption: In DBIter: : Corrupted Key: Internal Key too small. Size=0. F20260221 121036.633503 140430751995456 tablet_meta.cpp:231] fail to save tablet metaCorruption Corruption: In DBIter: : Corrupted Key: Internal Key too small. Size=0. be/src/storage/tablet_meta_manager.cpp:425 store->get_meta()->OptDeleteRange(META_COLUMN_FAMILY_INDEX, lower, upper, &batch). tablet_id=16388, schema_hash=642944190 * Aborted at 1771663236 (unix time) try "date -d @1771663236" if you are using GNU date * PC: @ 0x7fba68f739fc pthread_kill * SIGABRT (@0x27076e) received by PID 2557806 (TID 0x7fb895169640) LWP(2558771) from PID 2557806; stack trace: *

Hi there! Rocky here. I can help you with that error. The error message

Corruption: In DBIter: : Corrupted Key: Internal Key too small

indicates that the RocksDB metadata store on one of your BE's data disks has become physically corrupted. This metadata (stored in the

meta/

subdirectory of your storage paths) tracks tablet information like versions and rowsets. When the BE tries to read or update this metadata and encounters corruption, it triggers a crash to prevent further data inconsistency. In your specific case, the crash is triggered by operations on tablet_id 16388. How to Fix Since StarRocks typically uses a replication factor (e.g., 3), the safest and most common way to fix this is to let the cluster's self-healing mechanism recover the data. Option 1: Clear the corrupted data directory (Recommended) If your data is replicated, you can simply wipe the storage path on the affected BE. 1. Identify the affected path: Check your

be.conf

for the

storage_root_path

. If you have multiple disks, identify which one contains the metadata for tablet

16388

(or just check which one the BE logs are pointing to). 2. Stop the BE pod: Ensure the BE is not running. 3. Clear the data: Delete the contents of the

data/

and

meta/

folders for the affected storage path. • Note: In a Kubernetes environment, you might need to delete the specific PersistentVolumeClaim (PVC) associated with that BE pod if you want a fresh start, or manually exec into the pod/host to clear the mount. 4. Restart the BE: Once started with an empty directory, the BE will register itself with the FE. 5. Auto-Recovery: The FE will notice that replicas are missing for the tablets that were on that disk and will automatically trigger "Clone" tasks to restore those replicas from other healthy BEs. Option 2: Delete specific tablet metadata (Advanced) If you want to try and fix only the specific tablet without wiping the whole disk, you can use the

meta_tool

utility while the BE is stopped:

bash
# Navigate to your BE bin directory
./bin/meta_tool --operation=delete_meta --tablet_id=16388

Note: If the underlying RocksDB structure is severely corrupted, this tool might also fail with the same error. In that case, Option 1 is necessary. Preventive Measures * Check Disk Health: RocksDB corruption is often a symptom of underlying hardware issues. Run

dmesg

or check your cloud provider's disk health metrics for IO errors. * File System: Ensure you are using a stable file system (like XFS or Ext4) and that the disk is not hitting 100% utilization. * Clean Shutdowns: Ensure your Kubernetes termination grace period is long enough for the BE to shut down gracefully. Does your table have at least 3 replicas? If it's a single-replica table, please let me know before deleting anything, as that would lead to data loss! References • administration/management/BE_configuration.md • administration/Meta_recovery.md

Hey @Rocky. I have some PK tables with Qurrom all. I encountered an error: publish on tablet 4022191 failed during data insertion. I attempted to mark the lower-version tablet as bad so it would clone from the healthy tablet. However, the cloning process doesn't seem to complete successfully, as I observed it still running in the cluster_balance's 'running' section since yesterday. What should I do? This is causing some transactions to wait.

Hello! This is Rocky. Dealing with a stuck tablet when you have

write_quorum

set to

ALL

can be frustrating because it effectively blocks your data ingestion until every single replica is healthy. Here is a step-by-step guide to resolve this and unblock your transactions: 1. Immediate Workaround: Change Write Quorum Since you are using

write_quorum = 'ALL'

, any failure or delay on a single replica (like tablet 4022191) will cause the entire load transaction to wait or fail. To unblock your cluster immediately, you can change the write quorum to

MAJORITY

. This allows the transaction to succeed as long as the majority of replicas are healthy. Run this command:

sql
ALTER TABLE <your_table_name> SET ("write_quorum" = "MAJORITY");

Note: This property is supported from v2.5 onwards. 2. Diagnose Why Cloning is Stuck If the tablet has been in the

RUNNING

state in

cluster_balance

since yesterday, it's likely encountering an error during the actual data transfer or index rebuilding (which is heavy for Primary Key tables). Check the specific error message by running:

sql
SHOW PROC '/cluster_balance/running_tablets';

Look for the

ErrMsg

column for tablet

4022191

. Common issues include: * No source replica: The FE cannot find a healthy replica with a version high enough to clone from. * Disk full: The destination Backend (BE) might not have enough space. * Timeout: The clone is taking too long (common if the tablet is very large or the Primary Key index is massive). 3. Verify Tablet Replica Status To see exactly what's happening with the versions of your replicas, run:

sql
SHOW TABLET 4022191;

Copy the

DetailCmd

from the result and run it. It will look like:

sql
SHOW PROC '/dbs/<db_id>/<table_id>/partitions/<part_id>/<index_id>/4022191';

Check the following in the output: * Version: Compare the versions of all replicas. If the "bad" one is very far behind, cloning will take significant time. * IsBad: Confirm if the replica you marked as bad is indeed showing

true

. * State: Check if any replica is in

CLONE

,

DECOMMISSION

, or

NORMAL

. 4. Check for Primary Key Index Issues Primary Key tables rely on an index (in-memory or persistent). If the tablet is large, the "cloning" process includes rebuilding this index on the new BE. * Check the BE logs (

<http://be.INFO|be.INFO>

) on the destination node for any errors related to

tablet 4022191

or

Primary Key Index

. * If you see "Primary-key index exceeds the limit," you might need to enable

persistent_index

for that table if it isn't already. 5. Force a Reschedule (If necessary) If the task is stuck and the

ErrMsg

isn't helping, you can try to restart the FE (Follower/Observer) that is currently the Leader to refresh the

TabletScheduler

, though this is a last resort. Usually, once you change the

write_quorum

to

MAJORITY

, the pressure on the transactions will drop, allowing the system more "breathing room" to finish the background repair. Summary Recommendation: Set

write_quorum

to

MAJORITY

first. This is the safest way to get your production traffic moving while the background repair (cloning) finishes. Let me know if you see a specific error message in the

running_tablets

output! References • faq/loading/Loading_faq.md • table_design/table_types/primary_key_table.md • sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md • best_practices/primarykey_table.md • deployment/shared_data/feature-support-shared-data.md

Hey @Rocky below are the logs: r_version(EditVersion(version, 0), 55000, ul) be/src/storage/snapshot_manager.cpp:683 tablet->capture_consistent_rowsets(Version(0, snapshot_version), &snapshot_rowsets) be/src/storage/tablet_manager.cpp:1846 SnapshotManager::instance()->make_snapshot_on_tablet_meta(tablet) W20260225 092959.538187 140043068888640 agent_task.cpp:419] clone failed. signature: 4022191 W20260225 092959.538189 140039075907136 agent_task.cpp:419] clone failed. signature: 4028256

Based on the logs you provided, it appears that a clone task (likely for replica repair or balancing) is failing on a StarRocks Backend (BE). Here is a breakdown of what the logs indicate and how to troubleshoot the issue: What the logs mean 1.

capture_consistent_rowsets(Version(0, snapshot_version), ...)

: This function is called when the BE tries to create a snapshot of a tablet (usually to send it to another BE during a clone). It attempts to find a continuous chain of data files (rowsets) from version 0 to the current version (in your case, version 55000). 2.

clone failed

: The task failed because the BE could not create a consistent snapshot. This typically happens if the source tablet has a version gap or missing rowsets. For example, it might have version 0-100 and 150-55000, but is missing 101-149. Recommended Troubleshooting Steps 1. Identify the failing Tablet Look further up in your BE logs (search for the timestamp

09:29:59

) to find the tablet ID associated with the failed clone task or signature (

4022191

or

4028256

). 2. Check Replica Status Once you have the tablet ID, run the following command in your MySQL client to check the health of its replicas:

sql
ADMIN SHOW REPLICA STATUS FROM <table_name> WHERE TabletId = <tablet_id>;

* Check

IsBad

: If any replica is marked as

true

, it is corrupted. * Check

Version

: Look for discrepancies between replicas. If one replica has a much lower version or a "LastFailedVersion", it might be the cause. 3. Inspect Tablet Details Use the

SHOW TABLET

command to see the version details:

sql
SHOW TABLET <tablet_id>;

This will provide a

DetailCmd

. Execute that command to see the specific rowsets and check if there are gaps in the version range (e.g., version 0-10, then suddenly version 15-55000). 4. How to Fix If you find a replica that is inconsistent or "bad," you can force the FE to schedule a new clone by marking the problematic replica as "BAD":

sql
-- Replace <tablet_id> and <backend_id> with the actual values
ADMIN SET REPLICA STATUS TABLET <tablet_id> BACKEND <backend_id> STATUS = 'BAD';

The Frontend (FE) will then automatically delete the bad replica and clone a fresh, healthy copy from another BE. 5. Check Disk Space and IO Sometimes snapshots fail if: * The BE disk is nearly full. * The BE is under extremely high IO load, causing timeouts. * There are hardware-level disk errors (check

dmesg

or system logs on the BE host). Note: If this is happening across many tablets, check if a specific Backend node is having connectivity issues or if there was a recent hard crash that might have corrupted the metadata. References • data_source/catalog/iceberg/iceberg_meta_table.md • data_source/catalog/iceberg/iceberg_timetravel.md • administration/cluster_snapshot.md

Hey @Rocky the full log is: W20260225 092959.538153 140043068888640 tablet_manager.cpp:1222] Fail to remove or move /opt/starrocks/be/storage/data/983/4022191 Timeout wait_for_version timeout(56003ms) version:992 tablet:4022191 #version:4 [989 989@0 992] pending: rowsets7[id/seg/row/del/byte/compaction] [0/1/836617/0/171.70 MB/84.30 MB],[1/1/370344/0/75.19 MB/180.81 MB],[2/1/2638/0/1.78 MB/254.22 MB],[3/1/545/0/319.61 KB/255.69 MB],[4/1/11/0/29.25 KB/255.97 MB],[5/1/11/0/29.08 KB/255.97 MB],[6/1/7/0/26.74 KB/255.97 MB] be/src/storage/tablet_updates.cpp:3909 _wait_for_version(EditVersion(version, 0), 55000, ul) be/src/storage/snapshot_manager.cpp:683 tablet->capture_consistent_rowsets(Version(0, snapshot_version), &snapshot_rowsets) be/src/storage/tablet_manager.cpp:1846 SnapshotManager::instance()->make_snapshot_on_tablet_meta(tablet) W20260225 092959.538172 140043068888640 tablet_manager.cpp:1263] before adding new cloned tablet, delete stale TABLET_SHUTDOWN tablet failed after 0 times retry, tablet:4022191 stTimeout wait_for_version timeout(56003ms) version:992 tablet:4022191 #version:4 [989 989@0 992] pending: rowsets7[id/seg/row/del/byte/compaction] [0/1/836617/0/171.70 MB/84.30 MB],[1/1/370344/0/75.19 MB/180.81 MB],[2/1/2638/0/1.78 MB/254.22 MB],[3/1/545/0/319.61 KB/255.69 MB],[4/1/11/0/29.25 KB/255.97 MB],[5/1/11/0/29.08 KB/255.97 MB],[6/1/7/0/26.74 KB/255.97 MB] be/src/storage/tablet_updates.cpp:3909 _wait_for_version(EditVersion(version, 0), 55000, ul) be/src/storage/snapshot_manager.cpp:683 tablet->capture_consistent_rowsets(Version(0, snapshot_version), &snapshot_rowsets) be/src/storage/tablet_manager.cpp:1846 SnapshotManager::instance()->make_snapshot_on_tablet_meta(tablet) W20260225 092959.538174 140039075907136 tablet_manager.cpp:1222] Fail to remove or move /opt/starrocks/be/storage/data/31/4028256 Timeout wait_for_version timeout(56003ms) version:1255 tablet:4028256 #version:12 [1244 1244@0 1255] pending: rowsets32[id/seg/row/del/byte/compaction] [0/1/1865/0/444.08 KB/255.57 MB],[1/1/2457880/87989/405.02 MB/-76.53 MB],[2/1/2454330/0/417.09 MB/-161.09 MB],[3/1/2630122/59886/407.70 MB/-105.29 MB],[4/1/1488622/0/269.28 MB/-13.28 MB],[5/1/1577839/0/268.63 MB/-12.63 MB],[6/1/1905432/0/332.08 MB/-76.08 MB],[7/1/1977307/0/341.43 MB/-85.43 MB],[8/1/1493780/0/267.30 MB/-11.30 MB],[9/1/2043298/0/336.57 MB/-80.57 MB],[10/1/1669259/0/271.15 MB/-15.15 MB]...,[23/1/21526/0/4.82 MB/251.18 MB],[24/1/19667/0/4.53 MB/251.47 MB],[25/1/19667/0/4.52 MB/251.48 MB],[26/1/20951/0/4.63 MB/251.37 MB],[27/1/20951/0/4.64 MB/251.36 MB],[28/1/17233/0/4.01 MB/251.99 MB],[29/1/17233/0/4.04 MB/251.96 MB],[30/1/4661/0/1.42 MB/254.58 MB],[31/1/4661/0/1.38 MB/254.62 MB] be/src/storage/tablet_updates.cpp:3909 _wait_for_version(EditVersion(version, 0), 55000, ul) be/src/storage/snapshot_manager.cpp:683 tablet->capture_consistent_rowsets(Version(0, snapshot_version), &snapshot_rowsets) be/src/storage/tablet_manager.cpp:1846 SnapshotManager::instance()->make_snapshot_on_tablet_meta(tablet) W20260225 092959.538179 140039075907136 tablet_manager.cpp:1263] before adding new cloned tablet, delete stale TABLET_SHUTDOWN tablet failed after 0 times retry, tablet:4028256 stTimeout wait_for_version timeout(56003ms) version:1255 tablet:4028256 #version:12 [1244 1244@0 1255] pending: rowsets32[id/seg/row/del/byte/compaction] [0/1/1865/0/444.08 KB/255.57 MB],[1/1/2457880/87989/405.02 MB/-76.53 MB],[2/1/2454330/0/417.09 MB/-161.09 MB],[3/1/2630122/59886/407.70 MB/-105.29 MB],[4/1/1488622/0/269.28 MB/-13.28 MB],[5/1/1577839/0/268.63 MB/-12.63 MB],[6/1/1905432/0/332.08 MB/-76.08 MB],[7/1/1977307/0/341.43 MB/-85.43 MB],[8/1/1493780/0/267.30 MB/-11.30 MB],[9/1/2043298/0/336.57 MB/-80.57 MB],[10/1/1669259/0/271.15 MB/-15.15 MB]...,[23/1/21526/0/4.82 MB/251.18 MB],[24/1/19667/0/4.53 MB/251.47 MB],[25/1/19667/0/4.52 MB/251.48 MB],[26/1/20951/0/4.63 MB/251.37 MB],[27/1/20951/0/4.64 MB/251.36 MB],[28/1/17233/0/4.01 MB/251.99 MB],[29/1/17233/0/4.04 MB/251.96 MB],[30/1/4661/0/1.42 MB/254.58 MB],[31/1/4661/0/1.38 MB/254.62 MB] be/src/storage/tablet_updates.cpp:3909 _wait_fo

Based on the logs you provided, the error

Timeout: wait_for_version timeout(56003ms)

indicates that a Primary Key tablet operation is timing out while waiting for committed data (rowsets) to be "applied" to the primary index. This typically happens during tablet migration, cloning, or snapshotting when the Backend (BE) needs to ensure a tablet is in a consistent state before moving or deleting it. What the Log Tells Us 1. Wait Timeout: The system waited about 55–56 seconds (

55000ms

) for the tablet to catch up to a specific version (e.g., version 992), but it failed. 2. Pending Rowsets: * Tablet

4022191

had 7 pending rowsets. * Tablet

4028256

had 32 pending rowsets. 3. Operation: The BE was trying to remove a stale tablet (

TABLET_SHUTDOWN

) or move it to trash after a clone task, but it couldn't finish the metadata snapshot because the "Apply" process was too slow. ────────── Root Causes & Solutions This issue is almost always caused by the Primary Key Apply process being slower than the ingestion/migration rate. 1. High Disk I/O Pressure The Primary Key engine uses a Persistent Index stored on disk. If your disks (especially if they are not SSDs) are under high I/O wait, the process of reading and updating the index for each pending rowset will slow down significantly. * Action: Monitor disk I/O usage (e.g.,

iostat -x 1

or StarRocks metrics). Ensure your storage can handle the write load. 2. Increase Apply Concurrency By default, the number of threads for applying Primary Key updates is limited. You can increase this to speed up the processing of pending rowsets. * Configuration: Add or update this in your `be.conf`:

conf
    # Default is usually equal to the number of CPU cores.
    # Try increasing it if you have spare CPU resources.
    transaction_apply_worker_count = 16

Note: Restart the BE for this to take effect. 3. Check Memory Limits for PK Index If the memory allocated for the Primary Key index is too low, the system will frequently swap index data to disk (I/O), slowing down the "Apply" phase. * Configuration: Check

update_memory_limit_percent

in

be.conf

(default is 60%). If your BE is under heavy memory pressure, this might be throttled. 4. Ingestion Frequency (Small Files) If you are performing many very small, frequent loads (e.g., every few seconds via Stream Load), you generate many small rowsets/versions. This puts heavy pressure on the Primary Key engine to apply them all. * Action: Batch your data into larger loads or increase the

batch_size

/

consume_interval

for Routine Load or Spark Connector. 5. Persistent Index Cache If you have a very large number of tablets, the Persistent Index cache might be thrashing. * Check: Monitor

mem_tracker

for the

update

type to see how much memory the Primary Key indexes are consuming.

bash
    curl http://<BE_IP>:<BE_HTTP_PORT>/mem_tracker?type=update

Summary of Recommended Fix 1. Increase

transaction_apply_worker_count

in

be.conf

. 2. Ensure your data is stored on SSDs for Primary Key tables. 3. Reduce the frequency of small loads to decrease the number of pending rowsets. If the error persists and prevents tablet repairs (clones), you may need to temporarily throttle ingestion to allow the BE to catch up with its background tablet management tasks. References • administration/management/FE_configuration.md • administration/management/BE_configuration.md

Hey @Rocky What the cause of this erro? How should i fix it? Status Failed starrocks-sharednothing-be-2.starrocks-sharednothing-be-search.starrocks-sharednothing-arch.svc.cluster.local Fail to save committed rowset. tablet_id 20223643, txn_id 10510614 Duration : 012 TS : 2026-02-26 231153.576097+03:30

The error "Fail to save committed rowset" usually occurs during the Publish Version phase of a data load. This is when the Backend (BE) has finished writing the data files (rowsets) and is attempting to update the tablet's metadata to make that data visible. Common Causes 1. Disk Space Issues: The most common cause. If the disk volume associated with the BE (specifically where the tablet metadata is stored) is full or has reached the high-water mark, the BE cannot write the metadata update. 2. I/O Errors or Hardware Issues: If the underlying storage has encountered a hardware failure or the file system has become read-only, the BE will fail to persist the rowset metadata. 3. Metadata Corruption: In rare cases, the local metadata store (RocksDB) on the BE might be corrupted, preventing updates to specific tablet entries. 4. High Load/Concurrency: Intense resource contention (CPU/IO) or high frequency of small loads can sometimes cause timeouts or failures in the metadata persistence layer. ────────── How to Fix It 1. Check BE Disk Space Check the disk usage on the specific BE mentioned in your error (

starrocks-sharednothing-be-2

):

bash
# Log into the BE pod/server and check disk space
df -h

If the disk is full, you will need to clean up old logs, expand the volume, or add more BE nodes to redistribute the data. 2. Inspect BE Logs for Detailed Errors The error you shared is a high-level status. To find the root cause, look into the

<http://be.INFO|be.INFO>

or

be.WARNING

logs on the affected BE around the time of the failure:

bash
grep "20223643" <http://be.INFO|be.INFO> | grep "10510614"

Look for keywords like

io_error

,

No space left on device

, or

meta service error

. 3. Verify Tablet Status Check the state of the specific tablet mentioned (

20223643

) using MySQL:

sql
SHOW TABLET 20223643;

* Identify which table and partition this belongs to. * Check if the tablet is "IsHealthy". If multiple replicas are failing, the issue might be broader than just one node. 4. Check for Version Lag (Cumulative Compaction) If you are loading data very frequently, the BE might be struggling with "too many versions." Run this to check the version count for that tablet:

sql
SELECT * FROM information_schema.be_tablets WHERE tablet_id = 20223643;

If the

version_count

is very high (e.g., > 1000), you should: * Batch your loads into larger chunks. * Increase compaction threads in `be.conf`:

properties
    cumulative_compaction_num_threads_per_disk = 4
    base_compaction_num_threads_per_disk = 2

5. Restart the Affected BE If disk space and I/O look fine, the metadata engine might be in a transient stuck state. Restarting the affected BE (

be-2

) can often clear temporary locking or memory issues. Summary: Start by checking disk space on BE-2 and then search the BE logs for the specific

io_error

that caused the save failure. References • faq/loading/Loading_faq.md • faq/Sql_faq.md • knowledge/admin_and_operation/shared_data_compaction.md

Hey @Rocky What the cause of this erro? How should i fix it? transmit chunk rpc failed dest_instance_id3baa08b7-134b-11f1-9dc6-421a15f92c07 dest10.10.9.388060 detailbrpc failed, errorHost is down, error_textE112Not connected to 10.10.9.388060 yet, server_id816 R1E112Not connected to 10.10.9.388 Duration : 0562 TS : 2026-02-26 231216.161536+03:30

The error message

transmit chunk rpc failed ... brpc failed, error Host is down, error_text E112 Not connected to 10.10.9.38:8060 yet

indicates that a component in your StarRocks cluster (likely an FE or another BE) failed to establish or maintain a network connection with the Backend (BE) node at

10.10.9.38

on port

8060

. In bRPC, the error code E112 (ENOTCONN) specifically means that the connection hasn't been established yet or the target host is unreachable. Common Causes 1. BE Process Crash or Stop: The BE node at

10.10.9.38

might have crashed (due to OOM, a core dump, etc.) or was manually stopped. 2. Network Connectivity Issues: A firewall, security group, or routing issue is blocking traffic on the bRPC port (default

8060

) between the nodes. 3. Incorrect IP Configuration: If the BE has multiple network interfaces and

priority_networks

is not configured in

be.conf

, the BE might have registered with the FE using an incorrect or unreachable IP address. 4. High Resource Pressure: If the BE is under extreme CPU or memory pressure, it may become unresponsive, leading to connection timeouts or failures. ────────── How to Fix It 1. Check BE Status Log in to the node at

10.10.9.38

and check if the BE process is running:

bash
ps aux | grep starrocks_be

If it is not running, check the logs for the cause of the crash: *

be.out

: Look for "Out of memory" (OOM) or stack traces. *

<http://be.INFO|be.INFO>

: Check for recent error messages before the process stopped. 2. Verify Port Listening Ensure the BE is actually listening on port `8060`:

bash
netstat -tlnp | grep 8060

3. Test Network Connectivity From the node that reported the error (the sender), try to connect to the target BE's bRPC port:

bash
telnet 10.10.9.38 8060
# OR
nc -vz 10.10.9.38 8060

If this fails, check your firewall (`iptables`/`firewalld`) or cloud security groups. 4. Verify

priority_networks

Check the

be.conf

file on the target node. If the machine has multiple IPs, ensure

priority_networks

is set to the correct CIDR:

conf
# Example: if your IP is 10.10.9.38
priority_networks = 10.10.9.0/24

After modifying this, you must restart the BE and potentially re-add it to the cluster if the registered IP was wrong. 5. Check for OOM (Out of Memory) If the BE crashed, run

dmesg -T | grep -i oom

to see if the Linux OOM killer terminated the process. If it did, you may need to increase the machine's RAM or adjust

mem_limit

in

be.conf

. References • faq/Deploy_faq.md • knowledge/usage/best-practice.md • administration/Meta_recovery.md • administration/management/BE_configuration.md • administration/management/FE_configuration.md • faq/loading/Loading_faq.md

@Rocky I checked dmesg of be_2 and found this: root@starrocks-sharednothing-be-2:/var/log# dmesg | grep seg [4690038.572261] update_compact[4090603]: segfault at 0 ip 00000000096ce791 sp 00007fd20b9d71c0 error 4 in starrocks_be[80e8000+bf3d000] likely on CPU 34 (core 2, socket 1) [4744767.819528] publish_version[2110555]: segfault at 10 ip 0000000013f57188 sp 00007fcd039e0a08 error 4 in starrocks_be[80e8000+bf3d000] likely on CPU 123 (core 27, socket 1) [4779889.910484] publish_version[3312092]: segfault at 10 ip 0000000013f571ea sp 00007ff13a6daa48 error 4 in starrocks_be[80e8000+bf3d000] likely on CPU 11 (core 11, socket 0) [4862721.864704] publish_version[2614710]: segfault at 10 ip 0000000013f57188 sp 00007f128a3e8a08 error 4 in starrocks_be[80e8000+bf3d000] likely on CPU 58 (core 26, socket 1)

Hello! I'm Rocky. I see you're encountering segmentation faults in your StarRocks BE process, specifically in the

update_compact

and

publish_version

threads. This usually indicates a crash in the BE engine. To help diagnose this, could you please provide a bit more information? 1. StarRocks Version: What version of StarRocks are you running? 2. Table Type: Are these crashes happening on a Primary Key table? The

update_compact

and

publish_version

threads are heavily involved in the metadata management and compaction for Primary Key tables. 3. Check `be.out`: When a BE crashes, it often prints a stack trace to the

be.out

log file (located in the

log/

directory of your BE). Please check that file for any lines starting with

[0x...

or a backtrace. 4. Core Dumps: To pinpoint the exact cause, we usually need a core dump. * Check if core dumps are enabled on your system (

ulimit -c unlimited

). * If you have a core file, you can use

gdb

to get the stack trace:

bash
        gdb <path_to_starrocks_be_binary> <path_to_core_file>
        (gdb) thread apply all bt

Common Causes for these specific threads: * publish_version: Often related to a race condition or a timeout during the final stage of a transaction commit. * update_compact: Usually related to Primary Key table compaction or memory issues when handling large numbers of deletes/updates. If you can find the stack trace in

be.out

, please share it here (stripping any sensitive info) so I can help you further!

@Rocky what is be.out?

In StarRocks,

be.out

is a log file that captures the standard output (stdout) and standard error (stderr) of the Backend (BE) process. Here are the key details about this file: 1. Purpose While the primary operational logs (like

<http://be.INFO|be.INFO>

,

be.WARNING

, and

be.ERROR

) store structured application logs,

be.out

is used to catch anything printed directly to the console. This is critical for: * Startup Troubleshooting: If the BE fails to start before the main logging system is initialized, the error message will likely be in

be.out

. * Crashes and Panics: If the BE process encounters a catastrophic failure (like a segmentation fault or a "panic"), the stack trace or error dump is often written here. * Embedded JVM Logs: Since the BE contains an embedded JVM (used for things like HDFS/Object Storage access), certain JVM-level errors or prints may end up in this file. 2. Location By default, you can find it in the

log

directory of your StarRocks BE installation path:

path_to_be/log/be.out

3. Maintenance Tip Unlike the

.INFO

or

.WARNING

logs,

be.out

is often not automatically rotated or compressed by StarRocks. * Disk Space: If your BE process is frequently outputting errors or if you have certain debug flags enabled, this file can grow quite large over time. * Cleaning: It is generally safe to truncate or delete this file if it becomes too large, though it is recommended to check its contents first to ensure you aren't missing recurring error reports. Summary: If your BE process isn't running and you don't see anything useful in

be.WARNING

,

be.out

is the next place you should look.