@Rocky why it crashes? version 4.1.1

2026-06-08 18:04:31	
*** Aborted at 1780967071 (unix time) try "date -d @1780967071" if you are using GNU date ***
2026-06-08 18:04:31	
PC: @         0x15f39074 std::_Rb_tree_insert_and_rebalance(bool, std::_Rb_tree_node_base*, std::_Rb_tree_node_base*, std::_Rb_tree_node_base&)
2026-06-08 18:04:31	
*** SIGSEGV (@0x10) received by PID 33 (TID 0xfffed098e7c0) LWP(744) from PID 16; stack trace: ***
2026-06-08 18:04:31	
    @     0xffffb336abdc (/usr/lib/aarch64-linux-gnu/libc.so.6+0x8abdb)
2026-06-08 18:04:31	
    @         0x135ee424 google::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*)
2026-06-08 18:04:31	
    @     0xffffb4441060 PosixSignals::chained_handler(int, siginfo_t*, void*)
2026-06-08 18:04:31	
    @     0xffffb4441348 JVM_handle_linux_signal
2026-06-08 18:04:31	
    @     0xffffb4b25838 ([vdso]+0x837)
2026-06-08 18:04:31	
    @         0x15f39074 std::_Rb_tree_insert_and_rebalance(bool, std::_Rb_tree_node_base*, std::_Rb_tree_node_base*, std::_Rb_tree_node_base&)
2026-06-08 18:04:31	
    @         0x100e17ac std::_Rb_tree_iterator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::pair<starrocks::RuntimeProfile::Counter*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > std���
2026-06-08 18:04:31	
    @         0x100eecd8 std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<starrocks::RuntimeProfile::Counter*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::less<std::__cxx11::basic_string���
2026-06-08 18:04:31	
    @         0x100e65f0 starrocks::RuntimeProfile::add_counter_unlock(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, starrocks::TUnit::type, starrocks::TCounterStrategy const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::���
2026-06-08 18:04:31	
    @         0x100e68fc starrocks::RuntimeProfile::add_child_counter(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, starrocks::TUnit::type, starrocks::TCounterStrategy const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::a���
2026-06-08 18:04:31	
    @          0xe3b9f64 starrocks::connector::DataSource::update_profile(starrocks::connector::DataSource::Profile const&)
2026-06-08 18:04:31	
    @          0xe3b29dc starrocks::pipeline::ConnectorChunkSource::close(starrocks::RuntimeState*)
2026-06-08 18:04:31	
    @          0xe34dad8 starrocks::pipeline::ScanOperator::_close_chunk_source_unlocked(starrocks::RuntimeState*, int)
2026-06-08 18:04:31	
    @          0xe34ca20 starrocks::pipeline::ScanOperator::_finish_chunk_source_task(starrocks::RuntimeState*, int, long, long, long)
2026-06-08 18:04:31	
    @          0xe352c90 auto starrocks::pipeline::ScanOperator::_trigger_next_scan(starrocks::RuntimeState*, int)::{lambda(auto:1&)#1}::operator()<starrocks::workgroup::YieldContext>(starrocks::workgroup::YieldContext&) const [clone .isra.0]
2026-06-08 18:04:31	
    @          0xe2a544c starrocks::workgroup::ScanExecutor::worker_thread()
2026-06-08 18:04:31	
    @         0x1014e384 starrocks::ThreadPool::dispatch_thread()
2026-06-08 18:04:31	
    @         0x101450cc starrocks::Thread::supervise_thread(void*)
2026-06-08 18:04:31	
    @     0xffffb336595c (/usr/lib/aarch64-linux-gnu/libc.so.6+0x8595b)
2026-06-08 18:04:31	
    @     0xffffb33cbb4c (/usr/lib/aarch64-linux-gnu/libc.so.6+0xebb4b)

Based on the stack trace provided, your StarRocks Backend (BE) is crashing due to a segmentation fault (SIGSEGV) during a map insertion within the

RuntimeProfile

class. This specifically occurs when an external connector (like Hive, Iceberg, or JDBC) is closing its data source and updating performance metrics. Analysis of the Crash * Location: The crash is in

std::_Rb_tree_insert_and_rebalance

, which is an internal C++ standard library function for inserting into a

std::map

. * Trigger: The function

starrocks::RuntimeProfile::add_counter_unlock

is being called. In StarRocks, the

_unlock

suffix typically indicates a function that expects the caller to already hold a lock. * Root Cause: This is likely a concurrency race condition. Two threads are attempting to modify the same

RuntimeProfile

object simultaneously without proper synchronization: 1. One thread is executing

ConnectorChunkSource::close()

(as seen in your trace), which calls

DataSource::update_profile()

. 2. Another thread might be reporting the profile or another parallel scan task might be updating the same parent profile. * Architecture: This crash is occurring on aarch64 (ARM), which can sometimes expose race conditions more frequently due to different memory consistency models compared to x86. Recommended Steps 1. Check for Known Issues: Version 4.1.1 appears to be a very recent release. Similar issues with

RuntimeProfile

concurrency have been addressed in recent patches (e.g., Issue #72904). Since you are on 4.1.1, you should check if there is a newer hotfix (e.g., 4.1.2) that specifically addresses

ConnectorChunkSource

profile races. 2. Temporary Workaround: * Disable Profiles: You can try to disable query profiling to see if it mitigates the crash frequency. Run:

sql
        SET GLOBAL enable_profile = false;
        SET GLOBAL enable_runtime_profile_report = false;

3. Identify the Query: The

be.out

or

<http://be.INFO|be.INFO>

log preceding this crash usually contains a

query_id

. Identifying which specific query (likely involving an External Catalog/Connector) triggers this can help you isolate the problematic table or connector type. 4. Community Support: Because this involves a potential race condition in the core pipeline engine's interaction with external connectors, it is highly recommended to: * Open a GitHub issue with the full stack trace. * Provide details on which connector you are using (e.g., Iceberg, Hive, JDBC). * Upload the

<http://be.INFO|be.INFO>

log leading up to the crash. If the crash persists even with profiles disabled, it suggests the race is inherent to the cleanup logic of the

ConnectorChunkSource

and will require a code fix from the StarRocks maintainers. References • data_source/datalake_faq.md • faq/Others.md • best_practices/query_tuning/query_profile_operator_metrics.md • knowledge/usage/best-practice.md • loading/loading_introduction/troubleshooting_loading.md • administration/management/proc_profile.md

@Rocky it fails for

SET GLOBAL enable_runtime_profile_report = false;

saying it doesn't exists

I apologize for the confusion. The variable name

enable_runtime_profile_report

does not exist in StarRocks. In StarRocks 3.x and 4.x, the "Runtime Profile" feature (which reports progress for long-running queries) is automatically enabled whenever query profiling is turned on. To disable the profiling that is causing the

update_profile

crash, you should use the following variables: 1. Disable Query Profiling Globally Ensure the main profiling switch is off. By default it is

false

, but you should verify:

sql
SET GLOBAL enable_profile = false;

2. Disable "Slow Query" Profiling StarRocks can automatically generate profiles for queries that take longer than a certain threshold, even if

enable_profile

is false. Set this to

0s

(the default) to ensure it doesn't trigger:

sql
SET GLOBAL big_query_profile_threshold = '0s';

3. Check FE Configuration There is also a configuration on the Front End (FE) that controls profile collection. You can check this by running:

sql
ADMIN SHOW FRONTEND CONFIG LIKE 'enable_collect_query_detail_info';

If it is

true

, you can set it to

false

(requires

ADMIN

privileges):

sql
ADMIN SET FRONTEND CONFIG ('enable_collect_query_detail_info' = 'false');

Why this is happening (Version 4.1.1) The stack trace confirms a known race condition in the

RuntimeProfile

class (similar to Issue #72904). In version 4.1.1, when an external connector (like Hive, Iceberg, or JDBC) finishes a task, it updates the profile at the same time the system might be trying to report or serialize it. This concurrent access to the internal map (

std::map

) causes the crash. Recommendation: * Update: Check if a patch release (e.g., 4.1.2 or later) is available, as this specific concurrency bug in the profile engine has been a focus for recent stability fixes. * Verification: You can verify your settings by running

SHOW VARIABLES LIKE '%profile%';

to ensure all profile-related features are disabled. References • sql-reference/System_variable.md • best_practices/query_tuning/query_profile_overview.md • administration/management/FE_configuration.md

@Rocky query that crash nodes

insert overwrite `mv` SELECT `sub2`.`date`, `a`.`c`, CAST(`a`.`r` AS DOUBLE) AS `r` FROM (SELECT `catalog`.`schema`.`table`.`base`, `catalog`.`schema`.`table`.`ap` AS `r`, date(date_format(from_unixtime(`catalog`.`schema`.`table`.`timestamp`), '%Y-%m-%d')) AS `date`, row_number() OVER (PARTITION BY date_format(from_unixtime(`catalog`.`schema`.`table`.`timestamp`), '%Y-%m-%d') ORDER BY `catalog`.`schema`.`table`.`timestamp` DESC ) AS `rank` FROM `catalog`.`schema`.`table`) `sub2` , LATERAL unnest(map_keys(`sub2`.`r`),map_values(`sub2`.`r`)) a(`c`,`r`) WHERE `sub2`.`rank` = 1

data size is very little

The crash in version 4.1.1 is caused by a race condition in the StarRocks Backend (BE) during the collection of performance metrics (profiling) for external data sources. Why it crashes Your stack trace shows the crash occurs at

std::_Rb_tree_insert_and_rebalance

inside

starrocks::RuntimeProfile::add_counter_unlock

. This happens because: 1. Concurrent Access: The query involves an external catalog and a

LATERAL unnest

. This creates a complex execution pipeline where the

ConnectorChunkSource

(the part reading your external table) is trying to update its performance counters (like "RowsRead" or "BytesRead") while another thread (likely the one reporting the profile to the FE) is simultaneously reading or modifying the same profile object. 2. Thread Safety: The internal map used to store these counters is not thread-safe for simultaneous insertion and reading. This leads to the SIGSEGV (segmentation fault) you see in the log. 3. Trigger: The

INSERT OVERWRITE

on an external table with

LATERAL unnest

often involves high concurrency and specific cleanup phases that make this race condition more likely to occur. How to fix it 1. Disable all Profiling Variables Since the crash happens during the "update profile" step, turning off profiling completely should stop the crash. You were close, but the variable names in StarRocks are specific:

sql
-- Disable the main profile switch
SET GLOBAL enable_profile = false;

-- Disable automatic profiling for "big" queries 
-- (Set to 0 to disable automatic triggers)
SET GLOBAL big_query_profile_threshold = 0;

-- Ensure the FE doesn't try to collect detail info
-- (Run as an admin user)
ADMIN SET FRONTEND CONFIG ('enable_collect_query_detail_info' = 'false');

2. Update to a Patch Version This is a known issue in the early 4.1.x and 4.0.x releases (related to Issue #72904 and similar concurrency bugs in the pipeline engine). * Check if 4.1.2 or a later hotfix is available. Maintainers have been releasing patches specifically for

RuntimeProfile

stability in the 4.x line. 3. Query Optimization Workaround If you cannot upgrade immediately, you can sometimes avoid this specific race condition by simplifying the "External Scan" part of the query. * The crash is happening during the

ConnectorChunkSource::close

phase. _ If your

map

column

r

is large, the

unnest(map_keys, map_values)

creates a lot of internal overhead. Try to filter as much as possible _inside* the subquery before the

unnest

to reduce the number of chunks being processed by the external scanner. Summary: The crash is a bug in the profiling code of the external connector. Disabling

enable_profile

globally is the most effective immediate workaround until you can apply a 4.1.x patch release patch. References • knowledge/trouble_shooting/fe_deadlock_problems.md • knowledge/usage/best-practice.md