@Rocky, I'm getting repeated crashes on compute nodes due to this segmentation fault, What could be the possible cause? If there is an issue is this fixed in 4.1.3: I20260904 114130.695544 135478558545600 tablet_sink_sender.cpp:360] Olap table sink statistics. load_id: 01a06b09-c069-7c93-8a5a-42391b4b835b, txn_id: 10046506, add chunk time(ms)/wait lock time(ms)/num: {10599:(0)(0)(0)} {10889:(0)(0)(0)} {10425:(0)(0)(0)} {10744:(0)(0)(0)} {10560:(0)(0)(0)} {10705:(0)(0)(0)} {10840:(0)(0)(0)} {10474:(0)(0)(0)} {10206:(0)(0)(0)} {10388:(0)(0)(0)} {10011:(0)(0)(0)} {10928:(0)(0)(0)} {10339:(0)(0)(0)} {10513:(0)(0)(0)} {10252:(0)(0)(0)} {10302:(0)(0)(0)} {10178:(0)(0)(0)} {10008:(0)(0)(0)} {10791:(0)(0)(0)} {10646:(0)(0)(0)} @ 0x7b3916261ed3 (/usr/lib/x86_64-linux-gnu/libc.so.6+0xa1ed2) I20260904 114130.721149 135478692828864 query_context.cpp:64] finished query_id:01a06b09-c069-7c93-8a5a-42391b4b835b context life time:52480114097 cpu costs:60875321414 peak memusage:1722525970 scan_bytes:3505054259 spilled bytes:2824122731 cache_hit_ratio:100.0% @ 0x14c1c846 google::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) I20260904 114130.745623 135478412011200 query_context.cpp:64] finished query_id:01a06b09-c069-7652-9d0f-af46c41d3b38 context life time:52261825777 cpu costs:556717191337 peak memusage:28887112895 scan_bytes:1458104139 spilled bytes:130223449 cache_hit_ratio:100.0% I20260904 114130.756895 135478634079936 query_context.cpp:64] finished query_id:01a06b0a-63ba-7482-baac-7603f5ef4058 context life time:40156910827 cpu costs:521029027229 peak memusage:43057587092 scan_bytes:5162147269 spilled bytes:563459196 cache_hit_ratio:100.0% @ 0x7b39173dd8e0 PosixSignals::chained_handler(int, siginfo_t*, void*) @ 0x7b39173ddbf4 JVM_handle_linux_signal @ 0x7b3916205330 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x4532f) @ 0xe90a4dc starrocks:spillPartitionedSpillerWriter:_compact_skew_chunks(unsigned long, std:vector<stdshared ptr<starrocks:Chunk>, std:allocator<stdshared ptr<starrocks:Chunk> > >&, std:shared ptr<starrocks:AggregatorParams>&) @ 0xe90c540 starrocks:spillPartitionedSpillerWriter:_pick_and_compact_skew_partitions(std:vector<starrocksspill:SpilledPartition*, std:allocator<starrocksspill:SpilledPartition*> >&) @ 0xd880636 starrocks::Status starrocks:spillPartitionedSpillerWriterflush<starrocksspill:IOTaskExecutor, starrocks:spillResourceMemTrackerGuard<stdweak ptr<starrockspipeline:QueryContext>, std:weak ptr<starrocksspill:Spiller> >&>(starrocks::Runtim@ @ 0xd8811c7 starrocks::Status starrocks:spillPartitionedSpillerWriterspill<starrocksspill:IOTaskExecutor, starrocks:spillResourceMemTrackerGuard<stdweak ptr<starrockspipeline:QueryContext>, std:weak ptr<starrocksspill:Spiller> >&>(starrocks::Runtim@ @ 0xd881400 starrocks::Status starrocks:spillSpillerspill<starrocksspill:IOTaskExecutor, starrocks:spillResourceMemTrackerGuard<stdweak ptr<starrockspipeline:QueryContext>, std:weak ptr<starrocksspill:Spiller> > >(starrocks::RuntimeState*, std::sha@ @ 0xd3ead17 starrocks:Aggregator:spill_aggregate_data(starrocks::RuntimeState*, std:function<starrocksStatusOr<stdshared ptr<starrocks:Chunk> > ()>) @ 0xee1bf73 starrocks:pipelineSpillablePartitionWiseAggregateSinkOperator:_spill_all_data(starrocks::RuntimeState*, bool) @ 0xee2c529 starrocks:pipelineSpillablePartitionWiseAggregateSinkOperator:_try_to_spill_by_auto(starrocks::RuntimeState*, std:shared ptr<starrocks:Chunk> const&) @ 0xee2ce37 starrocks:pipelineSpillablePartitionWiseAggregateSinkOperator:push_chunk(starrocks::RuntimeState*, std:shared ptr<starrocks:Chunk> const&) @ 0xd32e8f6 starrocks:pipelinePipelineDriver:process(starrocks::RuntimeState*, int) @ 0xe99a290 starrocks:pipelineGlobalDriverExecutor:_worker_thread() @ 0x10e0cb5b starrocks:ThreadPool:dispatch_thread() @ 0x10e02f9e starrocks:🧵:supervise_thread(void*) I20260904 114131.368565 135065117914816 service.cc:56] Add shard id: 138066019, fs type: GS, fs key: 36f9e737-cc32-4b36-a4e4-f8471a247795, file path: gs://fdp-starrocks-metricstore-prod/warehouse/19701ee5-5ffc-4945-b6f0-5c3dc94a574d/db10235/95679113/138066014, cache_enable: 1, cache_ttl: 0, replicas210; hash_code: 2192, base_fs_info: I20260904 114131.375690 135065117914816 service.cc:56] Add shard id: 138066068, fs type: GS, fs key: 36f9e737-cc32-4b36-a4e4-f8471a247795, file path: gs://fdp-starrocks-metricstore-prod/warehouse/19701ee5-5ffc-4945-b6f0-5c3dc94a574d/db10235/95673998/138066045, cache_enable: 1, cache_ttl: 0, replicas210; hash_code: 2192, base_fs_info: @ 0x7b391625caa4 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x9caa3) I20260904 114131.417960 135065117914816 service.cc:56] Add shard id: 138066117, fs type: GS, fs key: 36f9e737-cc32-4b36-a4e4-f8471a247795, file path: gs://fdp-starrocks-metricstore-prod/warehouse/19701ee5-5ffc-4945-b6f0-5c3dc94a574d/db10235/95683210/138066112, cache_enable: 1, cache_ttl: 0, replicas210; hash_code: 2192, base_fs_info: I20260904 114131.419175 135065117914816 service.cc:56] Add shard id: 138066101, fs type: GS, fs key: 36f9e737-cc32-4b36-a4e4-f8471a247795, file path: gs://fdp-starrocks-metricstore-prod/warehouse/19701ee5-5ffc-4945-b6f0-5c3dc94a574d/db10235/95697268/138066081, cache_enable: 1, cache_ttl: 0, replicas210; hash_code: 2192, base_fs_info: I20260904 114131.419813 135065117914816 service.cc:56] Add shard id: 138066029, fs type: GS, fs key: 36f9e737-cc32-4b36-a4e4-f8471a247795, file path: gs://fdp-starrocks-metricstore-prod/warehouse/19701ee5-5ffc-4945-b6f0-5c3dc94a574d/db10235/95679113/138066014, cache_enable: 1, cache_ttl: 0, replicas210; hash_code: 2192, base_fs_info: I20260904 114131.429621 135065117914816 service.cc:56] Add shard id: 138066051, fs type: GS, fs key: 36f9e737-cc32-4b36-a4e4-f8471a247795, file path: gs://fdp-starrocks-metricstore-prod/warehouse/19701ee5-5ffc-4945-b6f0-5c3dc94a574d/db10235/95673998/138066045, cache_enable: 1, cache_ttl: 0, replicas210; hash_code: 2192, base_fs_info: I20260904 114131.433627 135211265963712 agent_server.cpp:497] Submit task success. type=CREATE, signatures=138066019,138066029, task_count_in_queue=2 I20260904 114131.435478 135065117914816 service.cc:56] Add shard id: 138066119, fs type: GS, fs key: 36f9e737-cc32-4b36-a4e4-f8471a247795, file path: gs://fdp-starrocks-metricstore-prod/warehouse/19701ee5-5ffc-4945-b6f0-5c3dc94a574d/db10235/95683210/138066112, cache_enable: 1, cache_ttl: 0, replicas210; hash_code: 2192, base_fs_info: I20260904 114131.437700 135065117914816 service.cc:56] Add shard id: 138066103, fs type: GS, fs key: 36f9e737-cc32-4b36-a4e4-f8471a247795, file path: gs://fdp-starrocks-metricstore-prod/warehouse/19701ee5-5ffc-4945-b6f0-5c3dc94a574d/db10235/95697268/138066081, cache_enable: 1, cache_ttl: 0, replicas210; hash_code: 2192, base_fs_info: I20260904 114131.440541 135469194528448 agent_server.cpp:497] Submit task success. type=CREATE, signatures=138066117,138066119, task_count_in_queue=5 I20260904 114131.441458 135469194528448 agent_server.cpp:497] Submit task success. type=CREATE, signatures=138066101,138066103, task_count_in_queue=7 I20260904 114131.442952 135211265963712 agent_server.cpp:497] Submit task success. type=CREATE, signatures=138066051,138066068, task_count_in_queue=8 @ 0x7b39162e9a64 __clone /opt/starrocks/cn_entrypoint.sh: line 165: 27 Segmentation fault $STARROCKS_HOME/bin/start_cn.sh $addition_args

Based on the stack trace provided, the segmentation fault occurs in the

PartitionedSpillerWriter::_compact_skew_chunks

function during an aggregation spill-to-disk operation. Cause of the Crash This is a known issue (tracked as Issue #74081) in StarRocks. The root cause is a synchronization or state inconsistency error where the

num_rows

snapshot exceeds the actual row count in a data chunk. This typically happens when a partition-flush operation (part of the spill-to-disk logic for skewed data) interleaves with resource group cancellation or query termination. In your logs, the presence of

_compact_skew_chunks

and

_pick_and_compact_skew_partitions

indicates that the system was trying to handle skewed data during a spill when the memory access became invalid, leading to the

SIGSEGV

. Is it fixed in 4.1.3? Yes, this issue is fixed in StarRocks version 4.1.1 and later. Since version 4.1.3 (released July 14, 2026) is a subsequent patch release, it includes the fix for #74081. If you are currently running a version prior to 4.1.1 (such as an early 4.0.x or a 4.1.0 version), upgrading to 4.1.3 will resolve this specific segmentation fault. Recommendations 1. Upgrade: Move to version 4.1.3 or the latest available patch (e.g., 4.1.4, which includes additional stability fixes for spilling under memory pressure, such as #76702). 2. Verify Configuration: Ensure your spill-to-disk configuration (

spill_local_storage_dir

) is healthy and has sufficient disk space, as skew-handling increases I/O activity. 3. Monitor Workgroups: Since the crash is triggered by the interaction between flushing and resource group management, monitor if specific high-concurrency or frequently cancelled queries are the primary triggers. If you continue to see crashes on version 4.1.3 or later with the exact same stack trace, please collect the core dump and provide it to the StarRocks community for further analysis. References • knowledge/trouble_shooting/fe_deadlock_problems.md • knowledge/admin_and_operation/shared_data_compaction.md • administration/management/resource_management/spill_to_disk.md