https://github.com/StarRocks/starrocks/issues/71283 Hi everyone, @Rocky. [Bug] SIGSEGV in

ColumnReader::~ColumnReader()

calling

protobuf::Message::SpaceUsedLong()

during

publish_version

/ lake tablet metadata caching (shared-data mode) Environment StarRocks version:

4.0.8 RELEASE (build 2090bfc distro ubuntu arch x86_64)

Deployment: Kubernetes, CN-only cluster (shared-data / lake mode), 15 CN pods Component: CN (Compute Node) • Run mode:

shared_data

Steps to reproduce the behavior (Required) The crash occurs during background

publish_version

operations — no explicit user query is needed to trigger it. The conditions observed: 1. Run a CN cluster in

shared_data

mode with datacache enabled. 2. Sustain continuous data ingestion via Routine Load (Kafka) producing frequent version publishes. 3. Observe SIGSEGV crash during compaction tasks. Expected behavior (Required) CN pod runs stably. Tablet metadata caching and version publishing complete without crashing. Real behavior (Required) CN pod crashes with SIGSEGV. 5 out of 15 CN pods crashed within a ~30-minute window (1459–1528 UTC+2). Two variants observed depending on which protobuf method is hit at the crash site. Stack trace — Variant A:

Message::SpaceUsedLong

(CN-3, CN-14)

*** Aborted at 1775221180 (unix time) try "date -d @1775221180" if you are using GNU date ***
PC: @         0x1251cf83 google::protobuf::Message::SpaceUsedLong() const
*** SIGSEGV (@0x98) received by PID 22 (TID 0x7f5e1d9fe640) LWP(806) from PID 152; stack trace: ***
    @     0x7f5f0ee7dee8 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x99ee7)
    @         0x11fc49a8 google::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*)
    @     0x7f5f0ee26520 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x4251f)
    @         0x1251cf83 google::protobuf::Message::SpaceUsedLong() const
    @          0xd72453d starrocks::ColumnReader::~ColumnReader()
    @          0xd712479 std::_Rb_tree<int, std::pair<int const, std::unique_ptr<starrocks::ColumnReader, std::default_delete<starrocks::ColumnReader> > >, std::_Select1st<std::pair<int const, std::unique_ptr<starrocks::ColumnReader, std::default_delete<starrocks::ColumnReader> > > >, std::less<int>, std::allocator<std::pair<int const, std::unique_ptr<starrocks::ColumnReader, std::default_delete<starrocks::ColumnReader> > > > >::_M_erase(std::_Rb_tree_node<std::pair<int const, std::unique_ptr<starrocks::ColumnReader, std::default_delete<starrocks::ColumnReader> > > >*)
    @          0xd716008 starrocks::Segment::~Segment()
    @          0x8c99206 std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release_last_use_cold()
    @          0xdd2766c starrocks::lake::Metacache::cache_value_deleter(starrocks::CacheKey const&, void*)
    @          0xe45ba5d starrocks::LRUCache::insert(starrocks::CacheKey const&, unsigned int, void*, unsigned long, void (*)(starrocks::CacheKey const&, void*), starrocks::CachePriority)
    @          0xe45bd66 starrocks::ShardedLRUCache::insert(starrocks::CacheKey const&, void*, unsigned long, void (*)(starrocks::CacheKey const&, void*), starrocks::CachePriority)
    @          0xdd2492e starrocks::lake::Metacache::cache_tablet_metadata(std::basic_string_view<char, std::char_traits<char> >, std::shared_ptr<starrocks::TabletMetadataPB const>)
    @          0xdc5b899 starrocks::lake::TabletManager::cache_tablet_metadata(std::shared_ptr<starrocks::TabletMetadataPB const> const&)
    @          0xe265313 starrocks::lake::NonPrimaryKeyTxnLogApplier::finish()
    @          0xe25fe3a starrocks::lake::publish_version(starrocks::lake::TabletManager*, long, long, long, std::span<starrocks::TxnInfoPB const, 18446744073709551615ul>, bool)
    @          0xe23ea1b starrocks::LakeServiceImpl::publish_version(google::protobuf::RpcController*, starrocks::PublishVersionRequest const*, starrocks::PublishVersionResponse*, google::protobuf::Closure*)::{lambda()#1}::operator()() const
    @          0xe439e68 std::_Function_handler<void (), starrocks::ConcurrencyLimitedThreadPoolToken::submit(std::shared_ptr<starrocks::Runnable>, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >)::{lambda()#1}>::_M_invoke(std::_Any_data const&)
    @          0xe44172e starrocks::ThreadPool::dispatch_thread()
    @          0xe4381a5 starrocks::Thread::supervise_thread(void*)
    @     0x7f5f0ee78ac3 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x94ac2)
    @     0x7f5f0ef0a8d0 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x1268cf)

Same stack also observed on CN-14 at

1775222880

(same PC

0x1251cf83

, same call chain). Stack trace — Variant B:

Reflection::SpaceUsedLong

(CN-10)

*** Aborted at 1775221754 (unix time) try "date -d @1775221754" if you are using GNU date ***
PC: @         0x12502dd9 google::protobuf::Reflection::SpaceUsedLong(google::protobuf::Message const&) const
*** SIGSEGV (@0x88) received by PID 22 (TID 0x7f863daf1640) LWP(806) from PID 136; stack trace: ***
    @     0x7f873cf17ee8 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x99ee7)
    @         0x11fc49a8 google::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*)
    @     0x7f873cec0520 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x4251f)
    @         0x12502dd9 google::protobuf::Reflection::SpaceUsedLong(google::protobuf::Message const&) const
    @          0xd724685 starrocks::ColumnReader::~ColumnReader()
    @          0xd712479 std::_Rb_tree<int, std::pair<int const, std::unique_ptr<starrocks::ColumnReader, std::default_delete<starrocks::ColumnReader> > >, std::_Select1st<std::pair<int const, std::unique_ptr<starrocks::ColumnReader, std::default_delete<starrocks::ColumnReader> > > >, std::less<int>, std::allocator<std::pair<int const, std::unique_ptr<starrocks::ColumnReader, std::default_delete<starrocks::ColumnReader> > > > >::_M_erase(std::_Rb_tree_node<std::pair<int const, std::unique_ptr<starrocks::ColumnReader, std::default_delete<starrocks::ColumnReader> > > >*)
    @          0xd71225c std::_Rb_tree<int, std::pair<int const, std::unique_ptr<starrocks::ColumnReader, std::default_delete<starrocks::ColumnReader> > >, std::_Select1st<std::pair<int const, std::unique_ptr<starrocks::ColumnReader, std::default_delete<starrocks::ColumnReader> > > >, std::less<int>, std::allocator<std::pair<int const, std::unique_ptr<starrocks::ColumnReader, std::default_delete<starrocks::ColumnReader> > > > >::erase(std::_Rb_tree_const_iterator<std::pair<int const, std::unique_ptr<starrocks::ColumnReader, std::default_delete<starrocks::ColumnReader> > > >)
    @          0xd716008 starrocks::Segment::~Segment()
    @          0x8c99206 std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release_last_use_cold()
    @          0xdd2766c starrocks::lake::Metacache::cache_value_deleter(starrocks::CacheKey const&, void*)
    @          0xe45ba5d starrocks::LRUCache::insert(starrocks::CacheKey const&, unsigned int, void*, unsigned long, void (*)(starrocks::CacheKey const&, void*), starrocks::CachePriority)
    @          0xe45bd66 starrocks::ShardedLRUCache::insert(starrocks::CacheKey const&, void*, unsigned long, void (*)(starrocks::CacheKey const&, void*), starrocks::CachePriority)
    @          0xdd2492e starrocks::lake::Metacache::cache_tablet_metadata(std::basic_string_view<char, std::char_traits<char> >, std::shared_ptr<starrocks::TabletMetadataPB const>)
    @          0xdc5b899 starrocks::lake::TabletManager::cache_tablet_metadata(std::shared_ptr<starrocks::TabletMetadataPB const> const&)
    @          0xe265313 starrocks::lake::NonPrimaryKeyTxnLogApplier::finish()
    @          0xe25fe3a starrocks::lake::publish_version(starrocks::lake::TabletManager*, long, long, long, std::span<starrocks::TxnInfoPB const, 18446744073709551615ul>, bool)
    @          0xe23ea1b starrocks::LakeServiceImpl::publish_version(google::protobuf::RpcController*, starrocks::PublishVersionRequest const*, starrocks::PublishVersionResponse*, google::protobuf::Closure*)::{lambda()#1}::operator()() const
    @          0xe439e68 std::_Function_handler<void (), starrocks::ConcurrencyLimitedThreadPoolToken::submit(std::shared_ptr<starrocks::Runnable>, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >)::{lambda()#1}>::_M_invoke(std::_Any_data const&)
    @          0xe44172e starrocks::ThreadPool::dispatch_thread()
    @          0xe4381a5 starrocks::Thread::supervise_thread(void*)
    @     0x7f873cf12ac3 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x94ac2)
    @     0x7f873cfa48d0 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x1268cf)

Hi everyone, I'm Rocky, the StarRocks assistant. This issue looks like a known critical bug in the shared-data (lake) mode, specifically related to how metadata is cached and cleared in the

Metacache

. Issue Analysis The SIGSEGV in

ColumnReader::~ColumnReader()

calling

protobuf::Message::SpaceUsedLong()

indicates a use-after-free or destruction order issue. In StarRocks,

ColumnReader

objects often hold pointers to protobuf metadata (like

ColumnMetaPB

) which are actually owned by the

SegmentFooterPB

inside a

Segment

object. The stack trace shows the crash is triggered by

starrocks::lake::Metacache::cache_value_deleter

. This deleter is invoked when a

Segment

is evicted from the

Metacache

(LRU cache). During the destruction of the `Segment`: 1. The members of the

Segment

are destroyed. 2. If the

SegmentFooterPB

(which owns the metadata) is destroyed before the

_column_readers

(due to their declaration order in the

Segment

class), then the

ColumnReader

destructor will attempt to access the already-destroyed protobuf message to calculate its memory usage (

SpaceUsedLong()

). 3. This results in the observed SIGSEGV. Status & Recommendations This issue is being tracked in the GitHub issue you linked: #71283. 1. Upgrade to a Fixed Version This is a regression or a newly discovered bug in the 4.0.x branch. Please check for the release of StarRocks 4.0.9 or later, as this issue is likely to be prioritized for a hotfix. 2. Potential Workarounds (Use with Caution) Since the crash is triggered by cache eviction in the

Metacache

, you might temporarily mitigate the frequency of crashes by: * Increasing the Metadata Cache Limit: This can delay evictions, but it will not prevent the crash once the cache fills up. * Check the BE parameter:

lake_metadata_cache_limit

(if applicable in your version). * Reducing Ingestion Concurrency: High-frequency Routine Load produces many versions, leading to frequent metadata updates and cache evictions. Reducing the frequency of version publishing (e.g., increasing

desired_max_interval_ms

in Routine Load) might slow down the occurrence. 3. Debugging Information If you are in contact with support, providing the

core dump

and the

be.out

file (which you've already started to analyze) is the most helpful step. I'll keep an eye on the development of this issue. Let me know if you need help with specific BE configuration adjustments! References • knowledge/trouble_shooting/fe_deadlock_problems.md • release_notes/release-2.1.md • release_notes/release-3.0.md