Hi StarRocks team — asking for help diagnosing a BE crash that looks like data corruption during page read/decompression. with Starrocks version 3.5.0 BE pod

kube-starrocks-be-0

is CrashLoopBackOff — last exit code 139 (SIGSEGV) during startup while loading tablets from disk. Stack trace points to page read / decompression (StoragePageCache -> PageIO -> ColumnReader -> RowsetUpdateState) — suggests on-disk rowset/page corruption. Impact • FE reports heartbeat failures to

be-0

and “Tablet lost replicas” (loads/queries rolling back). • Other BEs (

be-1

,

be-2

) are healthy; issue looks local to

be-0

storage. How can we recover this situation ? Is this a known bug or regression in PageIO/decompression for this version? Thank you for your help. Full error message:

I20251029 14:56:33.019552 139807042827968 daemon.cpp:139] Current memory statistics: process(10145344), query_pool(0), load(0), metadata(0), compaction(0), schema_change(0), page_cache(0), update(0), chunk_allocator(0), passthrough(0), clone(0), consistency(0), datacache(0), jit(0)
I20251029 14:56:33.020976 139807002982080 data_dir.cpp:133] path: /opt/starrocks/be/storage, hash: -4679337503991790933
I20251029 14:56:33.027702 139807002982080 kv_store.cpp:106] rocksdb write buffer size: 173946175, total memory: 6957847018, disk count: 1
I20251029 14:56:33.059071 139806659049152 data_dir.cpp:277] begin loading tablet from meta /opt/starrocks/be/storage
f051cf0 RELEASE (build f051cf0)
query_id:00000000-0000-0000-0000-000000000000, fragment_instance:00000000-0000-0000-0000-000000000000
*** Aborted at 1761749798 (unix time) try "date -d @1761749798" if you are using GNU date ***
PC: @          0x84c5d03 starrocks::StoragePageCache::lookup(starrocks::StoragePageCache::CacheKey const&, starrocks::PageCacheHandle*)
*** SIGSEGV (@0x8) received by PID 27 (TID 0x7f26e1bfd6c0) LWP(231) from PID 8; stack trace: ***
    @     0x7f277bff6ed3 (/usr/lib/x86_64-linux-gnu/libc.so.6+0xa1ed2)
    @          0xf826369 google::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*)
    @     0x7f277bf9a330 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x4532f)
    @          0x84c5d03 starrocks::StoragePageCache::lookup(starrocks::StoragePageCache::CacheKey const&, starrocks::PageCacheHandle*)
    @          0x84c2fea starrocks::PageIO::read_and_decompress_page(starrocks::PageReadOptions const&, starrocks::PageHandle*, starrocks::Slice*, starrocks::PageFooterPB*)
    @          0x84aadff starrocks::ColumnReader::read_page(starrocks::ColumnIteratorOptions const&, starrocks::PagePointer const&, starrocks::PageHandle*, starrocks::Slice*, starrocks::PageFooterPB*)
    @          0x8572dcb starrocks::ScalarColumnIterator::_read_data_page(starrocks::OrdinalPageIndexIterator const&)
    @          0x85733bf starrocks::ScalarColumnIterator::seek_to_ordinal(unsigned long)
    @          0x85742e6 starrocks::ScalarColumnIterator::fetch_values_by_rowid(unsigned int const*, unsigned long, starrocks::Column*)
    @          0x8b35382 starrocks::TabletUpdates::get_column_values(std::vector<unsigned int, std::allocator<unsigned int> > const&, long, bool, std::map<unsigned int, std::vector<unsigned int, std::allocator<unsigned int> >, std::less<unsigned int>, std::allocator<std::pair<unsiT
    @          0x8b85e18 starrocks::RowsetUpdateState::_prepare_partial_update_states(starrocks::Tablet*, starrocks::Rowset*, unsigned int, bool, std::shared_ptr<starrocks::TabletSchema const> const&)
    @          0x8b873ef starrocks::RowsetUpdateState::_do_load(starrocks::Tablet*, starrocks::Rowset*)
    @          0x8b879db starrocks::RowsetUpdateState::load(starrocks::Tablet*, starrocks::Rowset*)::{lambda()#1}::operator()() const
    @     0x7f277bff6ed3 (/usr/lib/x86_64-linux-gnu/libc.so.6+0xa1ed2)
    @          0x8b7ff2a starrocks::RowsetUpdateState::load(starrocks::Tablet*, starrocks::Rowset*)
    @          0x8b498e8 starrocks::TabletUpdates::_apply_normal_rowset_commit(starrocks::EditVersionInfo const&, std::shared_ptr<starrocks::Rowset> const&)
    @          0x8b4ebb5 starrocks::TabletUpdates::_apply_rowset_commit(starrocks::EditVersionInfo const&)
    @          0x8b4efa8 starrocks::TabletUpdates::do_apply()
    @          0xb966ae3 starrocks::ThreadPool::dispatch_thread()
    @          0xb95db19 starrocks::Thread::supervise_thread(void*)
    @     0x7f277bff1aa4 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x9caa3)
    @     0x7f277c07ec3c (/usr/lib/x86_64-linux-gnu/libc.so.6+0x129c3b)
[1761749798.505][thread: 139804972930752] je_mallctl execute purge success
[1761749798.505][thread: 139804972930752] je_mallctl execute dontdump success
/opt/starrocks/be_entrypoint.sh: line 165:    27 Segmentation fault      (core dumped) $STARROCKS_HOME/bin/start_be.sh $addition_args

how about disable the storagepagecache, will it still crash?

We tried to enable this parameter:

disable_storage_page_cache

, but it did not help. We succeed to make the backend up again by downgrading Starrocks from 3.5.0 to 3.4.0. But I would be happy to understand more what happened and if it can happen again when we will upgrade

didn't find

f051cf0

from starrocks repo, guess there are some patches applied to the build. may need to enable the coredump generation and attach gdb to analyze the core.