I am usingf starrocks db version 4.0.3, shared dat...
# questions-and-troubleshooting
y
I am usingf starrocks db version 4.0.3, shared data configuration with 2 fe and 3 cn nodes. After I dropped 2 tables and created new ones with the same name but a different type (not primary key anymore, but duplicate key), I now encounter SIGSEGV crashes in the cn nodes when performing simple selects such as `select count(*) from XYZ`; What is even more trubling is that now even when I run the same select statement on a different table, the same SIGSEGV error happens. Anyone any idea ? The error:
Copy code
I20260415 02:46:26.790360 140045107308096 daemon.cpp:140] Current memory statistics: process(397565048), query_pool(0), load(0), metadata(17860687), compaction(0), schema_change(0), page_cache(3006937), update
4.0.3 RELEASE (build fbcf73d distro ubuntu arch x86_64)
query_id:019d8d50-f7f9-7676-a9e6-04e2c76cd7d8, fragment_instance:019d8d50-f7f9-7676-a9e6-04e2c76cd7d9, plan_node_id:0
*** Aborted at 1776192387 (unix time) try "date -d @1776192387" if you are using GNU date ***
PC: @          0x8bc5597 starrocks::BinaryColumnBase<unsigned int>::append(starrocks::Column const&, unsigned long, unsigned long)
*** SIGSEGV (@0x0) received by PID 39 (TID 0x7f5e53251640) LWP(366) from PID 0; stack trace: ***
    @     0x7f5ed47f8ee8 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x99ee7)
    @         0x11d20fc8 google::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*)
    @     0x7f5ed47a1520 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x4251f)
    @          0x8bc5597 starrocks::BinaryColumnBase<unsigned int>::append(starrocks::Column const&, unsigned long, unsigned long)
    @          0xc7d3022 starrocks::pipeline::UnionConstSourceOperator::pull_chunk(starrocks::RuntimeState*)
    @          0xae86f7e starrocks::pipeline::PipelineDriver::process(starrocks::RuntimeState*, int)
    @          0xc123fee starrocks::pipeline::GlobalDriverExecutor::_worker_thread()
    @          0xe1be4de starrocks::ThreadPool::dispatch_thread()
    @          0xe1b5085 starrocks::Thread::supervise_thread(void*)
    @     0x7f5ed47f3ac3 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x94ac2)
    @     0x7f5ed48858c0 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x1268bf)
[1776192387.223][thread: 140043098592832] je_mallctl execute purge
@Rocky Can you advise?
r
Hello! I'm Rocky, your StarRocks assistant. I’d be happy to advise you! To give you the most helpful guidance, could you please provide more details? For example: * Are you encountering a specific SQL error or syntax issue? * Are you looking for advice on table design (Primary Key vs. Duplicate Key, partitioning, etc.)? * Do you need help with performance tuning or a specific feature (like Materialized Views or Data Loading)? Feel free to share your query, error message, or use case, and I'll look into the information up for you!
y
@Rocky I am usingf starrocks db version 4.0.3, shared data configuration with 2 fe and 3 cn nodes. After I dropped 2 tables and created new ones with the same name but a different type (not primary key anymore, but duplicate key), I now encounter SIGSEGV crashes in the cn nodes when performing simple selects such as `select count(*) from XYZ`; What is even more trubling is that now even when I run the same select statement on a different table, the same SIGSEGV error happens. Anyone any idea ? The error:
Copy code
I20260415 02:46:26.790360 140045107308096 daemon.cpp:140] Current memory statistics: process(397565048), query_pool(0), load(0), metadata(17860687), compaction(0), schema_change(0), page_cache(3006937), update
4.0.3 RELEASE (build fbcf73d distro ubuntu arch x86_64)
query_id:019d8d50-f7f9-7676-a9e6-04e2c76cd7d8, fragment_instance:019d8d50-f7f9-7676-a9e6-04e2c76cd7d9, plan_node_id:0
*** Aborted at 1776192387 (unix time) try "date -d @1776192387" if you are using GNU date ***
PC: @          0x8bc5597 starrocks::BinaryColumnBase<unsigned int>::append(starrocks::Column const&, unsigned long, unsigned long)
*** SIGSEGV (@0x0) received by PID 39 (TID 0x7f5e53251640) LWP(366) from PID 0; stack trace: ***
    @     0x7f5ed47f8ee8 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x99ee7)
    @         0x11d20fc8 google::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*)
    @     0x7f5ed47a1520 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x4251f)
    @          0x8bc5597 starrocks::BinaryColumnBase<unsigned int>::append(starrocks::Column const&, unsigned long, unsigned long)
    @          0xc7d3022 starrocks::pipeline::UnionConstSourceOperator::pull_chunk(starrocks::RuntimeState*)
    @          0xae86f7e starrocks::pipeline::PipelineDriver::process(starrocks::RuntimeState*, int)
    @          0xc123fee starrocks::pipeline::GlobalDriverExecutor::_worker_thread()
    @          0xe1be4de starrocks::ThreadPool::dispatch_thread()
    @          0xe1b5085 starrocks::Thread::supervise_thread(void*)
    @     0x7f5ed47f3ac3 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x94ac2)
    @     0x7f5ed48858c0 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x1268bf)
[1776192387.223][thread: 140043098592832] je_mallctl execute purge
r
Based on the stack trace you provided and the specific version (4.0.3), you have encountered a known issue in the StarRocks 4.0.x series related to an optimization rule called
RewriteSimpleAggToMetaScanRule
. This rule attempts to speed up simple queries like
SELECT COUNT(*)
by reading metadata (MetaScan) instead of scanning the actual data. However, in version 4.0.3, there is a bug where this rewrite incorrectly handles nullable properties or column offsets (specifically in the
UnionConstSourceOperator
), leading to the
SIGSEGV
crash in the CN nodes you are seeing. Why is this happening? 1. The Trigger: When you dropped and recreated the tables (especially changing from Primary Key to Duplicate Key), the metadata changed. The optimizer in 4.0.3 may be aggressively trying to use the "MetaScan" optimization for these new tables, but failing due to a mismatch in the internal column structures (the
BinaryColumnBase::append
error in your trace refers to the "offsets" of a string/binary column being handled incorrectly). 2. Global Impact: Because this is a global optimizer rule, once it's triggered, it often attempts to apply the same faulty logic to other tables in the same database or even different databases, which is why you are seeing crashes on tables you didn't even touch. ────────── Immediate Workaround You can disable this specific optimization rule globally. This will force StarRocks to perform a standard scan for
COUNT(*)
instead of using the problematic MetaScan path. Run the following command as a user with administrative privileges:
Copy code
sql
SET GLOBAL enable_rewrite_simple_agg_to_meta_scan = false;
Note: You may need to restart your CN nodes if they are stuck in a crash loop, but once this variable is set to
false
, the FE will stop generating the plan that causes the CN to crash. ────────── Recommended Long-term Fix This issue is addressed in later patch releases of the 4.0 branch. I strongly recommend upgrading to v4.0.5 or later. * Multiple "MetaScan" and
UnionConstSourceOperator
stability fixes were included in 4.0.5. * Version 4.0.3 has several known stability issues in shared-data mode that are resolved in more recent patches. Additional Step (If crashes persist) In a shared-data configuration, a crash can sometimes leave behind corrupted or partially written entries in the local disk cache (Data Cache). If the
SET GLOBAL
command above doesn't resolve the issue for existing tables: 1. Stop the CN nodes. 2. Clear the Data Cache directory on each CN node (usually located under your
storage_root_path/datacache/
). 3. Restart the CN nodes. This ensures that any "poisoned" metadata or data blocks cached locally from the previous crashes are cleared. Referencesfaq/shared_data_faq.mdquick_start/shared-data.mddeployment/shared_data/azure.md