http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> I am ingesting some parquet files a...
# questions-and-troubleshooting
y
@Rocky I am ingesting some parquet files and some CN nodes are crashing with this error repeatedly:
Copy code
I20260506 07:04:29.760780 140002388260416 load_chunk_spiller.cpp:289] LoadChunkSpiller merge finished, load_id:TUniqueId(hi=116526348650707419, lo=-5754397456657471362) fragment_instance_id:TUniqueId(hi=8228754.0.8 RELEASE (build 2090bfc distro ubuntu arch x86_64) query_id:00000000-0000-0000-0000-000000000000, fragment_instance:00000000-0000-0000-0000-000000000000, plan_node_id:-1
*** Aborted at 1778051069 (unix time) try "date -d @1778051069" if you are using GNU date ***
PC: @          0xe34fa54 starrocks::JsonValue::to_vslice() const
*** SIGSEGV (@0x0) received by PID 39 (TID 0x7f543e7f6640) LWP(423) from PID 0; stack trace: ***
    @     0x7f54f8fc1ee8 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x99ee7)
    @         0x11fc49a8 google::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*)
    @     0x7f54f8f6a520 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x4251f)
I20260506 07:04:29.904269 140000686282304 tablet_sink_index_channel.cpp:905] OlapTableSink txn_id: 722343 load_id: 019dfc18-c24c-79db-b024-46005385287e commit 7 tablets: 822875,822877,822878,822880,822882,8228
    @          0xe34fa54 starrocks::JsonValue::to_vslice() const
    @          0xe37b786 void starrocks::JsonMerger::_merge_impl<true>(unsigned long)
    @          0xe36df4d starrocks::JsonMerger::merge(std::vector<starrocks::Cow<starrocks::Column>::ImmutPtr<starrocks::Column>, std::allocator<starrocks::Cow<starrocks::Column>::ImmutPtr<starrocks::Column> >
    @          0xd7bf4d3 starrocks::JsonMergeIterator::next_batch(starrocks::SparseRange<unsigned int> const&, starrocks::Column*)
    @          0xd87c349 starrocks::SegmentIterator::ScanContext::read_columns(starrocks::Chunk*, starrocks::SparseRange<unsigned int> const&)
    @          0xd8a912a starrocks::SegmentIterator::_read(starrocks::Chunk*, std::vector<unsigned int, std::allocator<unsigned int> >*, unsigned long)
    @          0xd88fc4f starrocks::SegmentIterator::_do_get_next(starrocks::Chunk*, std::vector<unsigned int, std::allocator<unsigned int> >*)
    @          0xd89f0c2 starrocks::SegmentIterator::do_get_next(starrocks::Chunk*)
    @          0xdaafd1f starrocks::MaskMergeIterator::fill(unsigned long)
    @          0xdab403e starrocks::MaskMergeIterator::do_get_next(starrocks::Chunk*, std::vector<starrocks::RowSourceMask, std::allocator<starrocks::RowSourceMask> >*)
    @          0xdcc4fde starrocks::lake::TabletReader::do_get_next(starrocks::Chunk*, std::vector<starrocks::RowSourceMask, std::allocator<starrocks::RowSourceMask> >*)
    @          0xdd1de0d starrocks::lake::VerticalCompactionTask::compact_column_group(bool, int, unsigned long, std::vector<unsigned int, std::allocator<unsigned int> > const&, std::unique_ptr<starrocks::lake
    @          0xdd1e944 starrocks::lake::VerticalCompactionTask::execute(std::function<starrocks::Status ()>, starrocks::ThreadPool*)
    @          0xdc3f372 starrocks::lake::CompactionScheduler::do_compaction(std::unique_ptr<starrocks::lake::CompactionTaskContext, std::default_delete<starrocks::lake::CompactionTaskContext> >)
    @          0xdc402e6 starrocks::lake::CompactionScheduler::thread_task(int)
    @          0xe44172e starrocks::ThreadPool::dispatch_thread()
    @          0xe4381a5 starrocks::Thread::supervise_thread(void*)
    @     0x7f54f8fbcac3 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x94ac2)
    @     0x7f54f904e8d0 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x1268cf)
I20260506 07:04:30.686015 140002749052480 daemon.cpp:142] Current memory statistics: process(6855195952) query_pool(24108840) load(0) metadata(8907101) compaction(5617033320) schema_change(0) page_cache(842698
[1778051070.451][thread: 139999802517056] je_mallctl execute purge success
[1778051070.451][thread: 139999802517056] je_mallctl execute dontdump success
How can I fix this ?
r
The crash log you provided indicates a Segmentation Fault (SIGSEGV) occurring in the 
starrocks::JsonValue::to_vslice()
function during a Vertical Compaction task on a shared-data (Lakehouse) table. This specific crash is often related to a known issue in older versions of StarRocks 3.x where the internal pointer cache for JSON columns could become invalid (dangling pointers) during complex operations like vertical compaction or filtering, leading to a null pointer dereference. Recommended Fixes and Workarounds 1. Disable Flat JSON Compaction (Immediate Workaround) A likely cause is the Flat JSON feature (which automatically optimizes JSON sub-fields). If you are on version 3.3.3 or higher, this is enabled by default for compaction. You can disable it to see if the crashes stop: * Action: Add/Update this parameter in your 
be.conf
or 
cn.conf
(or via 
curl
for a hot-fix):
Copy code
properties
    enable_compaction_flat_json = false
* Dynamic Update (no restart):
Copy code
bash
    curl -X POST http://<CN_IP>:<BE_HTTP_PORT>/update_config?enable_compaction_flat_json=false
2. Disable Vertical Compaction The stack trace explicitly mentions 
VerticalCompactionTask
. If disabling Flat JSON compaction doesn't work, you can disable vertical compaction entirely. This will cause the system to use the standard (horizontal) compaction path, which is more memory-intensive but avoids the specific code path causing the crash. * Action: Add to your `be.conf`/`cn.conf`:
Copy code
properties
    enable_vertical_compaction = false
* Dynamic Update:
Copy code
bash
    curl -X POST http://<CN_IP>:<BE_HTTP_PORT>/update_config?enable_vertical_compaction=false
3. Permanent Fix: Upgrade StarRocks This crash is fixed in newer patch releases of the 3.3 and 3.4 branches. Specifically, issues like #66990 (dangling pointers in JSON columns) and #65260 (crash on specific JSON paths) were addressed in: * StarRocks v3.4.10+ * StarRocks v3.3.6+ (approximate patch version, check release notes for your specific branch) * StarRocks v3.5.x Why is this happening? When you ingest Parquet files with JSON columns, StarRocks attempts to merge "rowsets" in the background (compaction). Vertical Compaction processes columns in groups to save memory. During this process: 1. The 
JsonMerger
attempts to combine JSON data from different segments. 2. If the internal memory for a 
JsonValue
has been resized or moved (common in Parquet/JSON processing), a pointer might point to stale memory. 3. Calling 
to_vslice()
(to get a string representation of the JSON) then tries to access that invalid address, causing the CN to crash. Next Steps for Investigation If the crashes continue after these changes, please check: 1. JSON Complexity: Are there extremely large JSON objects or deeply nested structures in your Parquet files? 2. Memory: Your log shows 
compaction(5617033320)
(approx 5.6GB). Ensure your CN nodes have enough memory headroom; if the system is under heavy memory pressure, it might exacerbate pointer/trigger pointer invalidation issues. References • knowledge/admin_and_operation/shared_data_compaction.md • knowledge/trouble_shooting/fe_deadlock_problems.md
y
@Rocky I am running starrocks 4.0.8 on K8s on AWS, shared data cluster
r
Hi, I'm Rocky! It looks like your CN nodes are hitting a Segmentation Fault (SIGSEGV) specifically within the 
JsonValue::to_vslice()
function during a Vertical Compaction task on your shared-data cluster. Given that you are running StarRocks v4.0.8, this crash is likely related to the Flat JSON feature, which is enabled by default starting in v4.0. When the system performs vertical compaction, it attempts to merge JSON data using the 
JsonMerger
. If there is a conflict or an edge case in the JSON structure (like a type mismatch at the same path across different rowsets), the internal pointer logic can fail, leading to this crash. Here are the steps to resolve or work around this issue: 1. Disable Flat JSON Compaction (Immediate Workaround) The most direct way to stop this crash without losing the benefits of Flat JSON for new loads is to disable the compaction logic for flattened JSON. This prevents the 
JsonMerger
from running during background compaction. * Action: Update your 
cn.conf
(and 
be.conf
if applicable):
Copy code
properties
    enable_compaction_flat_json = false
* Dynamic Update (No restart required): You can apply this to all CN nodes via `curl`:
Copy code
bash
    curl -X POST http://<CN_IP>:<CN_HTTP_PORT>/update_config?enable_compaction_flat_json=false
2. Disable Vertical Compaction The stack trace explicitly shows the crash happening within 
VerticalCompactionTask
. If the first step doesn't resolve it, you can force the system to use standard (horizontal) compaction. Vertical compaction is more memory-efficient for many columns, but horizontal compaction follows a different, more mature code path for JSON. * Action: Update your `cn.conf`:
Copy code
properties
    enable_vertical_compaction = false
* Dynamic Update:
Copy code
bash
    curl -X POST http://<CN_IP>:<CN_HTTP_PORT>/update_config?enable_vertical_compaction=false
3. Disable Flat JSON for the Specific Table If you suspect a specific table is causing the issue, you can disable the feature for that table entirely (supported in v4.0+):
Copy code
sql
ALTER TABLE <table_name> SET ("flat_json.enable" = "false");
Why is this happening? In StarRocks 4.0, the "Flat JSON" feature automatically extracts sub-fields from JSON columns into a hidden columnar format to speed up queries. During Vertical Compaction, the system tries to merge these sub-fields across different data files (rowsets). The crash in 
to_vslice()
indicates that a memory pointer for a JSON value became invalid during this merging process, causing the process to attempt to read from a null or protected memory address. Permanent Fix This behavior is often a bug related to specific JSON path conflicts (e.g., a path being an 
Object
in one file and an 
Array
in another). * Check if you can upgrade to a later patch (e.g., v4.0.9+ or v4.1) as several 
JsonMerger
and 
array_map
related crashes were addressed in those releases. * Review your Parquet files to see if there are schema changes or "dirty" data where the same JSON key contains different data types across files. References • administration/management/BE_configuration.md • using_starrocks/Flat_json.md
y
@Rocky I added
Copy code
enable_compaction_flat_json = false
enable_vertical_compaction = false
to the 
cn.conf
in the 
configmap
in k8s. I also did
Copy code
ALTER TABLE <table_name> SET ("flat_json.enable" = "false");
to the relevant table, the ones to which I added a lot of json objects in the last day. I am still getting crashes on CN nodes with messages like this:
Copy code
I20260506 08:30:00.341724 140155497588288 <http://service.cc:56]|service.cc:56]> Add shard id: 195795, fs type: S3, fs key: data.warehouse.us-east-2.prod, file path: <s3://data.warehouse.us-east-2.prod/starrocks/d8d6d338-2891-4e79-8103-80f7e62063c9/db132114/195792/195794>, cache_enable: 1, cache_ttl: 0, replicas:3:0; hash_code: 2192
I20260506 08:30:00.341743 140155497588288 <http://service.cc:56]|service.cc:56]> Add shard id: 195799, fs type: S3, fs key: data.warehouse.us-east-2.prod, file path: <s3://data.warehouse.us-east-2.prod/starrocks/d8d6d338-2891-4e79-8103-80f7e62063c9/db132114/195792/195794>, cache_enable: 1, cache_ttl: 0, replicas:3:0; hash_code: 2192
I20260506 08:30:00.341761 140155497588288 <http://service.cc:56]|service.cc:56]> Add shard id: 195830, fs type: S3, fs key: data.warehouse.us-east-2.prod, file path: <s3://data.warehouse.us-east-2.prod/starrocks/d8d6d338-2891-4e79-8103-80f7e62063c9/db132114/195792/195828>, cache_enable: 1, cache_ttl: 0, replicas:3:0; hash_code: 2192
I20260506 08:30:00.341777 140155497588288 <http://service.cc:56]|service.cc:56]> Add shard id: 195832, fs type: S3, fs key: data.warehouse.us-east-2.prod, file path: <s3://data.warehouse.us-east-2.prod/starrocks/d8d6d338-2891-4e79-8103-80f7e62063c9/db132114/195792/195828>, cache_enable: 1, cache_ttl: 0, replicas:3:0; hash_code: 2192
I20260506 08:30:00.341805 140155497588288 <http://service.cc:56]|service.cc:56]> Add shard id: 523837, fs type: S3, fs key: data.warehouse.us-east-2.prod, file path: <s3://data.warehouse.us-east-2.prod/starrocks/d8d6d338-2891-4e79-8103-80f7e62063c9/db132114/523834/523836>, cache_enable: 1, cache_ttl: 0, replicas:3:0; hash_code: 2192
I20260506 08:30:00.341822 140155497588288 <http://service.cc:56]|service.cc:56]> Add shard id: 523841, fs type: S3, fs key: data.warehouse.us-east-2.prod, file path: <s3://data.warehouse.us-east-2.prod/starrocks/d8d6d338-2891-4e79-8103-80f7e62063c9/db132114/523834/523836>, cache_enable: 1, cache_ttl: 0, replicas:3:0; hash_code: 2192
I20260506 08:30:04.826630 140153859339840 <http://service.cc:83]|service.cc:83]> Remove shard 687403
I20260506 08:30:04.827590 140153859339840 <http://service.cc:83]|service.cc:83]> Remove shard 757058
I20260506 08:30:04.832957 140153859339840 <http://service.cc:83]|service.cc:83]> Remove shard 685187
I20260506 08:30:04.833925 140153859339840 <http://service.cc:83]|service.cc:83]> Remove shard 10032
I20260506 08:30:04.839278 140153859339840 <http://service.cc:83]|service.cc:83]> Remove shard 10065
I20260506 08:30:04.840540 140153859339840 <http://service.cc:83]|service.cc:83]> Remove shard 10051
I20260506 08:30:04.845371 140153859339840 <http://service.cc:83]|service.cc:83]> Remove shard 10127
I20260506 08:30:04.846341 140153859339840 <http://service.cc:83]|service.cc:83]> Remove shard 10065
I20260506 08:30:04.850496 140153859339840 <http://service.cc:83]|service.cc:83]> Remove shard 10084
I20260506 08:30:04.856989 140153859339840 <http://service.cc:83]|service.cc:83]> Remove shard 10080
I20260506 08:30:04.858791 140153859339840 <http://service.cc:83]|service.cc:83]> Remove shard 756987
I20260506 08:30:04.859717 140153859339840 <http://service.cc:83]|service.cc:83]> Remove shard 10133
I20260506 08:30:04.868656 140153859339840 <http://service.cc:83]|service.cc:83]> Remove shard 10146
I20260506 08:30:04.869991 140153859339840 <http://service.cc:83]|service.cc:83]> Remove shard 757043
I20260506 08:30:04.870769 140153859339840 <http://service.cc:83]|service.cc:83]> Remove shard 10155
I20260506 08:30:04.871407 140153859339840 <http://service.cc:83]|service.cc:83]> Remove shard 688253
I20260506 08:30:04.874650 140153859339840 <http://service.cc:83]|service.cc:83]> Remove shard 757027
I20260506 08:30:04.878298 140153859339840 <http://service.cc:83]|service.cc:83]> Remove shard 665872
I20260506 08:30:04.880137 140153859339840 <http://service.cc:83]|service.cc:83]> Remove shard 687373
I20260506 08:30:04.892426 140153859339840 <http://service.cc:83]|service.cc:83]> Remove shard 687379
I20260506 08:30:04.917113 140153859339840 <http://service.cc:83]|service.cc:83]> Remove shard 687397
I20260506 08:30:04.924855 140153859339840 <http://service.cc:83]|service.cc:83]> Remove shard 688904
I20260506 08:30:04.945358 140153859339840 <http://service.cc:83]|service.cc:83]> Remove shard 688908
I20260506 08:30:04.946919 140153859339840 <http://service.cc:83]|service.cc:83]> Remove shard 688929
I20260506 08:30:04.956606 140153859339840 <http://service.cc:83]|service.cc:83]> Remove shard 689349
I20260506 08:30:04.958080 140153859339840 <http://service.cc:83]|service.cc:83]> Remove shard 689334
I20260506 08:30:04.978877 140153859339840 <http://service.cc:83]|service.cc:83]> Remove shard 756893
I20260506 08:30:04.998198 140153859339840 <http://service.cc:83]|service.cc:83]> Remove shard 756931
I20260506 08:30:05.009438 140153859339840 <http://service.cc:83]|service.cc:83]> Remove shard 756953
I20260506 08:30:05.036592 140153859339840 <http://service.cc:83]|service.cc:83]> Remove shard 756965
I20260506 08:30:05.038370 140153859339840 <http://service.cc:83]|service.cc:83]> Remove shard 756972
I20260506 08:30:05.040332 140153859339840 <http://service.cc:83]|service.cc:83]> Remove shard 757007
I20260506 08:30:05.049310 140153859339840 <http://service.cc:83]|service.cc:83]> Remove shard 757016
I20260506 08:30:05.056340 140153859339840 <http://service.cc:83]|service.cc:83]> Remove shard 757055
I20260506 08:30:05.062489 140153859339840 <http://service.cc:83]|service.cc:83]> Remove shard 757090
I20260506 08:30:05.064289 140153859339840 <http://service.cc:83]|service.cc:83]> Remove shard 757130
I20260506 08:30:05.072557 140153859339840 <http://service.cc:83]|service.cc:83]> Remove shard 757142
I20260506 08:30:05.074175 140153859339840 <http://service.cc:83]|service.cc:83]> Remove shard 757152
I20260506 08:30:05.075855 140153859339840 <http://service.cc:83]|service.cc:83]> Remove shard 757212
I20260506 08:30:11.990527 140157601576512 pipeline_executor_set_manager.cpp:116] [WORKGROUP] assign shared executors to workgroup [workgroup=(id:2, name:default_wg, version:2, cpu_weight:2, exclusive_cpu_cores:0, mem_limit:12524124634, concurrency_limit:0, bigquery: (cpu_second_limit:0, mem_limit:0, scan_rows_limit:0), spill_mem_limit_threshold:1)]
I20260506 08:30:12.202271 140157356639808 tablet_sink_sender.cpp:357] Olap table sink statistics. load_id: 019dfc68-ae01-7f19-b1a8-d2f037fe8333, txn_id: 722774, add chunk time(ms)/wait lock time(ms)/num: {10003:(0)(0)(1)} {10002:(0)(0)(1)} {777090:(0)(0)(1)}
I20260506 08:30:12.298223 140157356639808 tablet_sink_sender.cpp:357] Olap table sink statistics. load_id: 019dfc68-af28-72be-a8cc-fea3d62c7a12, txn_id: 722775, add chunk time(ms)/wait lock time(ms)/num: {10003:(0)(0)(1)} {10002:(0)(0)(1)} {777090:(0)(0)(1)}
I20260506 08:30:15.077191 140155497588288 <http://service.cc:83]|service.cc:83]> Remove shard 10023
4.0.8 RELEASE (build 2090bfc distro ubuntu arch x86_64)
query_id:00000000-0000-0000-0000-000000000000, fragment_instance:00000000-0000-0000-0000-000000000000, plan_node_id:-1
*** Aborted at 1778056217 (unix time) try "date -d @1778056217" if you are using GNU date ***
PC: @          0xe34fa54 starrocks::JsonValue::to_vslice() const
*** SIGSEGV (@0x0) received by PID 30 (TID 0x7f78b6df3640) LWP(429) from PID 0; stack trace: ***
    @     0x7f7973119ee8 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x99ee7)
    @         0x11fc49a8 google::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*)
    @     0x7f79730c2520 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x4251f)
    @          0xe34fa54 starrocks::JsonValue::to_vslice() const
    @          0xe37b786 void starrocks::JsonMerger::_merge_impl<true>(unsigned long)
    @          0xe36df4d starrocks::JsonMerger::merge(std::vector<starrocks::Cow<starrocks::Column>::ImmutPtr<starrocks::Column>, std::allocator<starrocks::Cow<starrocks::Column>::ImmutPtr<starrocks::Column> > > const&)
    @          0xd7bf4d3 starrocks::JsonMergeIterator::next_batch(starrocks::SparseRange<unsigned int> const&, starrocks::Column*)
    @          0xd87c349 starrocks::SegmentIterator::ScanContext::read_columns(starrocks::Chunk*, starrocks::SparseRange<unsigned int> const&)
    @          0xd8a912a starrocks::SegmentIterator::_read(starrocks::Chunk*, std::vector<unsigned int, std::allocator<unsigned int> >*, unsigned long)
    @          0xd88fc4f starrocks::SegmentIterator::_do_get_next(starrocks::Chunk*, std::vector<unsigned int, std::allocator<unsigned int> >*)
    @          0xd89f0c2 starrocks::SegmentIterator::do_get_next(starrocks::Chunk*)
    @          0xdaafd1f starrocks::MaskMergeIterator::fill(unsigned long)
    @          0xdab403e starrocks::MaskMergeIterator::do_get_next(starrocks::Chunk*, std::vector<starrocks::RowSourceMask, std::allocator<starrocks::RowSourceMask> >*)
    @          0xdcc4fde starrocks::lake::TabletReader::do_get_next(starrocks::Chunk*, std::vector<starrocks::RowSourceMask, std::allocator<starrocks::RowSourceMask> >*)
    @          0xdd1de0d starrocks::lake::VerticalCompactionTask::compact_column_group(bool, int, unsigned long, std::vector<unsigned int, std::allocator<unsigned int> > const&, std::unique_ptr<starrocks::lake::TabletWriter, std::default_delete<starrocks::lake::TabletWriter> >&, s0
    @          0xdd1e944 starrocks::lake::VerticalCompactionTask::execute(std::function<starrocks::Status ()>, starrocks::ThreadPool*)
    @          0xdc3f372 starrocks::lake::CompactionScheduler::do_compaction(std::unique_ptr<starrocks::lake::CompactionTaskContext, std::default_delete<starrocks::lake::CompactionTaskContext> >)
    @          0xdc402e6 starrocks::lake::CompactionScheduler::thread_task(int)
    @          0xe44172e starrocks::ThreadPool::dispatch_thread()
    @          0xe4381a5 starrocks::Thread::supervise_thread(void*)
    @     0x7f7973114ac3 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x94ac2)
    @     0x7f79731a68d0 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x1268cf)
[1778056218.369][thread: 140156440884800] je_mallctl execute purge success
[1778056218.369][thread: 140156440884800] je_mallctl execute dontdump success
r
Hi, I'm Rocky! Based on your logs and the fact that you're on StarRocks v4.0.8 (shared-data/Lakehouse), the reason the crashes are continuing despite your configuration changes is that Compaction in shared-data clusters is scheduled by the Frontend (FE). Even if you changed the CN configuration, the FE may have already dispatched these tasks to the CNs, or the tasks are already "stuck" in a retry loop. Because the crash happens in 
JsonMerger
during 
VerticalCompactionTask
, the system is likely trying to merge a specific set of rowsets that contain conflicting JSON structures (which triggers the null pointer in 
to_vslice
). To stop the immediate crash loop and fix the configuration, please follow these steps: 1. Stop the FE from scheduling new compaction tasks This is the most critical step. Disabling the CN-side flag doesn't always prevent the FE from sending a task that is already in the queue. Run this on your leader FE:
Copy code
sql
-- Disable compaction globally for the shared-data cluster
ADMIN SET FRONTEND CONFIG ("lake_compaction_max_tasks" = "0");

-- Verify the setting
ADMIN SHOW FRONTEND CONFIG LIKE 'lake_compaction_max_tasks';
2. Kill existing "Stuck" Compaction Tasks Even with the FE disabled, the CNs might be trying to finish the tasks they already have. Since they are crashing and restarting in K8s, they pick up the same task again from the metadata. 1. Identify the failing tasks:
Copy code
sql
    -- Look for tasks where Error is NOT NULL or those that have been running for a long time
    SHOW PROC '/compactions';

    -- Check specific tablet progress
    SELECT * FROM information_schema.be_cloud_native_compactions WHERE STATUS != 'OK';
2. Cancel the offending compaction:
Copy code
sql
    -- Use the TxnID found in the previous step
    CANCEL COMPACTION WHERE TXN_ID = <TxnID>;
3. Verify CN Config is actually active Sometimes K8s ConfigMap updates don't propagate to the process. Verify that your CN nodes actually see the 
false
values:
Copy code
sql
SELECT * FROM information_schema.be_configs 
WHERE name IN ('enable_compaction_flat_json', 'enable_vertical_compaction');
If they still show 
true
, you must restart the CN pods manually to force them to reload the 
cn.conf
from the ConfigMap. 4. Special Configuration for Lake Tables In StarRocks 4.0, the 
lake
(shared-data) implementation sometimes has its own internal logic for vertical compaction. Try setting this additional parameter on your CN nodes (via 
curl
or 
cn.conf
):
Copy code
properties
-- Set this in cn.conf
lake_vertical_compaction_max_rowsets_per_batch = 0
Setting this to 0 effectively disables the batching logic for vertical compaction in Lakehouse tables. Why is this still happening? The crash 
SIGSEGV (@0x0) ... starrocks::JsonValue::to_vslice()
indicates a Null Pointer Dereference. In v4.0, the "Flat JSON" feature extracts JSON paths into columnar format. If you have a table where: • One rowset has a JSON key as a 
String
. • Another rowset has the same key as an 
Object
or 
Array
. • The 
JsonMerger
fails to resolve this type of conflict during a Vertical Compaction (which merges columns independently), resulting in an unitialized pointer being accessed. Summary of the "Clean State" procedure: 1. 
ADMIN SET FRONTEND CONFIG ("lake_compaction_max_tasks" = "0");
2. Restart all CN pods to clear the local task queue. 3. Run 
ALTER TABLE <table_name> SET ("flat_json.enable" = "false");
(which you've done). 4. Wait for the cluster to stabilize. 5. Re-enable compaction with 
lake_compaction_max_tasks = -1
(adaptive). If you are still seeing this after a full restart with the FE tasks disabled, it is a high-priority bug in the v4.0.8 
JsonMerger
and I recommend reaching out to the StarRocks community/support with the specific 
TxnID
and table schema. References • release_notes/release-4.0.md
y
@Rocky Running 
SHOW PROC '/compactions';
Gave me this:
Copy code
dw_raw.enrichments.819886	722541	2026-05-06 07:44:39		2026-05-06 07:47:17	A error occurred: errorCode=2001 errorMessage:Connection reset	{"sub_task_count":0,"read_local_sec":0,"read_local_mb":0,"read_remote_sec":0,"read_remote_mb":0,"read_segment_count":0,"write_segment_count":0,"write_segment_mb":0,"write_remote_sec":0,"in_queue_sec":0}
dw_raw.enrichments.819886	722558	2026-05-06 07:48:17		2026-05-06 07:50:38	A error occurred: errorCode=2001 errorMessage:Connection reset	{"sub_task_count":0,"read_local_sec":0,"read_local_mb":0,"read_remote_sec":0,"read_remote_mb":0,"read_segment_count":0,"write_segment_count":0,"write_segment_mb":0,"write_remote_sec":0,"in_queue_sec":0}
_statistics_.predicate_columns.10151	722572	2026-05-06 07:50:30	2026-05-06 07:51:28	2026-05-06 07:51:30		{"sub_task_count":8,"read_local_sec":0,"read_local_mb":0,"read_remote_sec":0,"read_remote_mb":0,"read_segment_count":25,"write_segment_count":4,"write_segment_mb":0,"write_remote_sec":0,"in_queue_sec":458}
dw_raw.enrichments.819886	722578	2026-05-06 07:51:38		2026-05-06 07:54:19	fail to call compact, error=Host is down, error_text=[E1014]Got EOF of Socket{id=1225 fd=308 addr=10.181.41.141:8060:56728} (0x0x7fe232519980) [R1][E112]Not connected to 10.181.41.141:8060 yet, server_id=1225 [R2][E112]Not connected to 10.181.41.141:8060 yet, server_id=1225 [R3][E112]Not connected to 10.181.41.141:8060 yet, server_id=1225	{"sub_task_count":3,"read_local_sec":28,"read_local_mb":2944,"read_remote_sec":0,"read_remote_mb":0,"read_segment_count":30,"write_segment_count":3,"write_segment_mb":2973,"write_remote_sec":64,"in_queue_sec":2}
dw_raw.enrichments.819886	722594	2026-05-06 07:55:19		2026-05-06 07:57:43	A error occurred: errorCode=2001 errorMessage:Connection reset	{"sub_task_count":0,"read_local_sec":0,"read_local_mb":0,"read_remote_sec":0,"read_remote_mb":0,"read_segment_count":0,"write_segment_count":0,"write_segment_mb":0,"write_remote_sec":0,"in_queue_sec":0}
dw_raw.enrichments.819886	722612	2026-05-06 07:58:43		2026-05-06 08:01:03	A error occurred: errorCode=2001 errorMessage:Connection reset	{"sub_task_count":0,"read_local_sec":0,"read_local_mb":0,"read_remote_sec":0,"read_remote_mb":0,"read_segment_count":0,"write_segment_count":0,"write_segment_mb":0,"write_remote_sec":0,"in_queue_sec":0}
dw_raw.enrichments.819886	722625	2026-05-06 08:02:03		2026-05-06 08:04:38	A error occurred: errorCode=2001 errorMessage:Connection reset	{"sub_task_count":0,"read_local_sec":0,"read_local_mb":0,"read_remote_sec":0,"read_remote_mb":0,"read_segment_count":0,"write_segment_count":0,"write_segment_mb":0,"write_remote_sec":0,"in_queue_sec":0}
dw_raw.enrichments.819886	722646	2026-05-06 08:05:38		2026-05-06 08:08:19	A error occurred: errorCode=2001 errorMessage:Channel inactive error!	{"sub_task_count":0,"read_local_sec":0,"read_local_mb":0,"read_remote_sec":0,"read_remote_mb":0,"read_segment_count":0,"write_segment_count":0,"write_segment_mb":0,"write_remote_sec":0,"in_queue_sec":0}
dw_raw.enrichments.819886	722665	2026-05-06 08:09:19		2026-05-06 08:11:52	A error occurred: errorCode=2001 errorMessage:Connection reset	{"sub_task_count":0,"read_local_sec":0,"read_local_mb":0,"read_remote_sec":0,"read_remote_mb":0,"read_segment_count":0,"write_segment_count":0,"write_segment_mb":0,"write_remote_sec":0,"in_queue_sec":0}
_statistics_.predicate_columns.10151	722682	2026-05-06 08:12:09	2026-05-06 08:12:10	2026-05-06 08:12:10		{"sub_task_count":8,"read_local_sec":0,"read_local_mb":0,"read_remote_sec":0,"read_remote_mb":0,"read_segment_count":22,"write_segment_count":4,"write_segment_mb":0,"write_remote_sec":0,"in_queue_sec":8}
dw_raw.enrichments.819886	722686	2026-05-06 08:12:52		2026-05-06 08:16:25	A error occurred: errorCode=2001 errorMessage:Connection reset	{"sub_task_count":0,"read_local_sec":0,"read_local_mb":0,"read_remote_sec":0,"read_remote_mb":0,"read_segment_count":0,"write_segment_count":0,"write_segment_mb":0,"write_remote_sec":0,"in_queue_sec":0}
_statistics_.predicate_columns.10151	722709	2026-05-06 08:16:42	2026-05-06 08:16:43	2026-05-06 08:16:44		{"sub_task_count":8,"read_local_sec":0,"read_local_mb":0,"read_remote_sec":0,"read_remote_mb":0,"read_segment_count":24,"write_segment_count":4,"write_segment_mb":0,"write_remote_sec":0,"in_queue_sec":0}
dw_raw.enrichments.819886	722712	2026-05-06 08:17:25		2026-05-06 08:20:05	A error occurred: errorCode=2001 errorMessage:Connection reset	{"sub_task_count":0,"read_local_sec":0,"read_local_mb":0,"read_remote_sec":0,"read_remote_mb":0,"read_segment_count":0,"write_segment_count":0,"write_segment_mb":0,"write_remote_sec":0,"in_queue_sec":0}
dw_raw.enrichments.819886	722728	2026-05-06 08:21:05		2026-05-06 08:23:23	A error occurred: errorCode=2001 errorMessage:Channel inactive error!	{"sub_task_count":0,"read_local_sec":0,"read_local_mb":0,"read_remote_sec":0,"read_remote_mb":0,"read_segment_count":0,"write_segment_count":0,"write_segment_mb":0,"write_remote_sec":0,"in_queue_sec":0}
dw_raw.enrichments.819886	722744	2026-05-06 08:24:23		2026-05-06 08:26:54	A error occurred: errorCode=2001 errorMessage:Connection reset	{"sub_task_count":0,"read_local_sec":0,"read_local_mb":0,"read_remote_sec":0,"read_remote_mb":0,"read_segment_count":0,"write_segment_count":0,"write_segment_mb":0,"write_remote_sec":0,"in_queue_sec":0}
dw_raw.enrichments.819886	722761	2026-05-06 08:27:54		2026-05-06 08:29:02	A error occurred: errorCode=2001 errorMessage:Channel inactive error!	{"sub_task_count":0,"read_local_sec":0,"read_local_mb":0,"read_remote_sec":0,"read_remote_mb":0,"read_segment_count":0,"write_segment_count":0,"write_segment_mb":0,"write_remote_sec":0,"in_queue_sec":0}
_statistics_.predicate_columns.10151	722772	2026-05-06 08:29:46	2026-05-06 08:29:47	2026-05-06 08:29:48		{"sub_task_count":8,"read_local_sec":0,"read_local_mb":0,"read_remote_sec":0,"read_remote_mb":0,"read_segment_count":17,"write_segment_count":3,"write_segment_mb":0,"write_remote_sec":0,"in_queue_sec":2}
dw_raw.enrichments.819886	722773	2026-05-06 08:30:02		2026-05-06 08:32:36	A error occurred: errorCode=2001 errorMessage:Channel inactive error!	{"sub_task_count":0,"read_local_sec":0,"read_local_mb":0,"read_remote_sec":0,"read_remote_mb":0,"read_segment_count":0,"write_segment_count":0,"write_segment_mb":0,"write_remote_sec":0,"in_queue_sec":0}
_statistics_.loads_history.824846	722784	2026-05-06 08:31:59	2026-05-06 08:32:00	2026-05-06 08:32:00		{"sub_task_count":3,"read_local_sec":0,"read_local_mb":0,"read_remote_sec":0,"read_remote_mb":0,"read_segment_count":30,"write_segment_count":3,"write_segment_mb":0,"write_remote_sec":0,"in_queue_sec":0}
dw_raw.enrichments.819886	722792	2026-05-06 08:33:36		2026-05-06 08:36:13	A error occurred: errorCode=2001 errorMessage:Connection reset	{"sub_task_count":0,"read_local_sec":0,"read_local_mb":0,"read_remote_sec":0,"read_remote_mb":0,"read_segment_count":0,"write_segment_count":0,"write_segment_mb":0,"write_remote_sec":0,"in_queue_sec":0}
Running '`CANCEL COMPACTION WHERE TXN_ID = <TXN_ID>;` did not cancel any of the compaction jobs.
r
Hi, I'm Rocky! The reason your CN nodes are still crashing is that the StarRocks Frontend (FE) is continuously rescheduling the compaction task for tablet 819886 (part of table 
dw_raw.enrichments
) as soon as the previous attempt fails (due to the crash). Because the crash happens in the 
JsonMerger
logic, even disabling "Vertical Compaction" or "Flat JSON" might not be enough if the standard compaction path still needs to merge those specific (likely corrupt or type-conflicting) JSON rowsets. To stop the crash loop immediately, you need to prevent the FE from scheduling compaction for this specific table. 1. Identify your Table ID Compaction control in shared-data mode often uses IDs. Find the ID for your table:
Copy code
sql
SELECT TABLE_ID FROM information_schema.tables 
WHERE table_schema = 'dw_raw' AND table_name = 'enrichments';
2. Disable Compaction for that Table (FE Level) Run this on your Leader FE. This will prevent the FE from generating any new compaction tasks for that table ID, effectively isolating the problem:
Copy code
sql
-- Replace <TABLE_ID> with the ID found in step 1
ADMIN SET FRONTEND CONFIG ("lake_compaction_disable_ids" = "<TABLE_ID>");
Note: If you have multiple tables causing issues, separate IDs with a semicolon (e.g., 
"12345;67890"
). 3. Clear Stuck Tasks and Stabilization Even after disabling scheduling, the CN nodes might have the crashing tasks in their local queues. Since you are in K8s, the easiest way to "flush" the system is: 1. Set global compaction tasks to 0 (temporarily):
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("lake_compaction_max_tasks" = "0");
2. Restart your CN pods. This will wipe the in-memory task queues on the compute nodes. 3. Wait a few minutes for the cluster to stabilize and ensure no more CNs are crashing. 4. How to resume safely Once the cluster is stable: 1. Enable global compaction again:
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("lake_compaction_max_tasks" = "-1");
_Since you added the table ID to 
lake_compaction_disable_ids
, it will be skipped, but other tables will continue compacting._ 2. Long-term fix: This is a confirmed bug in the JSON merging logic (likely related to type mismatches like a path being an Object in one file and a String in another). I strongly recommend upgrading to StarRocks v4.0.10+ or v4.1, where several 
JsonMerger
and 
to_vslice
null-pointer crashes have been fixed. Why 
CANCEL COMPACTION
didn't work for you: In your 
SHOW PROC
output, the tasks already show errors like 
Connection reset
or 
Channel inactive
. The 
CANCEL COMPACTION
command only works for tasks that are currently "In Progress" (where 
CommitTime
is NULL). Since your CNs are crashing, the tasks are failing immediately, and the FE is simply starting a brand new transaction (new 
TxnID
) every few minutes, which is why you see a long history of failed attempts. References • administration/management/compaction.md • faq/shared_data_faq.md • administration/management/FE_configuration.md References • administration/management/compaction.md • deployment/shared_data/feature-support-shared-data.md • faq/shared_data_faq.md • knowledge/admin_and_operation/shared_data_compaction.md
Open in Slack
PreviousNext
Powered by