http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> why it crashes? ```2026-06-16 08:56...
# questions-and-troubleshooting
a
@Rocky why it crashes?
Copy code
2026-06-16 08:56:41	
    @     0xffff98ce9e9c (/usr/lib/aarch64-linux-gnu/libc.so.6+0xe9e9b)

2026-06-16 08:56:41	
    @     0xffff98c80398 (/usr/lib/aarch64-linux-gnu/libc.so.6+0x80397)
2026-06-16 08:56:41	
    @          0xd000bd8 starrocks::Thread::supervise_thread(void*)
2026-06-16 08:56:41	
    @          0xd009f34 starrocks::ThreadPool::dispatch_thread()
2026-06-16 08:56:41	
    @          0xc03faac starrocks::MemtableFlushTask::run()
2026-06-16 08:56:41	
    @          0xc03e2b4 starrocks::FlushToken::_flush_memtable(starrocks::MemTable*, starrocks::SegmentPB*, bool, long*)
2026-06-16 08:56:41	
    @          0xc9642f4 starrocks::MemTable::flush(starrocks::SegmentPB*, bool, long*)
2026-06-16 08:56:41	
    @          0xcc428c0 starrocks::lake::SpillMemTableSink::flush_chunk(starrocks::Chunk const&, starrocks::SegmentPB*, bool, long*)
2026-06-16 08:56:41	
    @          0xc9fbb80 starrocks::lake::HorizontalGeneralTabletWriter::flush(starrocks::SegmentPB*)
2026-06-16 08:56:41	
    @          0xca1c678 starrocks::lake::HorizontalPkTabletWriter::flush_segment_writer(starrocks::SegmentPB*)
2026-06-16 08:56:41	
    @          0xc628ca8 starrocks::SegmentWriter::finalize(unsigned long*, unsigned long*, unsigned long*)
2026-06-16 08:56:41	
    @          0xc6285f8 starrocks::SegmentWriter::finalize_columns(unsigned long*)
2026-06-16 08:56:41	
    @          0xc65fdcc starrocks::FlatJsonColumnWriter::finish()
2026-06-16 08:56:41	
    @          0xc630e38 starrocks::StringColumnWriter::finish()
2026-06-16 08:56:41	
    @          0xc62e7e4 starrocks::ScalarColumnWriter::finish()
2026-06-16 08:56:41	
    @     0xffff9a338838 ([vdso]+0x837)
2026-06-16 08:56:41	
    @     0xffff99cd241c JVM_handle_linux_signal
2026-06-16 08:56:41	
    @     0xffff99cd1a64 PosixSignals::chained_handler(int, siginfo_t*, void*) [clone .part.0]
2026-06-16 08:56:41	
    @         0x10336738 google::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*)
2026-06-16 08:56:41	
    @     0xffff98c853dc (/usr/lib/aarch64-linux-gnu/libc.so.6+0x853db)
2026-06-16 08:56:41	
*** SIGSEGV (@0x44) received by PID 33 (TID 0xffea0b36fe00) LWP(141103) from PID 68; stack trace: ***
2026-06-16 08:56:41	
PC: @          0xc62e7e4 starrocks::ScalarColumnWriter::finish()
2026-06-16 08:56:41	
*** Aborted at 1781625401 (unix time) try "date -d @1781625401" if you are using GNU date ***
r
The stack trace indicates a Segmentation Fault (SIGSEGV) occurring on an ARM64 (
aarch64
) system during a Memtable Flush operation. Specifically, the crash happens when StarRocks is finishing the write process for a Flat JSON column. Why it is crashing The crash is triggered by the Flat JSON feature (introduced in v3.3), which automatically extracts sub-fields from JSON data into separate columns for better performance. 1. The Trigger: The crash occurs at 
starrocks::ScalarColumnWriter::finish()
. This is called by 
FlatJsonColumnWriter
when it attempts to finalize the storage for a sub-column it extracted from your JSON data. 2. Memory Access Error: The 
SIGSEGV (@0x44)
suggests a null-pointer dereference (likely an offset within an object that was not properly initialized or has been invalidated). 3. Architecture Factor: Your logs show 
aarch64-linux-gnu
, confirming you are on ARM64. There have been several known issues specifically involving memory alignment and signal handling on ARM64 for recent StarRocks versions (v3.3.x - v4.0.x) when processing complex types or JSON. Suggested Solutions 1. Disable Flat JSON (Immediate Workaround) If you need to stop the crashes immediately, you can disable the Flat JSON feature. This will stop the BE from trying to extract sub-columns and will store JSON as a single blob again. * For existing tables: Change the table property:
Copy code
sql
    ALTER TABLE <table_name> SET ("flat_json.enable" = "false");
* System-wide (for new tables): Starting from v4.0, this is enabled by default. You can try disabling it at the BE configuration level if the crash persists across multiple tables: * Set 
enable_lazy_dynamic_flat_json = false
in your 
be.conf
and restart. 2. Verify and Upgrade This crash pattern is often related to bugs in the dynamic schema detection logic of Flat JSON. * If you are on an early version of 3.3 (prior to 3.3.3) or 4.0, upgrade to the latest patch release (e.g., v3.3.5+ or v4.0.1+). Many SIGSEGV issues on ARM64 related to JSON and column writers were fixed in recent hotfixes. 3. Check for Nulls or Schema Changes The crash occurs during 
finish()
, which suggests that the data being flushed might contain a specific structure or a sudden change in JSON schema that the writer wasn't prepared for. * Check if the data you were loading around 
08:56:41
contains extremely large JSON objects, deeply nested structures, or fields that change types frequently (e.g., a field being an 
int
in one row and a 
string
or 
object
in another). If the crash continues after disabling Flat JSON, it may be a lower-level memory issue on ARM64, and you should consider sharing the full BE log (
<http://be.INFO|be.INFO>
) with the StarRocks community for a few lines before the crash occurred. References • using_starrocks/Flat_json.md
a
@Rocky I have no json columns in underlying Iceberg table that I use for the MV
r
Even if your underlying Iceberg table does not explicitly contain a 
JSON
type column, the presence of 
starrocks::FlatJsonColumnWriter
in the crash trace means StarRocks is internally using the Flat JSON engine to write data for your Materialized View (MV). This happens for a few common reasons: 1. MAP or STRUCT types in Iceberg In StarRocks v3.3 and v4.0, the "Flat JSON" logic is often applied to other semi-structured types like 
MAP
 or 
STRUCT
. If your Iceberg table has these types, StarRocks may be attempting to "flatten" them into sub-columns during the MV refresh to optimize performance. * The crash at 
ScalarColumnWriter::finish()
indicates the error happens when finalizing one of these automatically extracted sub-fields. 2. Known ARM64 (aarch64) Bug Your logs show you are running on ARM64 (
aarch64
). There is a known issue (e.g., GitHub Issue #64667) where the writer crashes on ARM during the 
finish()
or 
finalize()
phase. This is often due to: * Empty paths: An internal bug where an empty path string causes a crash when calling 
substr
(Fixed in v3.4.9/v3.3.5+). * Memory alignment: Specific instruction optimizations for ARM that fail when finalizing null bitmaps in complex types. 3. Asynchronous MV Refresh and "Lake" Tables The trace mentions 
starrocks::lake::HorizontalPkTabletWriter
. This indicates the MV is likely stored as a Cloud-Native (Lake) Primary Key table. In v4.0+, "Flat JSON" is enabled by default for these tables. Even if you didn't define a JSON column, the system may have mistakenly initialized this writer or is processing a column it thinks could benefit from flattening. ────────── How to fix this Step 1: Disable Flat JSON on the Materialized View Since the crash happens during the writing of the MV's internal data, you should disable the flattening feature for that specific MV:
Copy code
sql
-- Replace <mv_name> with your Materialized View name
ALTER TABLE <mv_name> SET ("flat_json.enable" = "false");
Step 2: Global Workaround (if the above fails) If you cannot run the 
ALTER
because the BE is unstable, you can disable the feature globally on all Backends (BEs) by adding this to your 
be.conf
and restarting:
Copy code
properties
enable_lazy_dynamic_flat_json = false
Step 3: Upgrade This specific crash (especially on ARM64) has seen multiple fixes in recent patch releases. If you are on an early version of 3.3 or 4.0, upgrade to: * v3.3.5 or higher * v4.0.1 or higher Recommendation: Check if your MV or Iceberg source uses 
MAP
or 
STRUCT
types. If it does, disabling 
flat_json
as shown in Step 1 is the most reliable way to stop the crash while maintaining your current crashes while remaining on your current version. References • using_starrocks/Flat_json.md • release_notes/release-3.4.md • release_notes/release-3.3.md
2 Views
Open in Slack
PreviousNext
Powered by