@Rocky I am on starrocks 4.0.8 running on k8s shared mode with 4 CN nodes and 2 FE nodes. During pipe file ingest I get this error: │ I20260510 131639.398113 140565980943936 pipeline_driver_executor.cpp:432] [Driver] Fail to report audit statistics due to query not found: fragment_instance_id=019e1207-75b8-7982-917f-adfaeb6afe68 │ │ PC: @ 0xd7e1c8d starrocks:ScalarColumnWriter:finish() │ │ * SIGSEGV (@0x44) received by PID 30 (TID 0x7fd45dfff640) LWP(1234) from PID 68; stack trace: * │ │ @ 0x7fd8967f8ee8 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x99ee7) │ │ @ 0x11fc49a8 google::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) │ │ @ 0x7fd8967a1520 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x4251f) │ │ @ 0xd7e1c8d starrocks:ScalarColumnWriter:finish() │ │ @ 0xd7e4c3c starrocks:StringColumnWriter:finish() │ │ @ 0xd81b6c7 starrocks:FlatJsonColumnWriter:finish() │ │ @ 0xd7da4c3 starrocks:SegmentWriter:finalize_columns(unsigned long*) │ │ @ 0xd7dac3f starrocks:SegmentWriter:finalize(unsigned long*, unsigned long*, unsigned long*) │ │ @ 0xdcabf15 starrocks:lakeHorizontalPkTabletWriter:flush_segment_writer(starrocks::SegmentPB*) │ │ @ 0xdc823e3 starrocks:lakeHorizontalGeneralTabletWriter:flush(starrocks::SegmentPB*) │ │ @ 0xdf7606c starrocks:lakeSpillMemTableSink:flush_chunk(starrocks::Chunk const&, starrocks::SegmentPB*, bool, long*) │ │ @ 0xdbbfd37 starrocks:MemTable:flush(starrocks::SegmentPB*, bool, long*) │ │ @ 0xd110e9a starrocks:FlushToken:_flush_memtable(starrocks::MemTable*, starrocks::SegmentPB*, bool, long*) │ │ @ 0xd112b2c starrocks:MemtableFlushTask:run() │ │ @ 0xe44172e starrocks:ThreadPool:dispatch_thread() │ │ @ 0xe4381a5 starrocks:🧵:supervise_thread(void*) │ │ @ 0x7fd8967f3ac3 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x94ac2) │ │ @ 0x7fd8968858d0 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x1268cf) │ │ [1778419000.21][thread: 140550086850112] je_mallctl execute purge success │ │ [1778419000.21][thread: 140550086850112] je_mallctl execute dontdump success What is happening

The error you are encountering is a Segmentation Fault (SIGSEGV) occurring in the StarRocks Storage Engine during data ingestion. Specifically, the crash happens when the system is finalizing (calling

finish()

) the segment files for a table using the Flat JSON feature. What is Happening? Based on the stack trace, here is the breakdown of the failure: 1. Flat JSON V2 (Default in 4.0): Starting with StarRocks 4.0, "Flat JSON" is enabled by default. This feature automatically extracts common fields from JSON data and stores them in separate columnar "sub-columns" to speed up queries. 2. Shared-Data PK Table Sink: You are using a Primary Key table in shared-data (Cloud Native) mode. During the pipe ingest, the

HorizontalPkTabletWriter

is flushing data from memory to segments in object storage. 3. Null Pointer Dereference (@0x44): The crash occurs at address

0x44

inside

starrocks::ScalarColumnWriter::finish()

. This address is a typical offset for a member variable. It indicates that the

FlatJsonColumnWriter

is attempting to finalize a sub-column writer that was either not initialized or was null-pointered (potentially due to a schema mismatch, empty JSON objects, or a race condition during the segment flush). 4. Audit Statistics Error: The log message

Fail to report audit statistics due to query not found

is a side effect. It simply means the Backend (CN) process crashed while the Frontend (FE) was still waiting for a status report. This is a known stability issue in the StarRocks 4.0.x release cycle related to the new Flat JSON V2 implementation, especially when combined with Primary Key tables and schema evolution. Immediate Workarounds To stop the crash and allow your pipe ingest to continue, you should disable the Flat JSON feature. You can do this at the global level or for the specific table. 1. Disable Flat JSON globally (Recommended for stability): Execute this via your SQL client:

sql
SET GLOBAL flat_json_enable = false;

Note: This will not affect existing data, but will stop the system from trying to "flatten" new incoming JSON data. 2. Disable Flat JSON for the specific table:

sql
ALTER TABLE your_table_name SET ("flat_json.enable" = "false");

3. Check for "Empty" JSON fields: If you can control the source data, check if your pipe is ingesting rows with empty strings

""

or highly inconsistent types in a JSON column, as these have been reported to trigger edge cases in the Flat JSON writer in version 4.0.8. Recommended Next Steps * Check for Schema Changes: If you recently added or dropped columns on this table, this crash might be related to a known bug in how Flat JSON handles

unique_id

mapping after schema changes (Issue #68279). * Upgrade/Hotfix: Version 4.0.8 is part of the initial 4.0 stable branch. Check if a later patch (like 4.0.10 or higher) is available for your environment, as several Flat JSON stability fixes were merged in late 4.0 patches. * Monitor CN Nodes: Since you are on K8s, ensure your CN nodes have enough memory. The trace shows

je_mallctl execute purge

, which suggests the system was under memory pressure or performing aggressive garbage collection just before the crash. References • loading/Json_loading.md • loading/StreamLoad.md • using_starrocks/Flat_json.md

@Rocky This is my table create:

CREATE TABLE IF NOT EXISTS enrichments (
    id STRING NOT NULL,
    taken_timestamp BIGINT NOT NULL,
    scrape_timestamp BIGINT NOT NULL,
    _index STRING NOT NULL,
    score DOUBLE,
    source JSON
)
PRIMARY KEY (id, taken_timestamp, scrape_timestamp)
PARTITION BY date_trunc('month', FROM_UNIXTIME(scrape_timestamp))
DISTRIBUTED BY HASH(id)
PROPERTIES (
    "enable_persistent_index" = "true",
    "persistent_index_type" = "CLOUD_NATIVE",
    "flat_json.enable" = "false"
);

And I still get this error. Also, When running

show create table enrichments

I do not see

"flat_json.enable" = "false"

in the properties. What am I missing ?

The reason you are still seeing this crash is that the Flat JSON feature is still active during your ingestion, despite your attempt to disable it in the

CREATE TABLE

statement. In StarRocks 4.0.8, there is a known issue where table properties related to Flat JSON (like

flat_json.enable

) may not be correctly persisted or recognized for Primary Key tables in Shared-Data (Lake) mode during the initial table creation. This explains why you don't see the property in

SHOW CREATE TABLE

and why the stack trace still shows

FlatJsonColumnWriter

(the component responsible for the crash). Why it is crashing The crash

ScalarColumnWriter::finish() @ 0x44

is a null pointer dereference. The

FlatJsonColumnWriter

is attempting to finalize a sub-column that was never properly initialized, likely due to a schema mismatch or an unexpected data pattern (like a JSON field that is always

null

or empty in one batch but has data in another) which the 4.0.8 writer logic doesn't handle gracefully in Primary Key tables. How to fix it Since the table property didn't "stick," you need to disable the feature using one of the following methods: 1. Force the property via ALTER TABLE Even if the

CREATE TABLE

failed to save it,

ALTER TABLE

often succeeds in updating the metadata for existing tables:

sql
ALTER TABLE enrichments SET ("flat_json.enable" = "false");

After running this, run

SHOW CREATE TABLE enrichments

again. If you still don't see the property, it means your version (4.0.8) has a bug in the metadata handler for this property on Lake PK tables. 2. Disable at the BE/CN Configuration (The most reliable fix) In StarRocks 4.0, Flat JSON is enabled by default at the Backend (CN) level. You can disable it globally for all ingestion on those nodes. Since you are on K8s, you can do this dynamically without a restart: Run this for each of your 4 CN nodes (replace

<CN_IP>

and

<CN_HTTP_PORT>

):

bash
curl -X POST http://<CN_IP>:<CN_HTTP_PORT>/api/update_config?enable_json_flat=false

Note: The default HTTP port for CN/BE is

8040

. To make this permanent on K8s, add it to your

cn.conf

in the ConfigMap:

conf
enable_json_flat = false

3. Set the Session Variable While loading via Pipe or Stream Load, you can also try setting the session variable, though BE configuration is more certain for ingestion crashes:

sql
SET GLOBAL flat_json_enable = false;

Summary of Recommendations 1. Check the CN logs: If you see

FlatJsonColumnWriter

in the trace again after applying the settings, the feature is still on. 2. Verify Data: If you must use Flat JSON, the crash is often triggered by JSON columns containing empty objects

{}

or arrays

[]

that change type frequently across rows. 3. Upgrade: This specific stability issue in the Flat JSON writer for Primary Key tables was addressed in patches later than 4.0.8. If possible, upgrade to 4.0.10+. Verification: After applying the

ALTER

or BE config, the crash should disappear because the system will treat the

source

column as a single binary JSON blob instead of trying to split it into sub-columns. References • release_notes/release-3.4.md • release_notes/release-2.2.md • knowledge/trouble_shooting/fe_deadlock_problems.md

@Rocky When running

SET flat_json_enable = false;

I get this answer:

SQL Error [1193] [HY000]: Getting analyzing error. Detail message: Unknown system variable 'flat_json_enable', the most similar variables are {'lower_case_table_names', 'skew_join_rand_range', 'default_storage_engine'}.

I apologize for the confusion in the previous response. The error

Unknown system variable 'flat_json_enable'

occurs because Flat JSON is not controlled via a session variable in StarRocks 4.0; it is controlled either by BE/CN configurations or Table Properties. The crash you are seeing in

ScalarColumnWriter::finish()

confirms that the Flat JSON extraction logic is still running and failing during the ingestion process. Why your

CREATE TABLE

property didn't work In StarRocks 4.0.8, there is a known issue where table-level properties like

"flat_json.enable"

may not be correctly persisted or visible in

SHOW CREATE TABLE

for Primary Key tables in Shared-Data (Lake) mode if they are set during the initial creation. How to correctly disable it To stop the crash and ensure the setting is applied, please try these two methods: 1. Use ALTER TABLE (Recommended) Even if it wasn't captured during

CREATE

, an

ALTER

command often forces the metadata update. Run this command and verify with `SHOW CREATE TABLE`:

sql
ALTER TABLE enrichments SET ("flat_json.enable" = "false");

Wait 10-20 seconds for the metadata to propagate to the CN nodes before resuming your pipe ingest. 2. Disable globally via CN Configuration (Most Reliable) Since you are on K8s and encountering a crash, the most robust way to stop the extraction logic is to disable it at the engine level. The parameter name is

enable_json_flat

. A. Temporary Fix (No restart): Run this

curl

command against each of your 4 CN nodes (replace

<CN_IP>

with the internal K8s IP or service endpoint, port is usually

8040

):

bash
curl -X POST http://<CN_IP>:8040/api/update_config?enable_json_flat=false

B. Permanent Fix (K8s ConfigMap): Add the following to your CN configuration (

cn.conf

):

conf
enable_json_flat = false

Why it is crashing in 4.0.8 The stack trace shows the crash happens when the system tries to "finalize" a sub-column. In StarRocks 4.0.x, the new Flat JSON V2 engine tries to automatically extract fields from your

source

JSON column. • The error

ScalarColumnWriter::finish() @ 0x44

usually indicates a null pointer dereference. • This is often triggered in Primary Key tables when the JSON data contains empty objects

{}

, arrays

[]

, or inconsistent types across rows that the 4.0.8 writer cannot map to a stable internal schema. • Fixed in later patches: Several stability fixes for this exact logic were merged in 4.0.9 and 4.0.10. If possible, I recommend upgrading your CN/FE nodes to a more recent 4.0.x patch. Summary 1. *Don't use `SET flat_json_enable`*; it doesn't exist. 2. Use

ALTER TABLE enrichments SET ("flat_json.enable" = "false");

. 3. If the crash persists, set the BE/CN config

enable_json_flat = false

via the HTTP API or config file. References • using_starrocks/Flat_json.md • administration/management/BE_configuration.md