< Rocky> I run cluster 4 1 4 and see the following error Com StarRocks #questions-and-troubleshooting

<@U0A71G31CDV> I run cluster 4.1.4. and see the fo...

Andrei Harbunou

09/15/2026, 10:41 PM

@Rocky I run cluster 4.1.4. and see the following error "Compaction job txnId=1726, partition=statistics.task_run_history.10488, warehouse=default_warehouse, cnGroup=0, scoreBefore={"avg":4.125,"p50":0.0,"max":28.0} failed: string length(1153813)" what is the issue?

Rocky

09/15/2026, 10:41 PM

The error

failed: string length(1153813)

occurs because a row in the

_statistics_.task_run_history

table contains a string that exceeds the maximum allowed length for the

VARCHAR

data type in StarRocks. Cause * Max Limit: In StarRocks (v2.1 and later), the maximum length for a

VARCHAR

column is 1,048,576 bytes (exactly 1 MB). * The Violation: Your error message indicates the compaction process encountered a string with a length of 1,153,813 bytes (~1.1 MB). * The Table: The

_statistics_.task_run_history

table stores metadata and error logs for asynchronous tasks (such as Materialized View refreshes or Routine Loads). It is likely that a specific task failed and generated an extremely large error message, query plan, or metadata blob that was ingested into the table but now prevents compaction from processing that data fragment. How to Resolve You need to identify and remove the problematic record(s) from the system table to allow compaction to resume. 1. Identify the problematic rows: Run a query to find columns with strings exceeding the 1MB limit. Usually, the

error_message

extra_info

columns are the culprits:

Copy code

sql
    SELECT query_id, task_name, LENGTH(error_message)
    FROM _statistics_.task_run_history
    WHERE LENGTH(error_message) > 1048576;

_(Note: Column names may vary slightly depending on your specific version; you can check them via

DESC _statistics_.task_run_history

.)_ 2. Delete the records: Once you have identified the

query_id

task_name

of the offending rows, delete them:

Copy code

sql
    DELETE FROM _statistics_.task_run_history WHERE query_id = 'xxxx';
    -- Or truncate the history if it's not critical:
    TRUNCATE TABLE _statistics_.task_run_history;

3. Prevention (Optional): If this was caused by a Materialized View task, you can limit the metadata stored in the FE to prevent future occurrences by adjusting the FE configuration:

Copy code

sql
    ADMIN SET FRONTEND CONFIG ("max_mv_task_run_meta_message_values_length" = "10000");

(The default is usually small, but very complex plans or massive partition lists can sometimes bloat these messages.) After the offending data is removed, the background compaction job for the

_statistics_

database should succeed on its next run. References • knowledge/usage/best-practice.md • sql-reference/data-types/string-type/VARCHAR.md • faq/Sql_faq.md • faq/table_design_faq.md • sql-reference/data-types/string-type/STRING.md • sql-reference/data-types/string-type/BINARY.md • sql-reference/System_limit.md

Andrei Harbunou

09/15/2026, 10:44 PM

@Rocky task run table structure is

Copy code

task_id
task_run_id
create_time
task_name
task_state
finish_time
expire_time
history_content_json

Rocky

09/15/2026, 10:44 PM

Based on the table structure you provided, the error occurs because the history_content_json
column contains a string that exceeds the 1,048,576 byte (1 MB) limit for the

VARCHAR

data type during the compaction process. Even if the data was successfully written to the table initially, the compaction background process (which merges data files) enforces these limits strictly and will fail if a single field exceeds them. Solution: Identify and Remove the Large Records You should identify the specific rows causing the issue and delete them from the internal statistics table. 1. Find the problematic tasks: Run this query to find which task runs have data exceeding the limit:

Copy code

sql
    SELECT task_id, task_run_id, LENGTH(history_content_json)
    FROM _statistics_.task_run_history
    WHERE LENGTH(history_content_json) > 1048576;

2. Delete the offending rows: Once you have the

task_run_id

(or

task_id

), delete those specific records to allow compaction to proceed:

Copy code

sql
    DELETE FROM _statistics_.task_run_history WHERE task_run_id = '<problem_id>';

_If the history is not critical, you can also safely

TRUNCATE TABLE _statistics_.task_run_history;

to clear all records._ Why did this happen? This most commonly occurs with Materialized View (MV) refresh tasks. If an MV has a very large number of partitions (thousands) or a highly complex query plan, the metadata stored in

history_content_json

(which maps to

EXTRA_MESSAGE

in the

task_runs

view) can grow beyond 1 MB. Prevention To prevent this from recurring, you can limit the amount of metadata StarRocks attempts to store for MV tasks by adjusting the FE configuration:

Copy code

sql
-- Limits the length of partition-related messages in the task history
ADMIN SET FRONTEND CONFIG ("max_mv_task_run_meta_message_values_length" = "100");

Note on Version: You mentioned version *4.1.4*; please verify this version number. As of now, StarRocks official releases are in the 3.x branch (e.g., 3.3.x). If you are using a vendor-specific distribution, the internal table names or limits might differ slightly, but the logic above remains the standard fix for compaction string-length errors. References • administration/management/FE_configuration.md • using_starrocks/async_mv/materialized_view_task_run_details.md • sql-reference/information_schema/task_runs.md • administration/http_interface/query_detail.md

Andrei Harbunou

09/15/2026, 10:46 PM

@Rocky current value for

max_mv_task_run_meta_message_values_length

is 8

Rocky

09/15/2026, 10:47 PM

The issue is that a record in the

_statistics_.task_run_history

table contains a string (specifically in the

history_content_json

column) that exceeds StarRocks' maximum internal limit for a single field, which is 1,048,576 bytes (1 MB). Even though you have set

max_mv_task_run_meta_message_values_length

to 8, the compaction process is still failing because: 1. Existing Data: Records created before you changed that setting to 8 may still exist in your data files. Compaction is a background process that merges old files; it will continue to fail as long as those large historical records remain. 2. Other Bloated Fields: The

history_content_json

column stores more than just partition lists. It can contain large

planBuilderMessage

diagnostic logs, query plans, or massive

error_message

strings if a task failed in a complex way. Even with only 8 items in a list, if those items themselves are very large, the total JSON string can exceed 1 MB. How to Fix You must manually remove the record that is blocking the compaction process. 1. Find the offending record: Run the following query to identify the specific task run causing the issue:

Copy code

sql
    SELECT task_id, task_run_id, LENGTH(history_content_json)
    FROM _statistics_.task_run_history
    WHERE LENGTH(history_content_json) > 1000000;

2. Delete the record: Delete the specific row(s) found. This will allow the compaction job to successfully merge the remaining data:

Copy code

sql
    DELETE FROM _statistics_.task_run_history WHERE task_run_id = '<the_id_from_above>';

_Note: If the history is not critical for your auditing, you can simply run

TRUNCATE TABLE _statistics_.task_run_history;

to clear all records and resolve the issue immediately._ 3. FE Configuration (Optional): In version 4.1.x (and 3.3.3+), you can also set the following to prevent the Frontend from crashing or failing to load if it encounters these large/corrupted records during metadata replay:

Copy code

sql
    ADMIN SET FRONTEND CONFIG ("ignore_task_run_history_replay_error" = "true");

Why does compaction fail but ingestion succeeded? Sometimes, initial writes (flushing from memory to disk) do not strictly enforce the 1MB limit on every field, or the data was ingested through a path that bypassed the check. However, Compaction performs a full read-merge-write cycle where StarRocks strictly enforces the

VARCHAR

limit. If a field is found to be larger than 1,048,576 bytes, the compaction transaction (

txnId=1726

in your case) will fail. References • using_starrocks/async_mv/materialized_view_task_run_details.md

Andrei Harbunou

09/15/2026, 10:48 PM

@Rocky what if I reduce ``max_mv_task_run_meta_message_values_length` for 1? will it help?

Rocky

09/15/2026, 10:49 PM

Reducing

max_mv_task_run_meta_message_values_length

to 1 will not help resolve the current compaction error. Here is why: 1. Compaction processes existing data The error

failed: string length(1153813)

is occurring because a record that is already stored on disk exceeds the 1MB limit. Compaction is a background process that reads these old data files and tries to merge them into new ones. * Changing the configuration only affects newly created task records. * The compaction process will continue to fail every time it encounters that specific 1.1MB record in your historical data until that record is removed. 2. You have a "Bad" record already Even with your current setting of

, you have a record that is ~1.1MB. This means either: * The 8 items being stored are individually very large (e.g., extremely long partition names or complex JSON paths). * The record was created when the setting was higher than 8. * The bloat is coming from a different field in the JSON (like a massive error stack trace or query plan) that isn't strictly controlled by that specific "length" setting. How to actually fix it To unblock the compaction job (and stop the error logs), you must remove the data that violates the limit: * Identify the row:

Copy code

sql
    SELECT task_run_id, LENGTH(history_content_json)
    FROM _statistics_.task_run_history
    WHERE LENGTH(history_content_json) > 1048576;

* Delete the row:

Copy code

sql
    DELETE FROM _statistics_.task_run_history WHERE task_run_id = <the_id_found_above>;

* Alternative (Fastest): If you don't need the task history, simply truncate the table:

Copy code

sql
    TRUNCATE TABLE _statistics_.task_run_history;

Summary of the Config * max_mv_task_run_meta_message_values_length
: Useful for prevention of future bloat. Setting it to 1 is likely overkill and will make your task history less useful (you'll only see 1 partition refreshed even if 100 were). * enable_check_string_lengths
: This is a BE setting (default

true

) that prevents "out-of-bounds" strings from being written during compaction. It is doing its job by stopping a corrupted or oversized record from causing further issues down the line. Recommendation: Delete the offending rows from

_statistics_.task_run_history

and keep your current setting of

. If the error happens again with new records, then consider reducing the config or investigating why your partition/error strings are so large. References • administration/management/BE_configuration.md • administration/management/compaction.md • knowledge/admin_and_operation/shared_data_compaction.md

Andrei Harbunou

09/15/2026, 10:50 PM

@Rocky I have MV refresh for 2 years, and "bad" history records are adding, so even if I delete they will appear again. how to prevent?

Rocky

09/15/2026, 10:50 PM

To prevent "bad" records from accumulating in the

_statistics_.task_run_history

table and blocking compaction, you should use a combination of history retention policies and data flow adjustments. 1. Disable Persistent Archiving If you do not strictly need a permanent audit trail of every MV refresh, you can disable the feature that writes these records to the internal physical table. This keeps history in the FE memory only (subject to memory limits and restarts), which avoids the disk-based compaction error entirely.

Copy code

sql
-- This stops new history from being written to the _statistics_ table
ADMIN SET FRONTEND CONFIG ("enable_task_history_archive" = "false");

2. Shorten the History Retention (TTL) If you want to keep the archive but prevent it from growing indefinitely, significantly reduce the Time-to-Live (TTL). The default is 7 days; reducing it to 1 day or a few hours will ensure the system purges these records much faster.

Copy code

sql
-- Set history retention to 1 day (86400 seconds)
ADMIN SET FRONTEND CONFIG ("task_runs_ttl_second" = "86400");

3. Handle the "2 Years" Refresh Scope Refreshing an MV for a 2-year range at once is likely what triggers the massive JSON strings. Even with

max_mv_task_run_meta_message_values_length

set to 8, StarRocks still tracks the status of all partitions involved in the task run internally. * Avoid Full Refreshes: Instead of one giant task, refresh the MV in smaller batches (e.g., month by month). * Increase
partition_refresh_number
: In your MV definition, ensure

partition_refresh_number

is set to a reasonable number (e.g., 10 or 50). This forces the system to split a large refresh into multiple smaller task runs, each with a much smaller metadata JSON. 4. FE Stability Setting To ensure that if a "bad" record does make it into the table, it doesn't cause the Frontend to crash or fail to load metadata, enable this setting (available in v3.3+):

Copy code

sql
ADMIN SET FRONTEND CONFIG ("ignore_task_run_history_replay_error" = "true");

Why
max_mv_task_run_meta_message_values_length
didn't fully work This setting only truncates specific lists (like the names of partitions refreshed). However, the

history_content_json

also contains: * Status Maps: A map of every partition attempted and its success/failure status. * Plan Diagnostics: The

planBuilderMessage

which describes how the refresh query was optimized. * Error Details: Large stack traces if a 2-year refresh fails halfway. If any of these fields independently exceed the 1MB limit (which often happens with 2-year partition sets), the compaction error will return. Disabling the archive (Step 1) is the most robust prevention. References • administration/management/FE_configuration.md • using_starrocks/async_mv/materialized_view_task_run_details.md • administration/management/compaction.md • sql-reference/information_schema/task_runs.md References • administration/management/FE_configuration.md

Open in Slack

Previous Next