@Rocky pymysql.err.OperationalError: (5609, 'Failed to load data into tablet 31510, because of too many versions, current/limit: 1002/1000. You can reduce the loading job concurrency, or increase loading data batch size. If you are loading data with Routine Load, you can increase FE configs routine_load_task_consume_second and max_routine_load_batch_size,: be:127.0.0.1')

Hi! This is Rocky. That error (5609) occurs when the number of data versions (rowsets) in a single tablet exceeds the system limit (default is 1000). This typically happens because data is being loaded too frequently in small batches, and the background compaction process cannot merge these versions into larger files fast enough to keep up. To resolve this, you can follow these three strategies: 1. Tune Routine Load (Recommended) If you are using Routine Load, the goal is to make each load task larger and less frequent. You can adjust these FE dynamic configurations:

sql
-- Increase the consumption time for each task (default 15s)
ADMIN SET FRONTEND CONFIG ("routine_load_task_consume_second" = "30");

-- Increase the maximum batch size (default 4GB, but check if your tasks hit this limit)
ADMIN SET FRONTEND CONFIG ("max_routine_load_batch_size" = "1073741824"); -- e.g., 1GB

2. Accelerate Background Compaction You can make the BE (Backend) merge data versions more aggressively by adjusting these BE configurations: * For Shared-nothing (Local Storage): Modify

be.conf

or use SQL to update dynamically:

sql
    -- Increase compaction threads (suggested: 25% of CPU cores)
    UPDATE information_schema.be_configs SET VALUE = "8" WHERE name = "compact_threads";

    -- Speed up individual compaction tasks by merging fewer files at once
    UPDATE information_schema.be_configs SET VALUE = "100" WHERE name = "max_cumulative_compaction_num_singleton_deltas";

* For Primary Key tables: Increase

update_compaction_num_threads_per_disk

in

be.conf

. 3. Immediate Workaround (Increase the Limit) If you need to resume loading immediately while waiting for compaction to catch up, you can increase the version limit (though this may slightly impact query performance):

sql
-- Increase the version limit per tablet (default 1000)
UPDATE information_schema.be_configs SET VALUE = "2000" WHERE name = "tablet_max_versions";

How to Monitor You can check which tablets have the highest version counts using:

sql
SHOW PROC '/compactions';

Look for tablets with a high

cumulative_compaction_score

. If the score remains high, it means compaction is struggling to keep up with your ingestion rate. References • administration/management/BE_configuration.md • administration/management/compaction.md • knowledge/admin_and_operation/shared_data_compaction.md