@Rocky is 500 batches per minute a problem for StarRocks?

Yes, 500 batches per minute (approx. 8.3 batches per second) can be a problem for StarRocks, especially if those batches are targeting the same table or a small number of tables. While StarRocks is designed for real-time analytics, it performs best when data is batched. Here is why high-frequency ingestion (8+ times per second) can be problematic: 1. The "Too Many Versions" Problem Each load transaction creates a new data version for the affected tablets. StarRocks must merge these versions through a background process called compaction. * The Risk: If you ingest data faster than StarRocks can compact it, you will accumulate too many versions. Eventually, you will encounter errors like

too many versions

or

compaction score too high

, which will cause ingestion to fail or slow down significantly to allow compaction to catch up. * Primary Key Tables: These are particularly sensitive to high-frequency writes because of the overhead of maintaining the primary key index and performing delete-and-insert operations. 2. Transaction Management Overhead The Frontend (FE) serializes transaction commits. * High-frequency commits (especially across many small batches) increase CPU load on the FE and can lead to Publish Version latency. * In shared-data (Cloud-Native) architecture, high-frequency commits can also lead to an excessive number of small files on object storage (S3/GCS/OSS), which degrades query performance. 3. Recommended Best Practices If you are currently at 500 batches per minute, consider the following optimizations: * Increase Batch Size / Decrease Frequency: The general recommendation is to aim for one batch every 5 to 10 seconds per table. This allows the system to process more rows per transaction with much less metadata overhead. * Routine Load Tuning: If using Routine Load (from Kafka), increase

max_routine_load_batch_size

and

routine_load_task_consume_second

to force larger, less frequent batches. * Shared-Data Optimization: If you are using StarRocks 3.2.3 or later in shared-data mode, enable the batch publish optimization:

sql
    ADMIN SET FRONTEND CONFIG ("lake_enable_batch_publish_version" = "true");

* Stream Load Transaction Interface: For high-concurrency scenarios, use the Stream Load transaction interface (introduced in v2.4) to combine multiple small writes into a single transaction. Summary: If the 500 batches/min are spread across hundreds of different tables, it may be manageable. However, if they target a single table, you should definitely look into batching your data on the client side before sending it to StarRocks. References • administration/management/monitoring/metrics.md • knowledge/usage/load/shared_data_batch_ingestion.md • faq/shared_data_faq.md • faq/Others.md • administration/management/BE_configuration.md • loading/Loading_intro.md

Sorry, this is specifically about materialisation - not ingestion @Rocky

Yes, 500 materialization refreshes per minute (approx. 8.3 per second) is a significant problem for StarRocks, even more so than for standard data ingestion. While StarRocks supports asynchronous Materialized Views (MVs), they are designed for analytical batch processing rather than sub-second "streaming" updates. Here is why such a high frequency is problematic: 1. The FE Scheduler Bottleneck StarRocks manages MV refreshes via a TaskManager in the Frontend (FE). * Scheduling Overhead: Every refresh task involves planning, metadata checks, and task state transitions. Attempting to schedule 500 tasks per minute can saturate the FE’s TaskManager and lead to high CPU/memory usage on the leader FE node. * Safety Limits: In StarRocks v3.3.0 and later, there is a configuration called

materialized_view_min_refresh_interval

which defaults to 60 seconds. The system explicitly prevents you from creating MVs with intervals shorter than this to protect the FE from being overwhelmed. 2. Task Merging and Serial Execution StarRocks enforces a rule that only one refresh task can run for a specific MV at a time. * If you trigger a refresh while the previous one is still running, the new trigger is either merged (if it's still in the queue) or queued (waiting for the current run to finish). * Unless your refresh task finishes in under 120 milliseconds (which is rare for analytical queries), the refreshes will simply back up, and you will not actually achieve 500 refreshes per minute. 3. Compaction and Versioning An MV refresh is essentially an

INSERT OVERWRITE

operation. * Version Creation: Each refresh creates a new version of the data. Refreshing 8 times a second will create versions much faster than the background compaction process can merge them. * Impact: This leads to the "Too Many Versions" error, which will eventually block the MV from refreshing and significantly degrade the performance of any queries reading from that MV. 4. Better Alternatives If you need data to be available for query in near-real-time: * Synchronous (Simple) MVs: If you are only doing simple aggregations or filtering, a "Synchronous" (Single-Table) MV updates automatically with every load and does not have the scheduling overhead of Async MVs. * Increase Refresh Interval: Aim for a refresh every 1 to 5 minutes at most. * Partitioned MVs: Use

PARTITION BY

on your MV. This allows StarRocks to perform an incremental refresh (only updating the specific partition that changed) rather than refreshing the whole dataset, which is much more efficient. * Query Rewrite: If the base tables are updated frequently, you can rely on Transparent Query Rewrite with a

query_rewrite_consistency

set to

LOOSE

or

DEFERRED

to use slightly stale MV data while avoiding constant refreshes. Summary: You should avoid refreshing MVs at sub-minute intervals. If you need sub-second freshness, you should look into optimizing your base table schema or using synchronous MVs rather than high-frequency asynchronous refreshes. References • administration/management/FE_configuration.md • using_starrocks/async_mv/troubleshooting_asynchronous_materialized_views.md • using_starrocks/async_mv/use_cases/data_modeling_with_materialized_views.md • using_starrocks/async_mv/use_cases/create_partitioned_materialized_view.md References • knowledge/usage/best-practice.md