@Rocky What is the data freshness for a table that is having results streamed into it using routine load, with exactly-once delivery semantics?

For a table using Routine Load in StarRocks, data freshness—the time it takes for data in Kafka to become visible in the table—is determined by the commit cycle of the individual load tasks. Even with exactly-once delivery semantics, Routine Load achieves near real-time ingestion. Here is the breakdown of how data freshness works and the parameters that control it: 1. The Commit Mechanism Routine Load splits a long-running job into a series of short-lived load tasks. Each task acts as an individual transaction: * Exactly-Once Guarantee: StarRocks consumes data from Kafka and commits the Kafka offsets to its own metadata storage in the same transaction as the data itself. Data is only visible in the table after the task successfully commits. * Visibility: Data is not visible row-by-row as it arrives in the BE (Backend); it becomes visible only in "batches" at the end of each task. 2. Key Parameters Affecting Freshness The frequency of these commits (and thus the freshness) is governed by three primary parameters: Parameter | Default Value | Description

max_batch_interval

| 10 seconds | The scheduling interval for tasks. It defines how often a new task is started.

task_consume_second

| 15 seconds | The maximum duration a task spends consuming data before it stops and commits.

max_routine_load_batch_size

| 4 GB | The maximum amount of data (in bytes) a task will consume before committing. 3. Typical Data Freshness Under default settings, data typically becomes visible within 10 to 15 seconds. * If data is flowing continuously, a task will consume for up to 15 seconds (or until the 4GB limit is hit) and then commit. * If data volume is very low, the system waits for the

max_batch_interval

to trigger new tasks. 4. Tuning for Lower Latency If you need higher freshness (lower latency), you can tune these parameters in your

CREATE ROUTINE LOAD

statement: * Decrease

max_batch_interval

: You can lower this to a minimum of 5 seconds. * Decrease

task_consume_second

: Setting this to a lower value (e.g., 5s) will force tasks to commit more frequently. Note on Performance: While you can reduce these values to achieve 5-second freshness, setting them too low (e.g., below 5s) can lead to an excessive number of small data versions (tablets), which increases the overhead on the StarRocks compaction process and can eventually impact query performance. For best performance, the recommended minimum for

max_batch_interval

is 10 seconds. References • administration/management/FE_configuration.md • sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md • administration/management/monitoring/metrics.md • faq/loading/Routine_load_faq.md