Hi All, I have a question regarding the performance of ingestion through routine loads: As far as I understand, it consumes data during task_consume_second or until it reaches max_routine_load_batch_size read, then it may take few seconds for commiting/publishing the data, and then will wait max_batch_interval before it starts reading again. With the default values of task_consume_second at 15s and max_batch_interval at 10s, it means it is roughly reading only half of the time. What it the reason for the max_bach_interval, and why can't it be set lower than 5s? I can see the warning in the documentation saying it may generate too many versions if it is set to a too low value but as far as I undestand, one version is created for each batch so why wouldn't we increase the task_consume_second if that happens instead of increasing the max_batch_interval ? I managed to increase the loading performance by setting the task_consume_second to 30 or 60s and lowering max_batch_interval to 5s and it seems it is just working fine but there is now a higher latency with data. Is there something that starrocks needs to do in between batches that would justify not lowering more the max_batch_interval ? Please let me know if there is something I did not understand correctly about the way it is working. THanks

Hello, the

max_batch_interval

is set to higher values to reduce the number of insertion, which in result reduce the number of tablet - For every time the max_batch_interval timer hits 0, the data is inserted and a new task is created.