Hi, is there a way to override the 1 task per BE limit for routine loads? > The actual task concurrency is defined by the minimum value among the number of BE nodes that are alive, the number of the pre-specified Kafka topic partitions, and the values of

desired_concurrent_number

and

max_routine_load_task_concurrent_num

.

No. From the doc, for each routine load: "The actual task parallelism is determined by the minimum value of the multiple parameters:

min(alive_be_number, partition_number, desired_concurrent_number, max_routine_load_task_concurrent_num)

". However each BE can have multiple concurrent routine load task across all routine loads, determined by fe config

max_routine_load_task_num_per_be

I'm aware of the docs simple smile I'm looking for a workaround, since it really limits the ingestion throughput from kafka

If your alive BEs is the limiting factor, then really there's no option to increase concurrency unless you add more nodes. You can still increase the throughput by increasing

max_routine_load_batch_size

,

routine_load_task_consume_second

fe config (dynamic config).