This message was deleted.

If there are more tasks in publishing you should check why it is not publishing within time. And we should check the configuration parameters for the ingestion SPEC and tune.

completionTimeout

: The length of time to wait before declaring a publishing task as failed and terminating it. If the value is too low, your tasks may never publish. The publishing clock for a task begins roughly after

taskDuration

elapses.no (default == PT30M)

we have already tried tweaking the required configuration for the ingestion. We observed its taking more time for the segment handoff, after tuning, we don't see enough improvement even after upgrading druid to 27. As we have more number of middlemangers, approximately 300. there is lot interprocess communication happening with coordinator/overlord process and causing the issues in all other places

The total number of tasks that are seen is also impacted by the number of replicas or the number of partitions in the kafka stream. You could also tune the segment size and increase it so that tasks are reading more data before publishing. That could reduce some of the instances of publishing and (hopefully) reduce the work on the middle managers