This message was deleted.

I would check the health of the overlord, see if it has any issues during that time

yea i have checked the overload logs, its pretty much throwing the same error.

Failed to stream task reports for task

2023-08-07T19:15:28,126 WARN [qtp1405357782-150] org.apache.druid.indexing.overlord.http.OverlordResource - Failed to stream task reports for task single_phase_sub_task_voice4g_radio_node_rollup_keibkelj_2023-08-03T21:06:48.699Z
org.jboss.netty.channel.ChannelException: Faulty channel in resource pool
at org.apache.druid.java.util.http.client.NettyHttpClient.go(NettyHttpClient.java:134) ~[druid-core-25.0.0.jar:25.0.0]
at org.apache.druid.java.util.http.client.CredentialedHttpClient.go(CredentialedHttpClient.java:48) ~[druid-core-25.0.0.jar:25.0.0]
at org.apache.druid.java.util.http.client.AbstractHttpClient.go(AbstractHttpClient.java:33) ~[druid-core-25.0.0.jar:25.0.0]
at org.apache.druid.indexing.overlord.hrtr.HttpRemoteTaskRunner.streamTaskReports(HttpRemoteTaskRunner.java:1031) ~[druid-indexing-service-25.0.0.jar:25.0.0]
at org.apache.druid.indexing.common.tasklogs.TaskRunnerTaskLogStreamer.streamTaskReports(TaskRunnerTaskLogStreamer.java:59

"Faulty channel in resource pool" this is usually a resource issue. Check the health of the overlord, memory and CPU. How many tasks are you running and what size is the Overlord?

we have 2 overload service pods running. we observed that, one of the pod has very high CPU utilization and other one 0. Not exactly sure why its skewed. do you have any idea by any chance ?

AFAIK only 1 overlord and coordinator take the load at the same time (leader) and cannot be scaled, multiple instances are only used for High Availability, so another instance can take over if the other fails (but I think it needs 3 for quorum purposes). Do you have JVM statistics (free heap + gc cycles) for all components? (overlord, coordinator, historical, peon, etc.). We often ran into issues that we undersized single components and when the Heap gets low it might spend all time in Garbage-Collection not answering any more requests. Same is true for System Memory on the Nodes. We only could achive a stable environment with average memory usage about 70-80% on all components and the same for Heap (from my experience as sys-admin this is true for all kinds of systems and not only druid).

Sure, we will definitely increase the CPU resources and see if it resolves the issue. currently we have almost 170 tasks in a running state with 6 CPU and 65GB for that container. more than 90% percent of the tasks are index_kafka and remaining are probably related to compaction. please recommend the CPU

We use 6 Nodes of each having 8 CPU and 64GB Memory to run 130 Tasks and 50 compactions. Each of the Node runs one middlemanager pod which uses 58GB of Memory (we leave 6 GB for Kubernetes Services and General Node OS Overhead). Each middlemanager (so each Node) runs 30 Tasks. The Task jvm heap settings are 128mn xms and 1g xmx and 1g direct Memory. Also the ingestion Tasks are tuned to 250MB max bytes in Memory. We also set max rows in Memory which we calculate on our own from existing Segmentsizes/rows

okay do you mean i have to tune the rollup config with max bytes in Memory and max rows in Memory ? Our set up has autoscaling enabled with middle managers of 4 CPU and 17GB ram, which means it spawns a middle manager for each task. and 2 coordinator + Overload service with each with 10 CPU with 65 GB ram. only error that i see even after increasing the resources is

org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexSupervisorTask - Encountered exception when getting live subtask report for task: single_phase_sub_task_voice4g_radio_node_rollup_mkheibmo_2023-08-04T01:07:38.612Z

In the supervisor spec there is the "tuningConfig" in the KafkaSupervisor, I am not sure if something similar exists for other supervisors: https://druid.apache.org/docs/latest/development/extensions-core/kafka-supervisor-reference#kafkasupervisortuningconfig In general regarding MiddleManagers they only scale depending on TaskCounts. In your middlemanager config you configure the amount of tasks it can sustain (druid.worker.capacity) (but you yourself decide how much this is) and druid checks when adding a new task if "amount of middlemanagers" * "druid.worker.capacity per MiddleManager" provides enough room for the tasks or the running tasks use this up. If there are too little free tasks it spins up a new middlemanager. This does however not change how many rows/memory a single task can sustain. I am talking about the kafka supervisor setup but I assume that this is the same for all tasks. You need to setup explicitly the following properties to tune this. Please note that this is an example what works for us, your settings might differ:

druid.indexer.runner.javaOptsArray=["-server","-Xms128m","-Xmx1g", "-XX:MaxDirectMemorySize=1g", "-XX:+IgnoreUnrecognizedVMOptions", "-XX:+UseG1GC", "-XX:+ExitOnOutOfMemoryError","-Duser.timezone=UTC","-Dfile.encoding=UTF-8", "-Djava.io.tmpdir=/mnt/tmp/task/tmp", "-Daws.region=eu-central-1", "-Dorg.jboss.logging.provider=slf4j", "-Dnet.spy.log.LoggerImpl=net.spy.memcached.compat.log.SLF4JLogger", "-Dlog4j.shutdownCallbackRegistry=org.apache.druid.common.config.Log4jShutdown", "-Dlog4j.shutdownHookEnabled=true","--add-exports=java.base/jdk.internal.ref=ALL-UNNAMED",	  "--add-exports=java.base/jdk.internal.perf=ALL-UNNAMED", "--add-exports=java.base/jdk.internal.misc=ALL-UNNAMED",	  "--add-opens=java.base/java.lang=ALL-UNNAMED",	  "--add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED",	  "--add-opens=java.base/java.io=ALL-UNNAMED",	  "--add-opens=java.base/java.nio=ALL-UNNAMED",	  "--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED",	  "--add-opens=java.base/sun.nio.ch=ALL-UNNAMED"]
druid.indexer.fork.property.druid.processing.buffer.sizeBytes=100m
druid.indexer.fork.property.druid.processing.numThreads=5
druid.indexer.fork.property.druid.processing.numMergeBuffers=2

If this is set to the settings about e.g. no matter how big your middle managers are a task can only take 1GB of Java Heap and 1GB of direct memory.