Apache Druid

I would think so, but you'd want to keep it up-to-date, maybe replication?  (And of course you'd have to update druid configs to use it if needed.)

also if you are gonna run the two clusters at the same time, you would want to avoid doing ingestion and data management operations on the backup cluster

yes, my plan is to not run middlemanager, just launch historicals so that they can take their sweet time loading all the segment caches.

what do you mean by "problem"?

they certainly pose challenges for any distributed DB, although druid's always been good at handling them relative to other options — esp. with features like bitmap indexes and pruning for fast filtering, &amp; various approximate algorithms for ranking, counting, etc

oh? Maybe my knowledge is old then? I thought the bitmap index for string type could get really big on high cardinality columns?

I suppose it depends on what you mean by "really big" and whether you think it's worth it

IMO, if you do a lot of `=` or `in` on this column, it's worth whatever storage space it takes up in terms of performance improvement

if you never do filters like that, and you don't find the index useful, you can disable it on a column by column basis

`createBitmapIndex: false` in the dimension definition in ingestion spec

Related question, in the old days, circa v0.15.x, broker/historical used to crash if you have very high cardinality string columns, does it no longer happen on groupBy v2 (since groupBy v2 can offload to disk)?

yeah, these days, worst you will get is ResourceLimitExceeded error

Hm… from the config, `druid.worker`  still points to `WorkerConfig` .

Never mind about my original question, it does use `druid.worker.category`

holy cow, the new indexer is 33% faster and consumes way less memory.

But I am confused about the server breakdown. I see:
• type: `indexer`, tier: `_default_tier` . This is the thing that performs ingestion.
• type: `peon` , tier: `my-custom-category` . What does this do?
What is the relationship between `peon`  and `indexer`?

I _believe_ that the peons serve to allow queries against the appending index.

Found my answer:
```  private AppenderatorConfig rewriteAppenderatorConfigMemoryLimits(AppenderatorConfig baseConfig)
  {
    long perWorkerLimit = workerConfig.getGlobalIngestionHeapLimitBytes() / workerConfig.getCapacity();
    return new MemoryParameterOverridingAppenderatorConfig(baseConfig, perWorkerLimit);
  }```
It’s divided by capacity

got a reference to what you're thinking of?

I don’t see the following on `ThreadingTaskRunner.java`
```final File logFile = new File(taskDir, "log");
final File reportsFile = new File(attemptDir, "report.json"); ```

<@U030MNRGBNG> Do you see your task logs when running `Indexer` daemon? I am seeing blank screen on the Druid UI for the `Indexer` .

In indexer mode the logs are routed to the `indexer` log itself, not the individual task VMs (which is where the UI is attempting to fetch from I think).  We send out `indexer` logs to stdout and collect to Kibana, and that contains all logs for the entire node.

ok, ok, I thought I messed up something in the log4j2.xml

Yeah it took me a bit to locate.  Since the indexers are all just running in a threadpool inside the single VM all logs go to one place.

is that the one that shows the name of the variable in NPEs?

I think it's on by default in JDK 17, I've noticed it when running unit tests and stuff. Definitely useful for testing

Not sure if we'll enable it in production when we update our prod to JDK 17. Depends on whether there's any performance overhead

If it’s on by default on JDK 17 then we have been running it in prod for a long time without performance regression, haha

image.png

interesting. do you see any errors in historical or broker logs?

I don't think there were, but I since removed the pods. I'm right now trying again from scratch and so far it looks like this time it works....

certain metadata fields (including `num_rows`) are populated in the background and take some time to appear when a segment is first loaded or when a Broker restarts. It doesn't affect regular queries, only metadata queries. You might have run into this

<@U0344FW86DD> I think this also had to do with shared volumes.
<https://apachedruidworkspace.slack.com/archives/C0309C9L90D/p1662491715810719?thread_ts=1662491715.810719&amp;cid=C0309C9L90D|https://apachedruidworkspace.slack.com/archives/C0309C9L90D/p1662491715810719?thread_ts=1662491715.810719&amp;cid=C0309C9L90D>

Did you get that for a SQL query?  You might file a bug report if so.  Without getting too technical, I saw an example of that happening that was a bug.

Getting it on reindexing as well as sometimes on queries...

I found this in an old report, maybe it'll help:

Thanks <@U0411DE2SS0>
I think this might have been the problem: <https://apachedruidworkspace.slack.com/archives/C0309C9L90D/p1662491715810719>

Oh - that makes sense!  I hadn't seen shared volumes used for historicals before.  They should each have their own storage, afaik.

Yes it assumes that the local segment cache is per historical.

<@U03TBGF0LTG> that means using local storage as deep storage is incompatible with more than one historical - unless every historical has its own storage?

local deep storage (`druid.storage.storageDirectory`) can be on a shared volume (like NFS or something like that) but the server caches (`druid.segmentCache.locations`) must be on dedicated per-server volumes

Ok, thanks for the explanation <@U0344FW86DD>

<@U0344FW86DD> can I disable the cache? Does not really seem to make sense in my scenario (all historicals in the same cluster accessing the same shared volume)?

<@U03SRJK8WCW> Segment cache would be a local disk per historical and would be much much faster than NFS/Shared volume

No you cannot disable segment cache location

As druid is build for :speedboat: each historical basically runs q's on the local cached segment which is super fast

If we serve the segment via a NFS , querying will slow down

the error is due to losing session w/ zookeeper

wild guess: most common reason for this is servers going temporarily offline due to very long GC pauses, which would in turn be due to memory pressure

is it possible this is happening in your case?

yes we are seeing few OOM errors and the hprof file size is around 80 GB . is there a easy way to identify which query is causing this issue ?

not really, since OOME happens at the level of the entire JVM. and an 80 GB heap file is going to be tough to open on a workstation!

i'd consider lowering the heap size a bit — maybe to 16 GB — triggering the OOME again, then opening the resulting hprof on a workstation that has &gt; 16 GB of RAM

this will let us see the objects that are taking up the space, which generally makes it pretty clear what the cause is

another wild guess, too — see <https://apachedruidworkspace.slack.com/archives/C0303FDCZEZ/p1659300492451949?thread_ts=1658856818.258609&amp;cid=C0303FDCZEZ>

that's going to be set up by default in 24.0 (next release)

just replied in that thread :raised_hands:

I found "org.apache.hadoop.hive.druid.DruidStorageHandler" and tried it in Hive and got an error I assume it's because I didn't set `hive.druid.broker.address.default`.

```hive&gt; CREATE EXTERNAL TABLE usr.druid
    &gt; STORED BY 'org.apache.hadoop.hive.druid.DruidStorageHandler'
    &gt; TBLPROPERTIES ("druid.datasource" = "druid data source name");
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.RuntimeException: MetaException(message:org.apache.hadoop.hive.serde2.SerDeException java.io.IOException: org.apache.hive.druid.org.jboss.netty.channel.ChannelException: Faulty channel in resource pool
	at org.apache.hadoop.hive.druid.DruidStorageHandlerUtils.submitRequest(DruidStorageHandlerUtils.java:196)
	at org.apache.hadoop.hive.druid.serde.DruidSerDe.submitMetadataRequest(DruidSerDe.java:266)
	at org.apache.hadoop.hive.druid.serde.DruidSerDe.initialize(DruidSerDe.java:182)
	at org.apache.hadoop.hive.serde2.AbstractSerDe.initialize(AbstractSerDe.java:54)
	at org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:533)
	at org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:453)
	at org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:440)
	at org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:281)
	at org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:263)
	at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:858)
	at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:892)
	at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:4356)
	at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:354)
	at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:203)
	at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
	at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2183)
	at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1839)
	at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1526)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227)
	at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
	at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
	at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821)
	at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:244)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:158)
Caused by: org.apache.hive.druid.org.jboss.netty.channel.ChannelException: Faulty channel in resource pool
	at org.apache.hive.druid.com.metamx.http.client.NettyHttpClient.go(NettyHttpClient.java:137)
	at org.apache.hive.druid.com.metamx.http.client.AbstractHttpClient.go(AbstractHttpClient.java:14)
	at org.apache.hadoop.hive.druid.DruidStorageHandlerUtils.submitRequest(DruidStorageHandlerUtils.java:194)
	... 31 more
Caused by: java.net.ConnectException: Connection refused: localhost/127.0.0.1:8082
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716)
	at org.apache.hive.druid.org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
	at org.apache.hive.druid.org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
	at org.apache.hive.druid.org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
	at org.apache.hive.druid.org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
	at org.apache.hive.druid.org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
	at org.apache.hive.druid.org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.apache.hive.druid.org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)
)```

simplest method is native batch ingest (<https://druid.apache.org/docs/latest/ingestion/native-batch.html>) with the appropriate input format (for example, if parquet use: <https://druid.apache.org/docs/latest/ingestion/data-formats.html#parquet>)

in the next Druid release we will also have SQL-based ingest!

those are JDBC connection params, not GET params; they are parsed by the client (Avatica driver)

the client/server protocol uses JSON or Protobuf messages

But looks like it is still HTTP based, yes?

interesting. The LDAP auth works fine via HTTP basic auth mechanism.
but the moment I extend Nginx to a custom HTTP proxy, Authentication stopped working only on Avatica connections.

eg. on how to use the custom proxy inside kubernetes nginx-ingress:
```<http://nginx.ingress.kubernetes.io/auth-url|nginx.ingress.kubernetes.io/auth-url>: '<http://ldap-proxy.svc.cluster.local>:port_number/auth'```

oo I found a very helpful comment.
```      // JDBC authentication uses the JDBC connection context instead of HTTP headers, skip the normal auth checks.
      // The router will keep the connection context in the forwarded message, and the broker is responsible for
      // performing the auth checks.```

yeah JDBC auth works differently from other APIs and basically requires user/password

segment files on deep storage are zip compressed. files, once downloaded locally, are decompressed. These decompressed files are memory-mapped by historical.

No. it can't be. since you might memory map a segment. if that segment gets deleted from disk but is still memory-mapped and is being queried then bad things can happen.

Ok, not sure if I made it clear what I mean. I don't mean volatile as in in-memory, but volatile as in stored in /tmp instead of a proper volume.

you mean like that files can just disappear or deleted by OS - is that right?

I don't think the OS will delete them. But I'm not sure if they will survive a pod restart.

thats fine. if the pod restarts and the data goes away, it would be downloaded again from deep storage.

I'm trying to 'fix' the datasource names bc of bad trino: <https://github.com/trinodb/trino/issues/6850>

Why don't you just re-ingest/re-index to a new datasource?

I only have ~ last 5 days or so in kafka, 30 days used/loaded on the historicals, and a few months of data not loaded (unused), although it could be fine to keep the 'old' segment as historical only (but during the migration I would have to rewrite all queries to query both datasources, paginations would be hard)

I think I gonna export to csv, import to s3 and load back to trino. I was doing the same thing, but download to local db, importing to postgres and then running some analytics with lots of joins where druid just can't beat pg with indexes

I mean reindex from druid.
```{
  "type": "index_parallel",
  "spec": {
    "ioConfig": {
      "type": "index_parallel",
      "inputSource": {
        "type": "druid",
        "dataSource": "OldDataSourceName",
        "interval": "1970-01-01T00:00:00/9999-01-01T00:00:00" 
      }
    },
...
    "dataSchema": {
...
      "dataSource": "newdatasourcename",
...
    }
  }
}```


This will leave the old datasource as-is and create a new datasource with a new name

while at it, you can fine tune your segment size et cetera

• turn off kafka ingestion to old datasource
• run Druid reindex task
• check that it worked (at least compare numRows between old and new datasource)
• turn on kafka ingestion to new datasource

oh! that's make sense, I can use this to reindex for the current data!

(Disclaimer, I'm not a Druid developer, just a user, but I did just that, and it worked great (I had lots of small segments before))

just use the wizard under "Load Data" to create the spec

yeah, I will start this job
I will not change the current ingestion right now, but I can run the analysis on a copy

Yes, re-indexing is the only real way of renaming your datasource.

easy task but I'm unlucky today!  First died with crazy error (Expected [700,958] bytes, only saw [615,989], potential corruption?)
Then I reduced merge tasks, try again, then out of space (can I set the path of this task only? I've setup only 11GB on the /var/druid for the MM {just to clean restarts} ,but I have 600gb local disks on the host VM)
the third I cancelled to remove some columns, tried again with just 2 days loaded, now I got out of memory ! haha

you should have better luck in the next druid release! we've implemented substantially more self tuning for the new SQL-based ingest feature

i have been enjoying using it in prerelease testing :slightly_smiling_face:

you might find some clues if you open the task log (the `.../log` url)

It sends periodic kill tasks for a given datasource if there is any unused segment for that datasource in the metadata store. Your segments have probably been marked as unused but haven't been deleted yet.

<@U0318CX1NH4> I checked the metadata database. There are no more entries for that datasource in table druid_segments

table druid_datasource also does not contain the datasource anymore

is it possible you have configured it to do killing for certain datasources, and that one is still in the list?

absent that, i don't think it has any way of knowing about a datasource if all the segments are gone

I am not 100% sure what you are trying to do so I might be off the mark here, but if you want an average value for each id then you need to have a GROUP BY

so... just from your description `for each id` =&gt; means there is a GROUP BY missing somewhere

can you give and example of the transformation you are looking to perform, doe not have to be your real data, but if you could provide a fake (or real) sample of a handful of rows and what you would expect to get out that would really help

Specifically I am not tracking the nesting you are trying to do... you are nesting a query within the select but it looks (to me) independent form the outer query

if I run this query:
```SELECT
  "system_id",
  (
    SELECT
      AVG("value")
    FROM table
    WHERE name = 'mem_usage'
  ) AS "avg_mem_usage",
  AVG("value") AS "avg_gpu_util"
FROM table
WHERE name = 'gpu_util'
GROUP BY 1```

I get results like this:
```system_id       avg_mem_usage       avg_gpu_util
328931          3.6                 36.53
328932          3.6                 24.31
328933          3.6                 2.67
328934          3.6                 9.88
328935          3.6                 98.42```

although I expect this:
```system_id       avg_mem_usage       avg_gpu_util
328931          3.6                 36.53
328932          7.54                24.31
328933          1.19                2.67
328934          2.51                9.88
328935          28.16               98.42```

the table rows would look similar to this:
```[
    {"name": "mem_usage", "timestamp": 1662730600, "system_id": 328931, "value": "2.3"},
    {"name": "gpu_util", "timestamp": 1662730605, "system_id": 328931, "value": "49.3"},
    {"name": "mem_usage", "timestamp": 1662730630, "system_id": 328931, "value": "5.88"},
    {"name": "gpu_util", "timestamp": 1662730635, "system_id": 328931, "value": "49.3"},
    {"name": "mem_usage", "timestamp": 1662730700, "system_id": 328931, "value": "6.71"},
    {"name": "gpu_util", "timestamp": 1662730705, "system_id": 328931, "value": "50.2"},
    {"name": "mem_usage", "timestamp": 1662730730, "system_id": 328931, "value": "3.56"},
    {"name": "gpu_util", "timestamp": 1662730735, "system_id": 328931, "value": "48.54"},
    {"name": "mem_usage", "timestamp": 1662730800, "system_id": 328931, "value": "4.66"},
    {"name": "gpu_util", "timestamp": 1662730805, "system_id": 328931, "value": "48.55"},
    ...
]```

Also a similar reporting here: <https://apachedruidworkspace.slack.com/archives/C0309C9L90D/p1649730257421159>

<@U03DW1FMU4X> Have you check if ZK was unstable around that time? Once we stabilized our ZK, we no longer see this problem.

<@U03149ALHN3> yea, ZK has been healthy all time. I don't see any error/warn logs in ZK either.

Did you happen to have a lot of rows inside druid_tasks?

yes, we currently have ~20k rows in that table.

ok, did you enable auto pruning settings in overlord?

We usually cleanup tasks older than 24 hours.

Something like this: <https://apachedruidworkspace.slack.com/archives/C0309C9L90D/p1661285855315429?thread_ts=1661281086.149479&amp;cid=C0309C9L90D>

We currently don't. I can try. But does that help with this issue?

Based on our trial and errors, this problem occurs when there are 2 situations:
1. ZK is unstable, goes up and down.
2. Overlord is unstable, goes up and down, confusing the membership logic. And Overlord typically becomes unstable when there are so many tasks it tries to load in heap. Our rule of thumb is to keep number of druid_tasks to less than 5k.

Doing no. 2 is worth it since there are a number of other bugs due to too many tasks in Overlord heap, so might as well.

Interesting.... Did you find changes in heap memory usage on overlord in those cases?

i checked internally (@ imply) and also i see a recent customer support case that looks similar (an exception during leader election while reacquiring locks). let me see if that is making progress as to figuring out a root cause

<@U03DW1FMU4X> definitely. Heap size and G1 GC collection count are the two metrics that strongly correlated with this bug.

<@U03149ALHN3> I don't see those jvm metrics spike when we see those errors.... But thanks for the suggestion, we do see the row count in task table decrease a lot. task_lock table still has a large row count (20k). Do you do anything with that table?

<@U0344FW86DD> any updates on the case you were looking at? :slightly_smiling_face:

task_lock should have been cleaned when ingestion is done. I recommend deleting all of the old records, just keep the ones for the currently running ingestion.

i just checked again — there are some believable theories as to how this may happen, but unfortunately we haven't been able to repro the problem in the lab, and we don't have enough logs in order to confirm the theories in this specific case

our current thinking is twofold

1. <https://github.com/apache/druid/pull/12901|PR 12901> was done to fix an unrelated bug, but it also modifies task shutdown behavior so the task is marked finished in the metadata store prior to the lock being released (it used to be the other way round). under one of the theories, this would fix the problem. however, like i mentioned, we haven't been able to confirm this

2. we think it's a good idea to modify the Overlord startup sequence such that, when there's a lock conflict like this, the relevant tasks are canceled and the Overlord continues (instead of refusing to start up). this will prevent it from borking a cluster

however, our preference would be to really get to the bottom of it. so if you have Overlord logs from the one that was leading prior to this happening (the new leader won't be of much help, sadly!) then posting them here will help us debug it: <https://github.com/apache/druid/issues/11653>

<@U0344FW86DD> Thanks for the details! I will keep an eye on it. We currently don't turn on info logs for overlord, do you need info logs or warn logs should be good enough?

for this, we would need INFO, since we're trying to observe a sequence of events

hmm. it should pick up the credentials in the properties. are you sure those credentials have access?

Yes the credentials have full s3 access. Looks like error says the problem is in the wrong key `InvalidAccessKeyId`, not `AccessDenied`

ioConfig.json

I’ve tried setup the cluster from scratch and noticed the credentials from tasks work correctly when the deep storage is local
```druid.storage.type=local
druid.storage.storageDirectory=var/druid/segments```


I tried this locally and it works fine for me:

• S3 deep storage configured with `druid.storage.type=s3`, `druid.storage.bucket`, `druid.storage.baseKey`, `druid.s3.accessKey`, and `druid.s3.secretKey`
• S3 input source accessing a bucket in a different account using `properties` and `accessKeyId` + `secretKeyId`

maybe there is an issue with the endpoint: it's possible the s3 input source is contacting minio due to server-wide config (such as `druid.s3.endpoint.url`) and the error is actually from minio

if that's it: try using `endpointConfig` in your s3 input source too

thank you for support
yes, you are correct the problem is in an endpoint

looks like there is not a workaround how to use multiply endpoints

I’ve even tried to setup separate MM with different endpoint, but without success as well

Hey <@U041FEKGWDB> I just had a quick check in Druid Forum as I seemed to remember someone else having a similar issue re: <http://Min.IO|Min.IO> / S3 – I really thought someone found a solution (!) but I don’t think one was found… however maybe you could DM some people in the conversation on there and see if they worked anything out?  <https://www.druidforum.org/t/fail-to-configure-data-ingestion-from-minio-object-storage/6783>

&gt; if that’s it: try using `endpointConfig` in your s3 input source too (edited)
I didn’t understand correctly this note
I see, there is that parameter in the task configuration
```        "endpointConfig": {
             "url" : "<http://s3-store.aws.com|s3-store.aws.com>",
             "signingRegion" : "us-west-2"
         },```
let me check this point

Cool thank you :slightly_smiling_face:. In fact <@U030AQ8AR8E> and <@U030K4UM3H7> and I were talking about this ourselves last week (!!!) as we’re building out a template POC environment with <@U030L4P6UDA> that uses <http://Min.IO|Min.IO>… so we will like to know, as well!

It works as expected!:tada:
Thank you guys!

When I have this issue, I use a `SELECT` (or `scan`) query via the POST API (they stream) but I have a script that breaks it down by day or hour and gets each one individually (or in batched parallel, for extra speed)

You mean using method 1 of API calls, it is doable by splitting the response into chunks ?

break down each day or hour to one api-request to parallelize the work, but it's not needed if you don't have enough historicals CPUS

You would need to set the correct value of `-XX:MaxDirectMemorySize=30g` in the `jvm.config` of your historical.

The `druid.indexer.runner.javaOptsArray` does not affect the historical.

I think taskContStatsMonitor is an overlord thing

We might have to update the documentation

Yes, `TaskCountStatsMonitor` works only with Overlord as it is bound with a concrete impl only in overlord. To use it correctly, you must specify this monitor in the overlord `runtime.properties` (and not `common.runtime.properties` or that of any other service).

<@U031AL57448>, does your use case require some task count at the middle manager level?

Since doc does not say anything about it, I thought I need to add it to middlemanagers since they are task related. We are running coordinator as overlord. I've added it on coordinator config. I didn't get any error but still I'm not able to get task metrics. I'm using prometheus emitter and see below warnings on metric page.
```# HELP druid_task_waiting_count Number of current waiting tasks. This metric is only available if the TaskCountStatsMonitor module is included.
# TYPE druid_task_waiting_count counter```

Okay it started showing after new tasks created. I'm confused because that warning is always there sorry. Thanks for help.

:+1: . Yeah, I think the warning can be a little misleading when the module is already included.

when i add the 3º supervisor druid changes all the datasources change to "UNHEALTHY_TASKS"