Apache Druid

I don't have all the details, but coordinator does (at least) two things: balance and load.  If historicals are unbalanced, it might decide to move some, up to maxSegmentsToMove per cycle.  For loading (from deep storage), it will attempt to load up to maxSegmentsInNodeLoadingQueue per cycle.

My understanding is that it also prioritizes new segments (segments without any replicas in historicals) which would be the ones related to handoff from realtime ingestion.

all segments awaiting handoff would get processed by the relevant LoadRule on each execution of RunRules

the only throttling that applies to fresh segments is `maxSegmentsInNodeLoadingQueue` (if servers have very deep load queues, we'll wait til the next RunRules before adding more to load)

*Natural Intelligence* has joined this channel by invitation from *Apache Druid*.

Also, in general, whats the best way to do some post processing of columns after a join?
We do a join of table_1 and table_2, and then want to combine col_A from table_1 and col_B from table_2 into a new column ( = col_A - col_B).
Currently using scan with virtual column to achieve this.

Hi Apoorv,  regarding your last question --

I just ran a test SQL join, e.g.:

```Select a.key, a.col_A - a.col_B diff
  from tableA a 
  join tableB b on a.key = b.key```
and did an explain plan to generate the native query JSON, and it looks like it turned it into a scan with virtual column, which is what you are using.

I don't know of any other performant way of doing this.  Is this a 1-1 join between the two tables or are them multiple matches for each key?

This is a 1-1 join, single match for each key.
However, the virtual columns approach is not working because of <https://github.com/apache/druid/issues/13851>

Did you see the last comment, that he got the query to work using some escaped quotation marks?  What's your expression?

Oh, if that's the case and this is just a syntax typo issue, then here's what came out of my explain plan for the SQL join query:

  "virtualColumns": [
    {
      "type": "expression",
      "name": "v0",
      "expression": "(\"added\" - \"j0.added\")",
      "outputType": "LONG"
    }
  ],

Note the escaped quotes around the column names used in the expression, including the table.column dot notation.

If you know SQL at all I would suggest you start by running this as a SQL query, make sure it works, then check the explain plan to see what it is coming up with for Native.

Thanks for the suggestions john and ben.
The write sql and explain sounds like a good approach going forward.
Saket and I are working together, so yeah I know about the solution now.
FYI, is there a reference for the syntax expected in the expression field?

It looks to me like this is regular SQL syntax quoting rules, e.g. chars like underscore _ don't require quoting, but hypens, spaces, etc, do, as do reserved keywords like "from" or "sum"

Quoting token names to preserve "illegal" characters and reserved keywords is double-quoted ... single-quote is used for literal strings in SQL.

Looking at the above example, I would guess that the quotes are those individual fields are not needed, e.g. this should work:

```  "virtualColumns": [
    {
      "type": "expression",
      "name": "v0",
      "expression": "(added - j0.added)",
      "outputType": "LONG"
    }
  ],```
It's worth trying out a few just to see what works and what doesn't.  You should find the rules are pretty consistent here.

Are you using the prometheus metrics emitter?
You should be able to use the metric `query/time` to get max/min/avg query times.
Docs: <https://druid.apache.org/docs/latest/operations/metrics.html#query-metrics>

<@U0318CX1NH4> yes i am using query/time metric
but confused how to use query_time_bucket to get max time

`query/time` value is histogram type, so the wholepoint is to use for a heatmap type chart or to get quantiles from it. As the data is bucketized, you can’t get a min/max value out it.

For quantiles use:
```histogram_quantile(0.5, sum(rate(query_time_bucket{druid_service="druid/broker"}[$__interval])) by (le)) 
histogram_quantile(0.95, sum(rate(query_time_bucket{druid_service="druid/broker"}[$__interval])) by (le)) ```
For a heatmap use something like:
```sum by(le) (increase(query_time_bucket{druid_service="druid/broker"}[$__interval]))```

with the heatmap view you can see what’s the window range for most your queries and even their max buckets. So I can tell I have very few awful queries in the 2-5 min range, but they are super rare. And even during the highest usage most my queries where &lt; 100 ms

<@U04L6JH4H2M> Thanks for help
`you can't get a min/max value` yes… its bucket so i cant get min max value got it

Yes, else the data , when queried, may not be accurate. Can you increase the segment cache ?

What’s the CPU core and memory of your historicals?

Thanks for you quick help, I can increase the cache, but if I got 20T data in druid in the future, then I need to increate the cache to 20T, right?

2 core and 10G memory for each historical

The question is whether you really need to query all of the time intervals for 20T. That’s why we have retention rules

I am afraid currently I need to cache all the data, as the query is randomly.

Kao-Jun a few things to check --

• If by "cached" you mean loaded onto historical for querying purposes, then this should be based on the disk capacity of your historicals, not the RAM or JVM Heap.  What is the total disk capacity of your historicals?
• How much "actual" datasource size do you have?  In the Druid Console Datasources tab, what does it in the "Size" and "Replicated Size" columns?
• In that same tab, what does it say in the "Availability" column for your datasource, both % availability and # of segments?
• If you have a large # segments then you should also confirm your segment structure (size and partitioning scheme) is optimal, as you may be able to reduce segment count and increase compression.  If you ingested via real-time streaming and did not run compaction, there is a chance that you have significant segment fragmentation due to the real-time ingestion jobs.  There are a few things to look at here but relatively easy to spot and clean up.

<@U03LC2ZJ3MK> Just to add to Brandon and John's comments, and to answer your question. 20T of raw data can be stored in a much smaller representation in Druid. If source is JSON formatted, we usually see about 90% compression, but your mileage will vary based on your raw data format and the data's demographics (distinct values and their distribution in time and across segments). So 20T of raw data could potentially fit into 2T of cache, but you will need enough cache space on the historicals to accommodate the segments that result from the ingestion.  It is also normal to have more than 1 replica of each segment on the historicals to provide more parallelism and high availability in the case of loss of a historical or rolling upgrade. So normally it is 2x the compressed size (sometimes more if desirable for higher availability or higher concurrency of queries on the same data).

Thanks so much for your reply.
• What is the total disk capacity of your historicals? 600G
• The "Total data size" is 1.6T, and the Replicated size is 2.4T
• As now I scale the historical to 6 pods, each of them has 500G cache, now it is fully available, and we have 138,745 segements
• For compaction, as most of the segments have 5 million rows, I don't think I have much space to compct.

Hi <@U03LC2ZJ3MK>, this is what I got:
```If replication is 1 you will need 20T across the 3 historicals, so like 6.66T / historical (20/3)```

OK, thanks. Another question is about the query time: I want to query the data using one field:
`SELECT * FROM table WHERE iri='xxx'`
But it is pretty slow now, it takes 300 seconds to return the result. Now the historical pods are launched with  2 cores and 10G memory, does it make sense to increase the cpu and memory if I want to optimize the query time?

Some suggestions :
1. You should do a range partitioning on `iri` with other commonly filtered dimensions
2. Are you using any emitters like Prometheus  ( or Imply Clarity) to determine where the bottleneck is? If segment scanned time is &gt;=250ms and &lt; 500 ms, the historicals are ok in general
3. Broker-wise, you can check the wait time for query and the httpNumThreads available

<@U03LC2ZJ3MK> Is there a timestamp field in your data? If so, the primary partitioning of the data will be on time and adding a time filter to your query will help to reduce the number of segments to process. Additionally secondary partitioning on `iri` as <@U04CGQ1P68Z> suggests would be ideal in either case, if you are using SQL ingestion, this is a `CLUSTERED BY iri` . It can be specified as `range` partitioning in native batch ingestion under the `partitionsSpec`  which will also reduce the segments to process when you use `WHERE iri='xxx'` . Without clustering/partitioning and no time filter, the query would need to process all 138,745 segments.

If you have 6 pods * 2 cores = 12 total for the historicals, this means that a maximum of 12 segments will be processing at any one time, and if each segment scan uses 250ms average, that means 138,745/12 = 2890 seconds.

Given that you are seeing 300 second response, some of my assumptions are wrong, perhaps you are filtering segments with time or the segment scan time is faster given that the automatic index on `iri` will likely quickly eliminate any segments that don't contain the value. The overhead of opening the segments to inspect them will still be at play though.

Let us know how it goes.

BTW.. if you are tight on the cache space, you may need to wipe the data before reingesting it with secondary partitioning applied. If you have room you can use a <https://druid.apache.org/docs/latest/data-management/compaction.html|compaction> job to apply the `partitionsSpec` in the <https://druid.apache.org/docs/latest/ingestion/native-batch.html#tuningconfig|tuningConfig> for the job.

Finally, to answer your question on more cpu and memory, the answer is yes, more processing threads for historicals will increase the parallelism involved in processing the segments to answer the query. So it will help, but you may not need it if the partitioning and query filter match and the segments to process are reduced significantly.

depending on the query you could break it up into multiple queries and fetch say 500 rows each in 6 queries and thus get the 3000 in parallel. Can you paste your query here?

Thanks for your response.

but sometimes the same query returns the result with 1 second. I don't know why it taking more than 1 minute.

here is the query I am executing:-
=========================
data = druid_client.scan(
                            datasource = &lt;table name&gt;,
                            granularity = 'none',
                            intervals = f"{str(start_time)}/{str(end_time)}",
                            context={"timeout": 1000},
                            filter = Dimension('route_id') == route_id)

do you have historical or broker cache enabled?

I think cache is not enabled? But I have asked platform team to confirm it.

Only one roiute_id using in the query.

cache being enabled will cause variation in query times. Beyond that you need to look at how many segments are being scanned by the query and the size of segments to see where the bottle neck is (also are other queries running at the time?).

There is only one segment available. As I have set segment_granularity = Week and Task Duration = PT164H

segment size I defined as 300-400 MB only...I believe there could be some issue related to configuration.
I was checking the druid server CPU/Memoy usage during data fetching. I found that coordinator's cpu usage is high at that time. So it might be impacting us.

Only one segment, and scanning takes over a minute?  I wonder why the coordinator's cpu is high.  What do you have for start_time/end_time?  Is the query better for smaller time intervals?

Caching is enabled at broker and historical level, but still taking longer time to fetch the data.

Start_time = timestamp(date+6:00AM)
end_time = timestamp(date+7:00PM)
Between this time interval, i want to fetch the data from druid.

As I hit the query, the broker might be adding my request to the queue.I'm simply speculating.

Yes, could be that the broker is overwhelmed (or any other node I guess), and needs tuning for your load.

org.apache.druid.java.util.metrics.SysMonitor - Failed to get DiskUsage[/opt/druid/var/druid]
org.hyperic.sigar.SigarFileNotFoundException: No such file or directory
	at org.hyperic.sigar.DiskUsage.gather(Native Method) ~[sigar-1.6.5.132.jar:?]
	at org.hyperic.sigar.DiskUsage.fetch(DiskUsage.java:30) ~[sigar-1.6.5.132.jar:?]
	at org.hyperic.sigar.Sigar.getDiskUsage(Sigar.java:680) ~[sigar-1.6.5.132.jar:?]
	at org.apache.druid.java.util.metrics.SysMonitor$DiskStats.emit(SysMonitor.java:310) ~[druid-core-24.0.0.jar:24.0.0]
	at org.apache.druid.java.util.metrics.SysMonitor.doMonitor(SysMonitor.java:107) ~[druid-core-24.0.0.jar:24.0.0]
	at org.apache.druid.java.util.metrics.AbstractMonitor.monitor(AbstractMonitor.java:53) ~[druid-core-24.0.0.jar:24.0.0]
	at org.apache.druid.java.util.metrics.BasicMonitorScheduler.lambda$startMonitor$0(BasicMonitorScheduler.java:56) ~[druid-core-24.0.0.jar:24.0.0]
	at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$4.run(ScheduledExecutors.java:163) ~[druid-core-24.0.0.jar:24.0.0]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_275]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_275]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_275]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[?:1.8.0_275]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_275]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_275]
	at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_275]

There is actually an open issue item on this – see <https://github.com/apache/druid/issues/10615>

And also this - <https://the-asf.slack.com/archives/CJ8D1JTB8/p1605124961461100?thread_ts=1605121473.460400&amp;cid=CJ8D1JTB8>

you don't need SysMonitor if you are getting machine metrics any other way.

it seems like if the lookup takes long to get registered for some more realtime intervals?

is this kafka ingest and is 2023-02-27T10:54:03Z/2023-02-27T11:24:03Z currentky being ingested?

it is kinesis ingest and yes, it should be ingested however I would need to double check

would it give that exception tho? instead of empty data?

it may be that the lookup is not loaded on the workers. can you look at the kinesis supervisor and task logs and see if there are nay messages about this lookup?

I have couple of lookups that connects to postgresql and from the logs I can see:
```2023-02-27T12:07:01,561 WARN [qtp1679714298-135] org.apache.druid.query.lookup.LookupUtils - Lookup [zip_to_country] could not be serialized properly. Please check its configuration. Error: Cannot construct instance of `org.apache.druid.query.lookup.namespace.JdbcExtractionNamespace`, problem: java.lang.ClassNotFoundException: org.postgresql.Driver
 at [Source: (byte[])":)
��versionW2023-02-23T18:11:40.729Z�lookupExtractorFactory��typeNcachedNamespace�extractionNamespace�BCjdbc�connectorConfig��connectURI�jdbc:<postgresql://test-db.wddrr5tyuj9o.us-east-1.rds.amazonaws.com:5432/analytics>��userJtestuser�password12345�createTables"��keyColumnXzip_key�tableXref_zip_to_country�valueColumnCname�tsColumnNlast_updated_at�pollPeriodBP1D�namespaceTzip_to_country��injective#��"; line: -1, column: 452] (through reference chain: org.apache.druid.query.lookup.LookupExtractorFactoryContainer["lookupExtractorFactory"]-&gt;org.apache.druid.query.lookup.NamespaceLookupExtractorFactory["extractionNamespace"])
2023-02-27T12:07:01,562 WARN [qtp1679714298-135] org.apache.druid.query.lookup.LookupUtils - Lookup [geo_to_city] could not be serialized properly. Please check its configuration. Error: Cannot construct instance of `org.apache.druid.query.lookup.namespace.JdbcExtractionNamespace`, problem: java.lang.ClassNotFoundException: org.postgresql.Driver
 at [Source: (byte[])":)
��versionW2023-02-23T18:08:26.770Z�lookupExtractorFactory��typeNcachedNamespace�extractionNamespace�BCjdbc�namespaceLgeo_to_city�connectorConfig��connectURI�jdbc:<postgresql://db-lookups.cluster-wddrr5tyuj9o.us-east-1.rds.amazonaws.com:5432/lookups>��userJtestuser�password12345��tableLgeo_to_city�keyColumnFuser_id�valueColumnDemail�pollPeriodBP1D�tsColumnAts��firstCacheTimeout��injective#��"; line: -1, column: 388] (through reference chain: org.apache.druid.query.lookup.LookupExtractorFactoryContainer["lookupExtractorFactory"]-&gt;org.apache.druid.query.lookup.NamespaceLookupExtractorFactory["extractionNamespace"])
2023-02-27T12:07:01,563 WARN [qtp1679714298-135] org.apache.druid.query.lookup.LookupUtils - Lookup [test] could not be serialized properly. Please check its configuration. Error: Cannot construct instance of `org.apache.druid.query.lookup.namespace.JdbcExtractionNamespace`, problem: java.lang.ClassNotFoundException: org.postgresql.Driver
 at [Source: (byte[])":)
��versionW2023-02-23T18:09:27.119Z�lookupExtractorFactory��typeNcachedNamespace�extractionNamespace�BCjdbc�namespaceItest�connectorConfig��connectURI�jdbc:<postgresql://test-db.wddrr5tyuj9o.us-east-1.rds.amazonaws.com:5432/analytics>��userJtestuser�password12345��tableSref_test�keyColumnBhex�valueColumnCtext�pollPeriodBP1D��firstCacheTimeout��injective"��"; line: -1, column: 375] (through reference chain: org.apache.druid.query.lookup.LookupExtractorFactoryContainer["lookupExtractorFactory"]-&gt;org.apache.druid.query.lookup.NamespaceLookupExtractorFactory["extractionNamespace"])
2023-02-27T12:07:01,564 WARN [qtp1679714298-135] org.apache.druid.query.lookup.LookupUtils - Lookup [id_to_email] could not be serialized properly. Please check its configuration. Error: Cannot construct instance of `org.apache.druid.query.lookup.namespace.JdbcExtractionNamespace`, problem: java.lang.ClassNotFoundException: org.postgresql.Driver
 at [Source: (byte[])":)
��versionW2023-02-23T18:08:07.423Z�lookupExtractorFactory��typeNcachedNamespace�extractionNamespace�BCjdbc�namespaceTid_to_email�connectorConfig��connectURI�jdbc:<postgresql://db-lookups.cluster-wddrr5tyuj9o.us-east-1.rds.amazonaws.com:5432/lookups>��userJtestuser�password12345��tableTid_to_email�keyColumnIpid�valueColumnDemail�pollPeriodBP1D�tsColumnAts��firstCacheTimeout��injective#��"; line: -1, column: 407] (through reference chain: org.apache.druid.query.lookup.LookupExtractorFactoryContainer["lookupExtractorFactory"]-&gt;org.apache.druid.query.lookup.NamespaceLookupExtractorFactory["extractionNamespace"])
2023-02-27T12:07:01,586 DEBUG [qtp1679714298-135] org.apache.druid.jetty.RequestLog - 10.5.137.163 POST //10.5.149.74:8105/druid/listen/v1/lookups/updates HTTP/1.1 202```
however if I go to lookups section in the unified console, I can see the lookups and fetch the data without errors

do you have the postgresql-metadata-storage extension added in the extension load list for the historicals?

logs are from one random index_kinesis task

and about the extensions:
```{"version":"25.0.0","modules":[{"name":"org.apache.druid.common.aws.AWSModule","artifact":"druid-aws-common","version":"25.0.0"},{"name":"org.apache.druid.common.gcp.GcpModule","artifact":"druid-gcp-common","version":"25.0.0"},{"name":"org.apache.druid.query.aggregation.histogram.ApproximateHistogramDruidModule","artifact":"druid-histogram","version":"25.0.0"},{"name":"org.apache.druid.query.aggregation.datasketches.theta.SketchModule","artifact":"druid-datasketches","version":"25.0.0"},{"name":"org.apache.druid.query.aggregation.datasketches.theta.oldapi.OldApiSketchModule","artifact":"druid-datasketches","version":"25.0.0"},{"name":"org.apache.druid.query.aggregation.datasketches.quantiles.DoublesSketchModule","artifact":"druid-datasketches","version":"25.0.0"},{"name":"org.apache.druid.query.aggregation.datasketches.tuple.ArrayOfDoublesSketchModule","artifact":"druid-datasketches","version":"25.0.0"},{"name":"org.apache.druid.query.aggregation.datasketches.hll.HllSketchModule","artifact":"druid-datasketches","version":"25.0.0"},{"name":"org.apache.druid.query.aggregation.datasketches.kll.KllSketchModule","artifact":"druid-datasketches","version":"25.0.0"},{"name":"org.apache.druid.metadata.storage.postgresql.PostgreSQLMetadataStorageModule","artifact":"postgresql-metadata-storage","version":"25.0.0"},{"name":"org.apache.druid.indexing.kinesis.KinesisIndexingServiceModule","artifact":"druid-kinesis-indexing-service","version":"25.0.0"},{"name":"org.apache.druid.data.input.parquet.ParquetExtensionsModule","artifact":"druid-parquet-extensions","version":"25.0.0"},{"name":"org.apache.druid.security.basic.BasicSecurityDruidModule","artifact":"druid-basic-security","version":"25.0.0"},{"name":"org.apache.druid.storage.s3.output.S3StorageConnectorModule","artifact":"druid-s3-extensions","version":"25.0.0"},{"name":"org.apache.druid.storage.s3.S3StorageDruidModule","artifact":"druid-s3-extensions","version":"25.0.0"},{"name":"org.apache.druid.firehose.s3.S3FirehoseDruidModule","artifact":"druid-s3-extensions","version":"25.0.0"},{"name":"org.apache.druid.data.input.s3.S3InputSourceDruidModule","artifact":"druid-s3-extensions","version":"25.0.0"},{"name":"org.apache.druid.indexing.kafka.KafkaIndexTaskModule","artifact":"druid-kafka-indexing-service","version":"25.0.0"},{"name":"org.apache.druid.emitter.statsd.StatsDEmitterModule","artifact":"statsd-emitter","version":"25.0.0"},{"name":"org.apache.druid.data.input.avro.AvroExtensionsModule","artifact":"druid-avro-extensions","version":"25.0.0"},{"name":"org.apache.druid.server.lookup.namespace.NamespaceExtractionModule","artifact":"druid-lookups-cached-global","version":"25.0.0"}],"memory":{"maxMemory":1572864000,"totalMemory":1572864000,"freeMemory":1035019832,"usedMemory":537844168,"directMemory":134217728}}```

ok...it seems that the worker is not loading the postgres driver.

Can you paste the common run time properties here? Specifically from the node running the worker with the error in the log

gathering them, in the meantime, logs from the peon:
```2023-02-27T12:06:38,482 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [druid-histogram], jars: druid-histogram-25.0.0.jar
2023-02-27T12:06:38,483 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [druid-datasketches], jars: commons-math3-3.6.1.jar, druid-datasketches-25.0.0.jar
2023-02-27T12:06:38,484 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [postgresql-metadata-storage], jars: checker-qual-3.5.0.jar, postgresql-42.4.1.jar, postgresql-metadata-storage-25.0.0.jar
2023-02-27T12:06:38,486 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [druid-kinesis-indexing-service], jars: amazon-kinesis-client-1.14.4.jar, aws-java-sdk-kinesis-1.12.317.jar, aws-java-sdk-sts-1.12.317.jar, commons-lang3-3.12.0.jar, commons-logging-1.1.1.jar, druid-kinesis-indexing-service-25.0.0.jar, guava-16.0.1.jar, jmespath-java-1.12.317.jar, protobuf-java-3.21.7.jar
2023-02-27T12:06:38,489 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [druid-parquet-extensions], jars: audience-annotations-0.12.0.jar, avro-1.9.2.jar, commons-configuration-1.6.jar, druid-parquet-extensions-25.0.0.jar, hadoop-annotations-2.8.5.jar, hadoop-auth-2.8.5.jar, hadoop-common-2.8.5.jar, hadoop-hdfs-client-2.8.5.jar, hadoop-mapreduce-client-core-2.8.5.jar, htrace-core4-4.0.1-incubating.jar, jackson-core-asl-1.9.13.jar, jackson-mapper-asl-1.9.13.jar, javax.annotation-api-1.3.2.jar, parquet-avro-1.12.0.jar, parquet-column-1.12.0.jar, parquet-common-1.12.0.jar, parquet-encoding-1.12.0.jar, parquet-format-structures-1.12.0.jar, parquet-hadoop-1.12.0.jar, parquet-jackson-1.12.0.jar, slf4j-api-1.7.36.jar, snappy-java-1.1.8.4.jar, zstd-jni-1.5.2-3.jar
2023-02-27T12:06:38,494 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [druid-basic-security], jars: druid-basic-security-25.0.0.jar
2023-02-27T12:06:38,495 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [druid-s3-extensions], jars: aws-java-sdk-core-1.12.317.jar, aws-java-sdk-sts-1.12.317.jar, commons-codec-1.13.jar, commons-logging-1.1.1.jar, druid-s3-extensions-25.0.0.jar, httpclient-4.5.13.jar, httpcore-4.4.11.jar, ion-java-1.0.2.jar, jackson-dataformat-cbor-2.10.5.jar, jmespath-java-1.12.317.jar, joda-time-2.10.5.jar
2023-02-27T12:06:38,498 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [druid-kafka-indexing-service], jars: druid-kafka-indexing-service-25.0.0.jar, kafka-clients-3.3.1.jar, lz4-java-1.8.0.jar, snappy-java-1.1.8.4.jar, zstd-jni-1.5.2-3.jar
2023-02-27T12:06:38,500 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [statsd-emitter], jars: asm-9.3.jar, asm-analysis-7.1.jar, asm-commons-9.3.jar, asm-tree-7.1.jar, asm-util-7.1.jar, java-dogstatsd-client-4.0.0.jar, jffi-1.2.23-native.jar, jffi-1.2.23.jar, jnr-a64asm-1.0.0.jar, jnr-constants-0.9.17.jar, jnr-enxio-0.30.jar, jnr-ffi-2.1.16.jar, jnr-posix-3.0.61.jar, jnr-unixsocket-0.36.jar, jnr-x86asm-1.0.2.jar, statsd-emitter-25.0.0.jar
2023-02-27T12:06:38,503 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [druid-avro-extensions], jars: avro-1.9.2.jar, avro-ipc-1.9.2.jar, avro-ipc-jetty-1.9.2.jar, avro-mapred-1.9.2.jar, common-config-5.5.1.jar, common-utils-5.5.1.jar, druid-avro-extensions-25.0.0.jar, gson-2.3.1.jar, jakarta.activation-1.2.1.jar, jakarta.annotation-api-1.3.5.jar, jakarta.inject-2.6.1.jar, javax.annotation-api-1.3.2.jar, jersey-client-1.19.4.jar, jersey-common-2.30.jar, kafka-clients-5.5.1-ccs.jar, kafka-schema-registry-client-5.5.1.jar, lz4-java-1.8.0.jar, osgi-resource-locator-1.0.3.jar, schema-repo-api-0.1.3.jar, schema-repo-avro-0.1.3.jar, schema-repo-client-0.1.3.jar, schema-repo-common-0.1.3.jar, slf4j-api-1.7.36.jar, snappy-java-1.1.8.4.jar, swagger-annotations-1.6.0.jar, velocity-engine-core-2.2.jar, zstd-jni-1.5.2-3.jar
2023-02-27T12:06:38,507 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [druid-lookups-cached-global], jars: druid-lookups-cached-global-25.0.0.jar, mapdb-1.0.8.jar
[1.302s][info   ][gc] GC(5) Pause Young (Normal) (G1 Evacuation Pause) 55M-&gt;20M(768M) 5.588ms
2023-02-27T12:06:38,692 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [druid-histogram], jars: druid-histogram-25.0.0.jar
2023-02-27T12:06:38,694 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [druid-datasketches], jars: commons-math3-3.6.1.jar, druid-datasketches-25.0.0.jar
2023-02-27T12:06:38,700 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [postgresql-metadata-storage], jars: checker-qual-3.5.0.jar, postgresql-42.4.1.jar, postgresql-metadata-storage-25.0.0.jar
2023-02-27T12:06:38,702 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [druid-kinesis-indexing-service], jars: amazon-kinesis-client-1.14.4.jar, aws-java-sdk-kinesis-1.12.317.jar, aws-java-sdk-sts-1.12.317.jar, commons-lang3-3.12.0.jar, commons-logging-1.1.1.jar, druid-kinesis-indexing-service-25.0.0.jar, guava-16.0.1.jar, jmespath-java-1.12.317.jar, protobuf-java-3.21.7.jar
2023-02-27T12:06:38,703 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [druid-parquet-extensions], jars: audience-annotations-0.12.0.jar, avro-1.9.2.jar, commons-configuration-1.6.jar, druid-parquet-extensions-25.0.0.jar, hadoop-annotations-2.8.5.jar, hadoop-auth-2.8.5.jar, hadoop-common-2.8.5.jar, hadoop-hdfs-client-2.8.5.jar, hadoop-mapreduce-client-core-2.8.5.jar, htrace-core4-4.0.1-incubating.jar, jackson-core-asl-1.9.13.jar, jackson-mapper-asl-1.9.13.jar, javax.annotation-api-1.3.2.jar, parquet-avro-1.12.0.jar, parquet-column-1.12.0.jar, parquet-common-1.12.0.jar, parquet-encoding-1.12.0.jar, parquet-format-structures-1.12.0.jar, parquet-hadoop-1.12.0.jar, parquet-jackson-1.12.0.jar, slf4j-api-1.7.36.jar, snappy-java-1.1.8.4.jar, zstd-jni-1.5.2-3.jar
2023-02-27T12:06:38,705 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [druid-basic-security], jars: druid-basic-security-25.0.0.jar
2023-02-27T12:06:38,706 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [druid-s3-extensions], jars: aws-java-sdk-core-1.12.317.jar, aws-java-sdk-sts-1.12.317.jar, commons-codec-1.13.jar, commons-logging-1.1.1.jar, druid-s3-extensions-25.0.0.jar, httpclient-4.5.13.jar, httpcore-4.4.11.jar, ion-java-1.0.2.jar, jackson-dataformat-cbor-2.10.5.jar, jmespath-java-1.12.317.jar, joda-time-2.10.5.jar
2023-02-27T12:06:38,708 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [druid-kafka-indexing-service], jars: druid-kafka-indexing-service-25.0.0.jar, kafka-clients-3.3.1.jar, lz4-java-1.8.0.jar, snappy-java-1.1.8.4.jar, zstd-jni-1.5.2-3.jar
2023-02-27T12:06:38,710 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [statsd-emitter], jars: asm-9.3.jar, asm-analysis-7.1.jar, asm-commons-9.3.jar, asm-tree-7.1.jar, asm-util-7.1.jar, java-dogstatsd-client-4.0.0.jar, jffi-1.2.23-native.jar, jffi-1.2.23.jar, jnr-a64asm-1.0.0.jar, jnr-constants-0.9.17.jar, jnr-enxio-0.30.jar, jnr-ffi-2.1.16.jar, jnr-posix-3.0.61.jar, jnr-unixsocket-0.36.jar, jnr-x86asm-1.0.2.jar, statsd-emitter-25.0.0.jar
2023-02-27T12:06:38,711 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [druid-avro-extensions], jars: avro-1.9.2.jar, avro-ipc-1.9.2.jar, avro-ipc-jetty-1.9.2.jar, avro-mapred-1.9.2.jar, common-config-5.5.1.jar, common-utils-5.5.1.jar, druid-avro-extensions-25.0.0.jar, gson-2.3.1.jar, jakarta.activation-1.2.1.jar, jakarta.annotation-api-1.3.5.jar, jakarta.inject-2.6.1.jar, javax.annotation-api-1.3.2.jar, jersey-client-1.19.4.jar, jersey-common-2.30.jar, kafka-clients-5.5.1-ccs.jar, kafka-schema-registry-client-5.5.1.jar, lz4-java-1.8.0.jar, osgi-resource-locator-1.0.3.jar, schema-repo-api-0.1.3.jar, schema-repo-avro-0.1.3.jar, schema-repo-client-0.1.3.jar, schema-repo-common-0.1.3.jar, slf4j-api-1.7.36.jar, snappy-java-1.1.8.4.jar, swagger-annotations-1.6.0.jar, velocity-engine-core-2.2.jar, zstd-jni-1.5.2-3.jar
2023-02-27T12:06:38,712 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [druid-lookups-cached-global], jars: druid-lookups-cached-global-25.0.0.jar, mapdb-1.0.8.jar```

I see the following in the docs
```If using JDBC, you will need to add your database's client JAR files to the extension's directory. For Postgres, the connector JAR is already included. See the MySQL extension documentation for instructions to obtain MySQL or MariaDB connector libraries. The connector JAR should reside in the classpath of Druid's main class loader. To add the connector JAR to the classpath, you can copy the downloaded file to lib/ under the distribution root directory. Alternatively, create a symbolic link to the connector in the lib directory.```
can you copy the postgresql-42.4.1.jar from extesnions/postgresql-metadata-storage to lib and check?

hmm I think I know where the problem is, I have deployed it in k8s by using pulumi and I have following transformation to copy the jar:
```// Copy postgresql driver to lib
                if (obj.kind === 'Deployment' &amp;&amp; (
                    obj.metadata.name === 'druid-' + clusterName + '-coordinator' ||
                    obj.metadata.name === 'druid-' + clusterName + '-router' ||
                    obj.metadata.name === 'druid-' + clusterName + '-broker'
                )) {
                    const lifecycle: k8sInputs.core.v1.Lifecycle = {
                        postStart: {
                            exec: {
                                command: [
                                    '/bin/sh',
                                    '-c',
                                    'cp /opt/druid/extensions/postgresql-metadata-storage/postgresql-42.4.1.jar /opt/druid/lib/ &amp;&amp; cd -',
                                ]
                            }
                        }
                    }
                    obj.spec.template.spec.containers[0].lifecycle = lifecycle
                }```
it looks like I only copied to deployments and thats why it works in druid unified console but it does not in middlemanager because they are StatefulSets

however not sure why it worked for some other intervals..

if the interval has no data in the worker (data already in historical) then the query won't go to the worker at all

all right so thats why for some other intervals will work

updated statefulsets and now works perfectly

it looks like I keep getting following errors in broker when queries sent via API to druid
```2023-02-28T13:01:12,511 INFO [qtp1333929103-165[groupBy_[test_data]_7013cfd7-0a11-42d0-b859-bf655a6f46c1]] org.apache.druid.server.log.LoggingRequestLogger - 2023-02-28T13:01:12.450Z    10.5.189.1    {"queryType":"groupBy","dataSource":{"type":"table","name":"test_data"},"intervals":{"type":"LegacySegmentSpec","intervals":["2023-02-28T12:31:10.000Z/2023-02-28T13:01:10.000Z"]},"filter":{"type":"and","fields":[{"type":"selector","dimension":"id","value":"12345"}]},"granularity":{"type":"all"},"dimensions":[{"type":"default","dimension":"to","outputName":"address","outputType":"STRING"},{"type":"default","dimension":"net","outputName":"net","outputType":"STRING"},{"type":"lookup","dimension":"test","outputName":"test","lookup":null,"retainMissingValue":false,"replaceMissingValueWith":"unknown test","name":"test_lookup","optimize":true}],"aggregations":[{"type":"doubleSum","name":"count","fieldName":"count"}],"limitSpec":{"type":"default","columns":[{"dimension":"address","direction":"descending","dimensionOrder":{"type":"lexicographic"}}],"limit":100},"context":{"priority":"100","queryId":"7013cfd7-0a11-42d0-b859-bf655a6f46c1","sqlQueryId":"3267738187128657978_timeseries_count_all_30m"}}    {"query/time":61,"query/bytes":-1,"success":false,"identity":"test_analytics","exception":"QueryInterruptedException{msg=Lookup [test_lookup] not found, code=Unknown exception, class=org.apache.druid.java.util.common.ISE, host=10.5.138.26:8106}","interrupted":true,"reason":"QueryInterruptedException{msg=Lookup [test_lookup] not found, code=Unknown exception, class=org.apache.druid.java.util.common.ISE, host=10.5.138.26:8106}"}```
the lookup is def there, and some queries work and other I got above error

it is hard to find the pattern here, looks like it happens for queries fetching near realtime data, but not sure 100%

Could be related to druid.lookup.enableLookupSyncOnStartup, maybe druid.manager.lookups.threadPoolSize (see druid.manager.lookups.threadPoolSize)

I do not have any of them configured in my settings, I guess then it is using default values

I am seeing this in my task logs, `2023-02-25T17:50:52,118 INFO [main] org.apache.druid.cli.CliPeon - * druid.lookup.enableLookupSyncOnStartup: false` so I guess that might be?

I think `druid.manager.lookups.threadPoolSize` should be probably set to 2 as default

turning on enableLookupSyncOnStartup could help, I think it's worth trying.  From what you describe, I thought a bit more and threadPoolSize is less likely.  (It controls how many background threads are available to communicate with the servers that load lookups to tell them when new versions are available and push out lookup changes to these nodes.)  Whereas enableLookupSyncOnStartup tells peons to load the lookups right away when they start up, iiuc.

Will that help with queries? Because the problem I'm seeing is with queries and the error log comes from coordinator

If you're seeing it for the near-real-time queries, I think it could help.  You'd want to enable it on MMs so the peons get the setting.

You applied, and restarted MMs (which should stop and restart any ongoing tasks)?  Is this constant, or occurs at times (like when tasks start?

I have not applied it yet. It looks like it is not constant, it happens from time to time, and only affecting queries. It does not affect ingestion

Oh, I misunderstood.  I think there's a good chance it would help.

I finally applied the changes however I do not think they are causing any effect. The error I get in coordinators:
```2023-03-01T13:19:34,040 WARN [qtp1333929103-159[groupBy_[test]_922f46e1-ec95-4042-8588-19776e45eb62]] org.apache.druid.server.QueryLifecycle - Exception while processing queryId [922f46e1-ec95-4042-8588-19776e45eb62] (QueryInterruptedException{msg=Lookup [sigs_test] not found, code=Unknown exception, class=org.apache.druid.java.util.common.ISE, host=10.5.168.46:8102})
2023-03-01T13:19:34,040 INFO [qtp1333929103-159[groupBy_[test]_922f46e1-ec95-4042-8588-19776e45eb62]] org.apache.druid.server.log.LoggingRequestLogger - 2023-03-01T13:19:33.984Z    10.5.100.235    {"queryType":"groupBy","dataSource":{"type":"table","name":"test"},"intervals":{"type":"LegacySegmentSpec","intervals":["2023-03-01T12:49:33.000Z/2023-03-01T13:19:33.000Z"]},"filter":{"type":"and","fields":[{"type":"selector","dimension":"id","value":"1234abcd"}]},"granularity":{"type":"all"},"dimensions":[{"type":"default","dimension":"addr","outputName":"address","outputType":"STRING"},{"type":"default","dimension":"code","outputName":"code","outputType":"STRING"},{"type":"lookup","dimension":"signature","outputName":"signature","lookup":null,"retainMissingValue":false,"replaceMissingValueWith":"unknown signature","name":"sigs_test","optimize":true}],"aggregations":[{"type":"doubleSum","name":"count","fieldName":"count"}],"limitSpec":{"type":"default","columns":[{"dimension":"address","direction":"descending","dimensionOrder":{"type":"lexicographic"}}],"limit":100},"context":{"priority":"100","queryId":"922f46e1-ec95-4042-8588-19776e45eb62","sqlQueryId":"222550709180619544_timeseries_count_all_30m"}}    {"query/time":56,"query/bytes":-1,"success":false,"identity":"analytics_user","exception":"QueryInterruptedException{msg=Lookup [sigs_test] not found, code=Unknown exception, class=org.apache.druid.java.util.common.ISE, host=10.5.168.46:8102}","interrupted":true,"reason":"QueryInterruptedException{msg=Lookup [sigs_test] not found, code=Unknown exception, class=org.apache.druid.java.util.common.ISE, host=10.5.168.46:8102}"}
2023-03-01T13:19:34,040 DEBUG [qtp1333929103-159] org.apache.druid.jetty.RequestLog - 10.5.100.235 POST //10.5.152.192:8082/druid/v2 HTTP/1.1 500
2023-03-01T13:19:34,847 WARN [qtp1333929103-137[groupBy_[test]_c71b4679-8e68-4ee3-a725-f1844aacbe0f]] org.apache.druid.client.JsonParserIterator - Query [c71b4679-8e68-4ee3-a725-f1844aacbe0f] to host [10.5.168.46:8102] interrupted
org.apache.druid.query.QueryException: Lookup [sigs_test] not found
at jdk.internal.reflect.GeneratedConstructorAccessor136.newInstance(Unknown Source) ~[?:?]
at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:490) ~[?:?]
at com.fasterxml.jackson.databind.introspect.AnnotatedConstructor.call(AnnotatedConstructor.java:124) ~[jackson-databind-2.10.5.1.jar:2.10.5.1]
at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.createFromObjectWith(StdValueInstantiator.java:283) ~[jackson-databind-2.10.5.1.jar:2.10.5.1]
at com.fasterxml.jackson.databind.deser.ValueInstantiator.createFromObjectWith(ValueInstantiator.java:229) ~[jackson-databind-2.10.5.1.jar:2.10.5.1]
at com.fasterxml.jackson.databind.deser.impl.PropertyBasedCreator.build(PropertyBasedCreator.java:198) ~[jackson-databind-2.10.5.1.jar:2.10.5.1]
at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:422) ~[jackson-databind-2.10.5.1.jar:2.10.5.1]
at com.fasterxml.jackson.databind.deser.std.ThrowableDeserializer.deserializeFromObject(ThrowableDeserializer.java:65) ~[jackson-databind-2.10.5.1.jar:2.10.5.1]
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:159) ~[jackson-databind-2.10.5.1.jar:2.10.5.1]
at com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:4189) ~[jackson-databind-2.10.5.1.jar:2.10.5.1]
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2476) ~[jackson-databind-2.10.5.1.jar:2.10.5.1]
at org.apache.druid.client.JsonParserIterator.init(JsonParserIterator.java:183) ~[druid-server-25.0.0.jar:25.0.0]
at org.apache.druid.client.JsonParserIterator.hasNext(JsonParserIterator.java:93) ~[druid-server-25.0.0.jar:25.0.0]
at org.apache.druid.java.util.common.guava.BaseSequence.toYielder(BaseSequence.java:70) ~[druid-core-25.0.0.jar:25.0.0]
at org.apache.druid.java.util.common.guava.BaseSequence.accumulate(BaseSequence.java:44) ~[druid-core-25.0.0.jar:25.0.0]
at org.apache.druid.java.util.common.guava.MergeSequence.toYielder(MergeSequence.java:63) ~[druid-core-25.0.0.jar:25.0.0]
at org.apache.druid.query.RetryQueryRunner$1.toYielder(RetryQueryRunner.java:133) ~[druid-server-25.0.0.jar:25.0.0]
at org.apache.druid.java.util.common.guava.YieldingSequenceBase.accumulate(YieldingSequenceBase.java:35) ~[druid-core-25.0.0.jar:25.0.0]
at org.apache.druid.common.guava.CombiningSequence.accumulate(CombiningSequence.java:62) ~[druid-core-25.0.0.jar:25.0.0]
at org.apache.druid.java.util.common.guava.TopNSequence$1.make(TopNSequence.java:54) ~[druid-core-25.0.0.jar:25.0.0]
at org.apache.druid.java.util.common.guava.BaseSequence.toYielder(BaseSequence.java:66) ~[druid-core-25.0.0.jar:25.0.0]
at org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:88) ~[druid-core-25.0.0.jar:25.0.0]
at org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:84) ~[druid-core-25.0.0.jar:25.0.0]
at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-core-25.0.0.jar:25.0.0]
...
at java.lang.Thread.run(Thread.java:829) ~[?:?]
2023-03-01T13:19:34,849 WARN [qtp1333929103-137[groupBy_[test]_c71b4679-8e68-4ee3-a725-f1844aacbe0f]] org.apache.druid.server.QueryLifecycle - Exception while processing queryId [c71b4679-8e68-4ee3-a725-f1844aacbe0f] (QueryInterruptedException{msg=Lookup [sigs_test] not found, code=Unknown exception, class=org.apache.druid.java.util.common.ISE, host=10.5.168.46:8102})
2023-03-01T13:19:34,849 INFO [qtp1333929103-137[groupBy_[test]_c71b4679-8e68-4ee3-a725-f1844aacbe0f]] org.apache.druid.server.log.LoggingRequestLogger - 2023-03-01T13:19:34.786Z    10.5.152.134    {"queryType":"groupBy","dataSource":{"type":"table","name":"test"},"intervals":{"type":"LegacySegmentSpec","intervals":["2023-03-01T12:49:33.000Z/2023-03-01T13:19:33.000Z"]},"filter":{"type":"and","fields":[{"type":"selector","dimension":"id","value":"1234abcd"}]},"granularity":{"type":"all"},"dimensions":[{"type":"default","dimension":"addr","outputName":"address","outputType":"STRING"},{"type":"default","dimension":"code","outputName":"code","outputType":"STRING"},{"type":"lookup","dimension":"signature","outputName":"signature","lookup":null,"retainMissingValue":false,"replaceMissingValueWith":"unknown signature","name":"sigs_test","optimize":true}],"aggregations":[{"type":"doubleSum","name":"count","fieldName":"count"}],"limitSpec":{"type":"default","columns":[{"dimension":"address","direction":"descending","dimensionOrder":{"type":"lexicographic"}}],"limit":100},"context":{"priority":"100","queryId":"c71b4679-8e68-4ee3-a725-f1844aacbe0f","sqlQueryId":"222550709180619544_timeseries_count_all_30m"}}    {"query/time":63,"query/bytes":-1,"success":false,"identity":"analytics_user","exception":"QueryInterruptedException{msg=Lookup [sigs_test] not found, code=Unknown exception, class=org.apache.druid.java.util.common.ISE, host=10.5.168.46:8102}","interrupted":true,"reason":"QueryInterruptedException{msg=Lookup [sigs_test] not found, code=Unknown exception, class=org.apache.druid.java.util.common.ISE, host=10.5.168.46:8102}"}
2023-03-01T13:19:34,849 DEBUG [qtp1333929103-137] org.apache.druid.jetty.RequestLog - 10.5.152.134 POST //10.5.152.192:8082/druid/v2 HTTP/1.1 500
2023-03-01T13:19:35,021 DEBUG [qtp1333929103-169] org.apache.druid.jetty.RequestLog - 10.5.147.124 GET //10.5.152.192:8082/status/health HTTP/1.1 200
2023-03-01T13:19:35,070 DEBUG [qtp1333929103-160] org.apache.druid.jetty.RequestLog - 10.5.147.124 GET //10.5.152.192:8082/status/health HTTP/1.1 200
2023-03-01T13:19:35,749 INFO [NodeRoleWatcher[PEON]] org.apache.druid.discovery.BaseNodeRoleWatcher - Node [<http://10.5.146.8:8106>] of role [peon] went offline.
2023-03-01T13:19:35,749 INFO [CuratorDruidNodeDiscoveryProvider-ListenerExecutor] org.apache.druid.client.HttpServerInventoryView - Server[10.5.146.8:8106] disappeared.
2023-03-01T13:19:35,749 INFO [CuratorDruidNodeDiscoveryProvider-ListenerExecutor] org.apache.druid.server.coordination.ChangeRequestHttpSyncer - Stopping ChangeRequestHttpSyncer[<http://10.5.146.8:8106/_1677676721376>].
2023-03-01T13:19:35,749 INFO [CuratorDruidNodeDiscoveryProvider-ListenerExecutor] org.apache.druid.server.coordination.ChangeRequestHttpSyncer - Stopped ChangeRequestHttpSyncer[<http://10.5.146.8:8106/_1677676721376>].
2023-03-01T13:19:35,767 INFO [FilteredHttpServerInventoryView-0] org.apache.druid.server.coordination.ChangeRequestHttpSyncer - Skipping sync() success for server[<http://10.5.146.8:8106/_1677676721376>].
2023-03-01T13:19:35,887 INFO [NodeRoleWatcher[PEON]] org.apache.druid.discovery.BaseNodeRoleWatcher - Node [<http://10.5.174.204:8101>] of role [peon] went offline.```

it is fetched from a postgresql, not sure how many rows it fetches into memory but the table itself has around 200k entries, I do not consider that a lot for a key/value map pair

true. Is this error always happening with queries accessing the data in the peon and not the data in the historical?

I think so, I think the error happens when near realtime and for this particular query, it always looks at last 30 minutes of current time. It is exposed through a public facing API so when users visit their stats dashboard a query is launched to see last 30 minutes

I made a rollout restart and I think this time the parameter has been picked up correctly. Have not seen many error but I keep monitoring it

I don't know if you've seen this, or it's helpful - <https://druid.apache.org/docs/latest/querying/nested-columns.html>

When the coordinator is restarted, it takes some time for it to learn about all the segments in metadata and various other things. Depending on how many segments you have in your metadata, this can last for tens of minutes (up to an hour if you're talking millions of segments).

If you enable debug logging, you can see this happening in the logs...

In that mean, will the ingestion and other application queries work proper?

FYI, in Druid 25 there were changes made to the console so it works better when the coordinator is being restarted

okay, but my point is, will there be any impact during the UI is stuck ?

if the time period is not too long (a few minutes) everything will be 100% fine. if it's very long (tens of minutes/hours) then you may have issues with delayed handoff from ingest tasks, and delayed rebalancing of historicals (in the case where rebalancing is needed).

we're working on improving these latter cases for very large clusters with very large numbers of segments; if you're in these cases let us know!

we may be able to suggest fixes to any particular issues you're having, and we're also always interested in feedback from people that run very large clusters

<@U0344FW86DD>, I am stuck with a different issue, where Druid is unaware of HA on deep storage HDFS, when ever we switch namenode, we are doing change in middle managers to change /etc/hosts with name node as current active.. Is there a better way to handle this?

hmm, do you have HA configured in `hdfs-site.xml` for your hadoop install generally?

if so you could put this in your `_common` druid config directory

Hi Santhosh - you should not need to do any scripting - `kill` tasks will also remove the files from deep storage.
From: <https://druid.apache.org/docs/latest/tutorials/tutorial-delete-data.html#run-a-kill-task>
&gt; we can submit a Kill Task, which will delete the disabled segments from metadata and *deep storage.*

Also - have a look at `druid.coordinator.kill` from <https://druid.apache.org/docs/latest/configuration/index.html#coordinator-operation>

Thanks!, how can I have a cronjob like thing to remove older than an year or so segments?

Druid uses Load Rules to configure retention policies - this can all be done automatically (without scripting) <https://druid.apache.org/docs/latest/operations/rule-configuration.html#period-load-rule>

<@U04B7RRGYN6> Hi. You should be able to jump directly to 25.0. But please take a look at the release notes for each intervening release to see if there are any changes to functionality that will affect you. One notable thing that has come up recently is that the default for interprocess communication has changed from Zookeeper based to HTTP based. This causes more activity on the historical, broker and middle manager HTTP thread pool, so you'll probably want to increase that a bit.

The release notes are:
<https://github.com/apache/druid/issues/11657|[Draft] 0.22.0 Release Notes>

<https://github.com/apache/druid/issues/12054|[Draft] 0.22.1 Release Notes>
<https://github.com/apache/druid/issues/12510|[Draft] 0.23.0 Release notes>
<https://github.com/apache/druid/issues/12825|[Draft] 24.0 Release notes>
<https://github.com/apache/druid/issues/13592|[Draft] 25.0.0 Release Notes>

Thanks, I was ofc reading through the release notes and except for the zk version dependency bump didn't see any blocker, just wanted to make sure, as this is a production system upgrade.

As always, doing the test of the upgrade on test/dev would be good. But I'm sure you know this.

what is the requirement? Can you paste a sample query here?

<@U03BJAMQ4H5>
like
row 1 … queryId = a, query = b
row 2 … queryId = a, query = null

groupBy queryId and get first non null value in this case b

maybe Earliest() ? might be answer?

what are you ordering on? You could use a having clause to filter out nulls in the inner query and then pick one row from that result in the outer query

<@U03BJAMQ4H5> there is case that
row 1 … queryId = a, query = b, colA= null
row 2 … queryId = a, query = null, colA=test
in this case i need to get value from row1 and row2

should the output be one row a,b,test or two separate rows?

```elect col1,earliest(col2,16) filter (where col2&lt;&gt;'null'),EARLIEST(col3,16) filter (where col3 &lt;&gt;'null') from test2  group by col1```

<@U03BJAMQ4H5> filter(where col2 is not null)
Solver the problem and got table what i want thank you

I got the solutions, seems like I would need to enable push gateway for prometheus as peon tasks are short lived.

That's right, peons can't be scraped, they need to push their metrics

You can take a look at the requests and responses that the UI is submitting and see the errors in the Developer view on the browser (F12 on Chrome). Here's what I see in terms of listing the services:

the response will likely shed some light on why it isn't working for you.

The SQL query the UI uses to retrieve information about running the instances under Services returns `is_leader: 0` for all the coordinators, which contradicts what the coordinators return via ReST. Is that stored on ZooKeeper or PostgreSQL?

Actually, neither.  It goes against sys.servers, which is a system table.  Those exist in pieces of code on each server, and get populated and maintained in memory.  I don't know what the API does, and why it works.  (Or why the system tables are wrong.)  I can't think of a way to pin this down atm for a good bug report, but if you want to fix it, it's possible that restarting things might help.

Good to know about the "sys.servers" table.

I tried roll restarting the coordinators (I'm running on K8s), but that didn't work.

I just tried deleting the Pod that was the leader, and I saw the following on the "new" leader:
```2023-02-28T19:52:25,886 INFO [LeaderSelector[/druid/coordinator/_COORDINATOR]] org.apache.druid.server.coordinator.DruidCoordinator - I am the leader of the coordinators, all must bow! Starting coordination in [PT10S].
2023-02-28T19:52:25,908 INFO [LeaderSelector[/druid/coordinator/_COORDINATOR]] org.apache.druid.curator.discovery.CuratorServiceAnnouncer - Announcing service[DruidNode{serviceName='druid/coordinator', host='10.244.34.4', bindOnHost=false, port=-1, plaintextPort=8081, enablePlaintextPort=true, tlsPort=-1, enableTlsPort=false}]
2023-02-28T19:52:25,914 INFO [LeaderSelector[/druid/overlord/_OVERLORD]] org.apache.druid.indexing.overlord.TaskMaster - By the power of Grayskull, I have the power!```
So, leader election happens but this "sys.servers" table is not updated correctly, and the logs have no errors.

Coordinators seem OK, I thought maybe restarting the router might help?

I tried that, but unfortunately didn't help.

could this be a permission issue on the Metadata DB?

<@U04NGB9388K> when you say "router cannot detect the coordinator/overlord leader for some reason" what do you mean exactly? (like what API or UI error are you getting?)

I don't think it'd be related to the metadata db. Possibly something related to leader election; although it'd be odd for the server to be discovered and leader election not also work.

<@U04NGB9388K> fyi the `is_leader` in `sys.segments` for coordinator is populated by calling `/druid/coordinator/v1/leader` on any one of the coordinators and comparing it to the host/port advertised by each coordinator

Hello <@U0344FW86DD> The SQL query that the Router uses to retrieve the information from `sys.servers` returns 0 for the `is_leader` column for all the coordinators. However, the endpoint you mentioned returns the correct result (`/druid/coordinator/v1/leader`). Leader election works (I tested it). That's why I don't understand what's happening.

Today, I made some changes to the Indexers and Historicals and roll-restarted their StatefulSets, and suddenly, now the leaders appear correctly. Not sure how this influenced something managed by Coordinators and Overlords.

It's good that it is working, but still very weird. <@U0344FW86DD> what process is in charge of updating the sys.servers table?

Perhaps there is a log entry there that explains this.

the Broker updates it based on its opinion of who the leader is: I wonder if the Broker lost the thread?

An admin, <@U030W50CNAC>, removed *Natural Intelligence* from this channel.

I think that would happen if you have a line starting with `["druid-histogram", "druid-datasketches", "druid-lookups-cached-global", "postgresql-metadata-storage", "druid-multi-stage-query"]`

Rather than `druid_extensions_loadList=["druid-histogram", "druid-datasketches", "druid-lookups-cached-global", "postgresql-metadata-storage", "druid-multi-stage-query"]`

Hey thanks for the help and sorry for the slow reply. I've been supplying an env var to the container:
```druid_indexer_runner_javaOptsArray=["-server", "-Xmx1g", "-Xms1g", "-XX:MaxDirectMemorySize=3g", "-Duser.timezone=UTC", "-Dfile.encoding=UTF-8", "-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager"]```
and
```druid_extensions_loadList=["druid-histogram", "druid-datasketches", "druid-lookups-cached-global", "postgresql-metadata-storage", "druid-multi-stage-query", "druid-kafka-indexing-service"]```
I'm fairly sure they are being provided to the containers correctly.

It's potentially interesting to note that for the `..._javaOptsArray` variable if you remove the `-Duser.` and the `-DFile.` options it doesn't throw the parse error. It seems to throw the error always when it comes to the `loadList` variable tough.

Maybe I'm not doing things correctly but locally this setup is fine when you run it under docker-compose following the appropriate docs. Puzzling me at the moment

hmm. i wonder if something, somewhere is splitting these lines?

Yeah I'm pretty stumped to be honest. These can be set instead in a config file and loaded that way though? I think... I'll need to go back and reorganise how I'm doing that config then

Okay so I'm not sure how to even debug more into this unfortunately. I've looked in the container and the env var seems to be set okay:
```bash-5.1$ echo $druid_indexer_runner_javaOptsArray
["-server", "-Xmx1g", "-Xms1g", "-XX:MaxDirectMemorySize=3g", "-Duser.timezone=UTC", "-Dfile.encoding=UTF-8", "-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager"]
bash-5.1$ echo $druid_extensions_loadList
["druid-histogram", "druid-datasketches", "druid-lookups-cached-global", "postgresql-metadata-storage", "druid-multi-stage-query", "druid-kafka-indexing-service"]
```
I only see the following log output:
```2023-03-12T23:21:47+00:00 startup service
Setting 25.0.1.126= in /tmp/conf/druid/cluster/misc/druid.host/runtime.properties
sed: bad regex '["-server", "-Xmx1g", "-Xms1g", "-XX:MaxDirectMemorySize=3g", "-Duser.timezone=UTC", "-Dfile.encoding=UTF-8", "-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager"]=': Invalid range end
```
I reread over the docs and it seems that using env vars to set the config state is the preferred way of doing things - just that then it comes to production you may wish to split up your environment files into service specific ones.

It works fine locally with docker so I don't know what's causing these weird issues. I guess the only option is to try the more production approach of having the `DRUID_COMMON_CONFIG` set to a custom config file, although that's going to have a bit more friction when testing just now.

`Setting 25.0.1.126=` looks like values are getting confused for keys

I suspect the same thing is happening with the loadList; since the `sed` error you get is from a line of code in a script that is trying to deal with a key

I am not sure why this is happening, but perhaps that hint will be helpful…

Thanks for your help on this, it's much appreciated. I think I'll maybe do a little more debugging - might try switching the druid version and such. After that I'm just going to have to try the `DRUID_COMMON_CONFIG` approach :man-shrugging:

wish I could help more, but I am not super familiar with this way of running druid!

I mean perhaps I'm going about things in a crazy way? I've just been attempting to transition the local docker based setup which works nicely into a cloud based container deploy. Didn't seem too crazy in my head but I'm new to druid so perhaps there's an easier way to go about this

if you're using k8s try checking out <#C04Q0047B4M|>, there's a bunch of people there using docker via k8s in production

Okay so for the record I'm a fool :joy: Turns out it wasn't executing the `druid.sh` with any service name argument. Hence why everything was being put in `misc` as seen in `/tmp/conf/druid/cluster/misc/`. The lesson here is don't try to wrangle new software *and* a new orchestration tool at the same time.

oof, i guess that's because `SERVICE="$1"` at the top of `druid.sh`

i wonder if the script can be improved to make that kind of error more obvious!

Looking at the docs inside that file:
```# It takes one required argument (the name of the service,
# e.g. 'broker', 'historical' etc). Any additional arguments
# are passed to that service.```
I guess if an argument was strictly required then I'd expect it to error out if one wasn't provided. Maybe people have found a use for running it without one though? Perhaps even a warning message when the first argument is not a valid service name would be a possible improvement.

Are any historicals loading segments in the console's Services view?  Anything in historical logs?  Are any of the historicals at 100%?

seems odd; datasource availability is based on the % of segments are unavailable. if it's 100% then there would not be any unavailable segments

what API / UI control are you looking at when you see the 100% availability and the unavailable segments?

perhaps knowing what you're looking at will help us figure out what's going on

image.png

apparently it's not always necessarily a problem: <https://apachedruidworkspace.slack.com/archives/C030K0Z7S1Z/p1678196501062679>

although, it's definitely odd for this condition to persist for a while. ordinarily such segments are marked unused relatively promptly (within a few minutes, an hour at most)

if the condition still persists i'd try checking for info on the unavailable ones using `sys.segments`. perhaps pick one datasource and one time range (a specific `"start"` value) and see what's going on there

what benefit are you hoping to achieve by having a multi geo Druid cluster?

I wish to ingest data from multiple zones and query from central zone so that data flow cost is avoided during query time

Like master and middle manager resides in respective country zone having common MySQL for metadata and s3 for segment 

probably what makes the most sense is to have multi-zone kafka (if you are using kafka for ingest) and single zone druid. typically that's what people do when collecting data from lots of different places into a single location for analytics

if you need multi-zone druid for DR, you can have the two druid zones read the same kafka data independently (i.e. no shared druid infra)

also what <@U030W50CNAC> said here (<https://apachedruidworkspace.slack.com/archives/C0309C9L90D/p1677635539788419>)

if the above suggestion doesn't work for you, it helps to know what you are hoping to gain?

Are you referring to Availability Zones?  Or Regions?  If the latter, then will something like Decodables work here?

@can I have same s3 bucket for multiple independent Druid

According to the documentation, if you specify a different _*druid.storage.baseKey*_ value for each cluster, then the files in the same S3 bucket will be kept separate from each other.

ref: <https://druid.apache.org/docs/latest/development/extensions-core/s3.html#deep-storage>

I do not know if there are any other common files that might conflict if multiple clusters tried to use the same S3 bucket ... but the docs do not mention any restrictions here.

If you can set environment variable in your MiddleManager nodes, you can use <https://druid.apache.org/docs/latest/operations/dynamic-config-provider.html#environment-variable-dynamic-config-provider|Dynamic Config Providers> to set the rack in your supervisor like:
```{
  "type": "kafka",
  "spec": {
    "ioConfig": {
      "type": "kafka",
      "consumerProperties": {
        "bootstrap.servers": "ignoreServer",
        "druid.dynamic.config.provider": {
          "type": "environment",
          "variables": {
            "bootstrap.servers": "KAFKA_SERVER",
            "client.rack": "CONSUMER_CLIENT_RACK"
          }
        }
      },
      "topic": "thingy"
    },
    "tuningConfig": {
      "type": "kafka"
    }
  }
}```
Here `KAFKA_SERVER` and `CONSUMER_CLIENT_RACK` are environment variables.  <@U030MMVU35Z> - anything to add as I gained this knowledge from you :wink:

Thank you <@U030MBK46BD>. The core elements are already covered in your comment. One more thing to add here is that the client rack environment variable needs to be available in the process that runs the ingestion. I usually add it in the script that starts the druid process itself. I am currently working on a blog post for this that should come out in the next few weeks.

Thanks <@U030MBK46BD>.

Hi <@U030MMVU35Z> ,
I didn't understand your statement.
```client rack environment variable needs to be available in the process that runs the ingestion. I usually add it in the script that starts the druid process itself.```
Please enlighten me on this.
I'm new to Druid Kafka Ingestion.

Hi <@U03C7CHRS9E> thank you for reaching out. As you can see from the dynamic config provided by <@U030MBK46BD> the client rack information is `"client.rack": "CONSUMER_CLIENT_RACK"` . Here : `CONSUMER_CLIENT_RACK`  is an environment variable that you need to set on the process that runs the kafka consumer. In my case I wrote a user init script that runs when every node starts and added this there:
```#!/bin/bash

# And add the environment variable before starting the Imply processes
sed -i '/^exec.*/i export\ CONSUMER_CLIENT_RACK=`curl -s <http://169.254.169.254/latest/meta-data/placement/availability-zone>` ' $(find /opt/grove -name run-druid | grep -v dist)```
This works for druid setups that are running on linux and gets initialised by a run-druid setup. For druid setups that use something else you will need to find the script that actually starts the druid processes and add the environment variable before the process initialisation. <@U030K4UM3H7> FYI this is related to the discussion we were having yesterday. So, I thought you may be interested.

I hope this helps <@U03C7CHRS9E>. If anything is not clear I am happy to answer more questions.

If no tasks are running, then we could check overlord log for any kind of error or warning. Check the middle manager where it is submitted to we can check that log as well.

404 generally suggest server is not reachable, so check if your zookeeper is running fine and check for any errors in zookeeper/overlord log.

I looked in the logs and can't find anything

2023-03-01T09:55:45,395 INFO [main] org.apache.zookeeper.server.ContainerManager - Using checkIntervalMs=60000 maxPerMinute=10000
2023-03-01T09:55:56,342 INFO [SyncThread:0] org.apache.zookeeper.server.persistence.FileTxnLog - Creating new log file: log.80c3
2023-03-01T09:56:15,722 INFO [SessionTracker] org.apache.zookeeper.server.ZooKeeperServer - Expiring session 0x10003d4e8290000, timeout of 30000ms exceeded
2023-03-01T09:56:15,723 INFO [SessionTracker] org.apache.zookeeper.server.ZooKeeperServer - Expiring session 0x10003d4e8290003, timeout of 30000ms exceeded
2023-03-01T09:56:15,723 INFO [SessionTracker] org.apache.zookeeper.server.ZooKeeperServer - Expiring session 0x10003d4e8290001, timeout of 30000ms exceeded
2023-03-01T09:56:15,723 INFO [SessionTracker] org.apache.zookeeper.server.ZooKeeperServer - Expiring session 0x10003d4e8290002, timeout of 30000ms exceeded

in middle: 2023-03-01T10:32:16,635 WARN [main] org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2023-03-01T10:32:17,726 INFO [main] com.google.inject.Guice - An exception was caught and reported. Message: org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "hdfs"
java.lang.RuntimeException: org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "hdfs"
        at org.apache.druid.security.ranger.authorizer.RangerSecurityDruidModule.configure(RangerSecurityDruidModule.java:59) ~[?:?]
        at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:340) ~[guice-4.1.0.jar:?]
        at com.google.inject.spi.Elements.getElements(Elements.java:110) ~[guice-4.1.0.jar:?]
        at com.google.inject.util.Modules$OverrideModule.configure(Modules.java:198) ~[guice-4.1.0.jar:?]
        at com.google.inject.AbstractModule.configure(AbstractModule.java:62) ~[guice-4.1.0.jar:?]
        at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:340) ~[guice-4.1.0.jar:?]
        at com.google.inject.spi.Elements.getElements(Elements.java:110) ~[guice-4.1.0.jar:?]
        at com.google.inject.internal.InjectorShell$Builder.build(InjectorShell.java:138) ~[guice-4.1.0.jar:?]
        at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:104) ~[guice-4.1.0.jar:?]
        at com.google.inject.Guice.createInjector(Guice.java:99) ~[guice-4.1.0.jar:?]
        at com.google.inject.Guice.createInjector(Guice.java:73) ~[guice-4.1.0.jar:?]
        at com.google.inject.Guice.createInjector(Guice.java:62) ~[guice-4.1.0.jar:?]
        at org.apache.druid.initialization.ExtensionInjectorBuilder.build(ExtensionInjectorBuilder.java:49) ~[druid-server-26.0.0-SNAPSHOT.jar:26.0.0-SNAPSHOT]
        at org.apache.druid.initialization.ServerInjectorBuilder.build(ServerInjectorBuilder.java:118) ~[druid-server-26.0.0-SNAPSHOT.jar:26.0.0-SNAPSHOT]
        at org.apache.druid.initialization.ServerInjectorBuilder.makeServerInjector(ServerInjectorBuilder.java:73) ~[druid-server-26.0.0-SNAPSHOT.jar:26.0.0-SNAPSHOT]
        at org.apache.druid.cli.GuiceRunnable.makeInjector(GuiceRunnable.java:85) ~[druid-services-26.0.0-SNAPSHOT.jar:26.0.0-SNAPSHOT]
        at org.apache.druid.cli.ServerRunnable.run(ServerRunnable.java:62) ~[druid-services-26.0.0-SNAPSHOT.jar:26.0.0-SNAPSHOT]
        at org.apache.druid.cli.Main.main(Main.java:112) ~[druid-services-26.0.0-SNAPSHOT.jar:26.0.0-SNAPSHOT]
Caused by: org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "hdfs"
        at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3332) ~[?:?]
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3352) ~[?:?]
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124) ~[?:?]
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3403) ~[?:?]
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3371) ~[?:?]
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:477) ~[?:?]
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:226) ~[?:?]
        at org.apache.druid.security.ranger.authorizer.RangerSecurityDruidModule.configure(RangerSecurityDruidModule.java:56) ~[?:?]
        ... 17 more

Error “No FileSystem for scheme “hdfs”" seems to be a hdfs issue, could you check at hdfs side if there is for error message

What are the hdfs version and druid are they compatible and also check if  libraries are same in below two locations , if not copy them and retry

copy all of the hadoop* jars from dist/druid/extensions/druid-hdfs-storage to dist/druid/extensions/druid-parquet-extensions

I used hadoop-client-3.1.1 an druid 26.0.0-SNAPSHOT

it worked for 2 months and suddenly stopped working

i tried to change the reading from hdfs to local but it didn't work

yeah, local is meant for single server deployments with local deep storage. What are your deep storage settings? `druid.storage.*`  properties? Can you access the hdfs deep storage location from the druid nodes?

You can try to increase the Readiness probe time or try tailing the historical log during a restart

I'm actually having the same issue. The problem is that since it is a statefulset, the old pod is deleted before the new one comes up and so I can't see what the issue was. The logs make it look like a graceful shutdown is being forced, but I don't know by what and there is nothing in the coordinator logs to indicate that it is responsible. A failing health check could be the result of it being shut down.

you have to determine if the shutdown is due to K8s or if it is due to something at the druid level

For me its due to k8s, we saw a clear event

Mine is likely k8s too. I will figure it out, even if I have to set the readiness to 8 hours and check periodically and catch it in the act.

It was k8s. It is evicting them because they are using too much local ephemeral disk space (they are using persistent volumes for segments, but that is not the same as this). It is not clear to me how much they are supposed to use, what they are using it for or what configuration options affect that, but I'll be looking to move whatever directory it is using into the volume somewhere.

It was because my volume was mounted in `/druid` instead of `/opt/var/druid` Hopefully your problem is that or something similar.

No it is not the same, if we incease the number of `failureThreshold` that solve the issue but it is not a good workaround...

I would say test the health endpoint yourself using kubectl port-forward, and determine if the node itself is even functioning. If it is responding to normal requests but not health checks, that's odd.

you can drop the terminationGracePeriod i think it defaults to 30, we changed all ours to something really small like low single digit seconds because we don’t have any PreStop hooks

The health endpoint responds but after a long time (few minutes)... and meanwhile we see in the log that the Historical is loading the segments... it is like if the health is not ready while all segments are not fully loaded... and as it took few minutes, k8s decided to restart the pod after the 3 default failures of the probes...
Do you know what are the behavior of theses probes ? when Druid considers the live probe ok and the ready ok ?

I think you can implement a startup probe so that the other probes are not run/checked until this completes

We have something like this which works well:
```     startUpProbe:
        httpGet:
          path: /druid/historical/v1/readiness
          port: 8088
        # up to 20 minutes to startup before other probes run
        periodSeconds: 10
        failureThreshold: 180

    livenessProbe:
        periodSeconds: 5
        failureThreshold: 3
        httpGet:
          path: /status/health
          port: 8088

    readinessProbe:
        periodSeconds: 5
        failureThreshold: 3
        httpGet:
          path: /druid/historical/v1/readiness
          port: 8088```

Although now that I look at it, the `readinessProbe` is not needed and redundant.

Good, thank you I'll look at this startUpProbe.
"`readinessProbe` is not needed and redundant.", you mean we can simply remove it ?

Yeah. I'd edit it out of my message, but this slack doesn't seem to allow editing.

Ok, what is under `/druid/historical/v1/readiness` ? which condition will validate this endpoint if you know ?

It is documented here <https://druid.apache.org/docs/latest/operations/api-reference.html> But honestly I took it from an example kubernetes config from one of the people who maintains druid-operator.

Try checking the overlord log for information on the task

Hey Ian , looks like coordinator try to call middle-manger via rest ap and failed ..
from coordinator - application log
*java.net.ConnectException: Connection refused: /10.1.3.215:8091*

this is the log of coordinator .., I dont have a seaprate deploymebnt for overload ..

the default .. both processes deploy on the same host ..right ?

yes, on a master server, but their should be two separate logs, are they running as separate java processes?

I dont know .. I just speen up a "coordinator" command on docker ..

ok and the middlemanager is running, correct?

yes .. on a diffrent ec2 node as docker container ..