Apache Druid

<@U04G9R2QA0K> we're working on updating the docs on this.  This metric is out of date now.  Here are some comments from the developers:

Hi <@U0411DE2SS0>

Thanks for your response

Hai we want to monitor the druid lag of all the products we have in grafana dashboard.can any one help me to write query

```The Unable to load native-hadoop library for your platform... using builtin-java classes where applicable warning messages can be safely ignored```
Also, have you followed the configurations:
1. <https://druid.apache.org/docs/latest/tutorials/tutorial-batch-hadoop.html#configure-druid-to-use-hadoop>
2. <https://druid.apache.org/docs/latest/development/extensions-core/hdfs.html>

Thank you very much for your answer. I've been reading this documentation and I'm looking for the reason why the druid won't connect to HDFS, the authentication is done after kerberos.

According to the documentation, I created a keytab file and the logs show that it works.

In the Ranger audit I have no connection to hdfs. When a druid starts, it should connect to HDFS?

I wonder whether this tutorial will be useful for you: <https://druid.apache.org/docs/latest/tutorials/tutorial-kerberos-hadoop.html>

Do check out this summit video: <https://druidsummit.org/americas/americas-agenda?agendaPath=session/1035730>

Also see Julian's last comment in <https://github.com/apache/druid/issues/9780> - he suggests using kafka or kinesis where possible for now, for structured streaming applications.

exploding a json array usually means creating a lot of columns (often empty as you have to create for the largest anticipated array size). Instead use the nested columns feature to store and query the array <https://druid.apache.org/docs/latest/querying/nested-columns.html>

We are also adding support for `UNNEST` that can be used to explode an array. <https://github.com/apache/druid/pull/13268> - Right now, only available for native queries.

Hi!
Could you please check if segments are not getting published or if the handoff is unsuccessful from the task logs?
If there is a log like `Waiting for handoff`  at the end of the task logs, could you please check the coordinator/time metric and historical availability as well?

Oh I found tons of `Still waiting for handoff for [%d] segments`

seems like peon tasks were successful but publishing failed
also, I found `Shutdown [index_kafka_xxx] because: [task is not in knownTaskIds...`

&gt; could you please check the coordinator/time metric and historical availability as well?
It would be great if you could share which duty is taking the most time in case the coordinator cycles are long

Hello, try increasing `completionTimeout` in your supervisor spec

By default, `taskDuration` = `completionTimeout` . In my case, `taskDuration` was 1800 seconds. I’ve increased `completionTimeout` to 3600 seconds, and issue is no more.
I’ve got this issue when lag is growing. When there is a lot of Kafka data to ingest, it seems the `taskDuration` is not long enough to publish segments and tasks failed with a long duration

It seems like the issue here is that segments have been published but not handed off based on the logs

I think the main issue here is : `"No task in the corresponding pending completion taskGroup[group_id] succeeded before completion timeout elapsed"`

And I’ve managed to get rid of it doing what I’ve suggested. I think it’s worth a shot

Yup, I must edit my previous message, but it seems I can’t

By default, `taskDuration` = `completionTimeout` . In my case, `taskDuration` was 1800 seconds. I’ve increased `completionTimeout` to 3600 seconds, and issue is no more.
I’ve got this issue when lag is growing. When there is a lot of Kafka data to ingest, it seems the `taskDuration` is not long enough and tasks failed with a long duration

Thanks for reply friends. I once changed `druid.segmentCache.locations=[{"path":"/data1/druid/segment-cache/","maxSize":"6200g"}]`
  to `druid.segmentCache.locations=[{"path":"/data1/druid/segment-cache/","maxSize":"4800g", "freeSpacePercent":20..0}`]  and as Amatya said, this could be problem.

I’ll consider what you  said. thansk!

Hey I found that `completionTimeout` should be quite long in some cases. I’m running  big cluster and consumes terabytes of data in a day. When kafka lag is huge, some tasks take more than 2 hours (`taskDuration`=1hr) and even `PT3600S` wasn’t enough. I gave `PT7200S` and Druid could manage tasks to be completed. After few hours, tasks took about 1HR again.
I twaeaked some other options so I’m not sure this is the only solution. but it did work anyway :sweat_smile:
Thank you !

Hello Eddie, yep this is what I’ve experienced too. Once the lag is caught up, everything goes back to normal. Glad to hear you solved it

which log are you seeing this error? In the supervisor? Can you take a look at the task log?

By default, there will be a segment replication of 2 i.e a segment is loaded into 2 historicals.

There are no permissions on a lookup by lookup level. There is ongoing work to make lookups be just datasources and when that happens you would be able to manipulate them with the `DATASOURCE` permission

Yes. You can create ingestion spec that uses a <https://druid.apache.org/docs/latest/ingestion/native-batch-input-sources.html#druid-input-source|druid table as a source> or if you are on Druid 24.x, even do this as an <https://druid.apache.org/docs/latest/multi-stage-query/examples.html#insert-for-reindexing-an-existing-datasource|INSERT/REPLACE using SQL-based ingestion> and SELECTing from a hourly table to insert into daily table and so forth. Since it is essentially a different ingestion process you can control rollup/aggregation at each step. You'll need to schedule the batches when you know the prior level is complete. i.e. schedule the daily load after the last hour of the day has been loaded to the hourly table.

To add to that, if you have Druid 24.0 have a look at the `Demo 2` demo query

if you do not have Druid 24.0 you can use native batch to `Re-index from Druid`

Thanks <@U030K4UM3H7> and <@U030W50CNAC>. Let me have a look

Hi <@U03UUDS1BPA>, what is "a lot of time"? One alternative is to do the lookup logic at ingestion time, such that you have a product_grp column in the base table. This will avoid the lookup altogether at query time and should make your queries faster.

<@U03UUDS1BPA>, can you try inverting the query, i.e. do the aggregation in the subquery, then do the lookup in the top level query.  This way there are fewer lookup operations to do.

You could try reducing the number of lookup operations by first pulling out the distinct list of item_nbr values to look up, e.g. use these CTEs to create the "item_lookup" temp table to join into the main query.

```With item_nbr_list as (
  Select distinct item_nbr 
    from table name 
   where __time between '2022-04-01' and '2022-05-01'
), item_lookup as (
  Select item_nbr, 
         LOOKUP(CAST(item_nbr AS VARCHAR),'lookup_table')) as product_grp 
         from item_nbr_list
)```
I don't know if a temp table is inherantly faster to access than a lookup though ...

why do you need the outer query? try
```select 
			day,
			
			LOOKUP(CAST(item_nbr AS VARCHAR),'lookup_table')) as product_grp,
            sum(sales)
			from 
			tablename
			where 
			__time between '2022-04-01' and '2022-05-01' group by 1,2```

<@U030K4UM3H7>: i tried the ingest the same dataset with lookup logic at ingestion time and while querying the new dataset its giving response within two second’s but with using a lookup approach its taking approximately 50secs .
But incase there is a change in the item product group mapping if one item moves from one product group to another then in that case i will need to do a complete historical ingestion of the data as product group would have been persisted in the dataset i want to avoid historical re-ingestion

<@U03BJAMQ4H5> without outer query the group by will group by item_nbr and not product_group

hmm… I tried
```select LOOKUP("Country",'store_sales_country_to_iso31661'),count(*) from "store-sales" group by 1```
and see

I tried select
```select Country,count(*) from "store-sales" group by 1```
and see

although I have not tried with many-one mapping in the lookup… I will check that out

confirmed this….the grouping is on the result of the lookup not the key
```select item,sum(qty) from itemtest group by 1```


```select LOOKUP(item,'item_grp'),sum(qty) from itemtest group by 1```


somehow for me when i am using  below query with auto limit it is working correctly but when using without auto limit it is grouping on the item_nbr
```select
            LOOKUP(CAST(item_nbr AS VARCHAR),'lookup_table') as group
            from
            table_name
            where
            __time between '2022-04-01' and '2022-05-01' group by 1```

How long after the datasources were dropped do they still appear in `INFORMATION_SCHEMA.TABLES` for? Do you see those tables in the `INFORMATION_SCHEMA.COLUMNS` table

It has been several days since I dropped the old tables.
I indeed still see the tables on `INFORMATION_SCHEMA.COLUMNS`.

Those old datasources are no longer in S3 as well as in the historical’s segment-cache folders.

strange.. what do you get if you query `SELECT COUNT(*) FROM &lt;mystery_table&gt;` ?

I am not the expert on this but I believe that setting "max segments in node loading queue" to `0` will effectively stop shuffling. Maybe someone with more knowledge can chime in or confirm. I definitely remember overhearing people talking about setting something in that dialog to `0`. to prevent shuffling.

I thought it would be maxSegmentsToMove you'd set to 0?  And bump up maxSegmentsInNodeLoadingQueue temporarily for the initial loading?

Ran df -h on middle manager

```bash-5.1$ df -h
Filesystem                Size      Used Available Use% Mounted on
overlay                  48.3G     34.3G     14.0G  71% /
tmpfs                    64.0M         0     64.0M   0% /dev
tmpfs                    31.4G         0     31.4G   0% /sys/fs/cgroup
shm                      64.0M         0     64.0M   0% /dev/shm
/dev/sda1                48.3G     34.3G     14.0G  71% /etc/hosts
/dev/sda1                48.3G     34.3G     14.0G  71% /dev/termination-log
/dev/sda1                48.3G     34.3G     14.0G  71% /etc/hostname
/dev/sda1                48.3G     34.3G     14.0G  71% /etc/resolv.conf
/dev/sda1                48.3G     34.3G     14.0G  71% /opt/druid/var
/dev/sdc                  3.9G      6.6M      3.8G   0% /opt/druid/var/druid
tmpfs                    56.6G     12.0K     56.6G   0% /var/run/secrets/kubernetes.io/serviceaccount
tmpfs                    31.4G         0     31.4G   0% /proc/acpi
tmpfs                    64.0M         0     64.0M   0% /proc/kcore
tmpfs                    64.0M         0     64.0M   0% /proc/keys
tmpfs                    64.0M         0     64.0M   0% /proc/timer_list
tmpfs                    64.0M         0     64.0M   0% /proc/sched_debug
tmpfs                    31.4G         0     31.4G   0% /proc/scsi
tmpfs                    31.4G         0     31.4G   0% /sys/firmware```

what is -Djava.io.tmpdir in your middle manager config pointing to?

Seems like the ingestion errored out while writing out the logfile which is placed in the baseTaskDir. If everything is set to defaults this should be equivalent to `java.io.tmpdir`. Setting the value to the disk with free space might help. Also did you run the df -h during the ingestion that throwed the error or post the failure?

I am using this helm chart and have just updated deep-store to point to Azure storage account

<https://github.com/apache/druid/blob/master/helm/druid/values.yaml>

I don't see `java.io.tmpdir` being set in java opts of middle manager in this values.yaml. Let me set it and try

Currently its set to `-<http://Djava.io|Djava.io>.tmpdir=var/tmp` <@U031AD802MN> <@U03BJAMQ4H5>

if that is in /opt/druid/var/druid then you will have a space issue. Where is the segment cache?

On middle manager segment cache is at `/opt/druid/var/druid/segment-cache`

```bash-5.1$ cd /opt/druid/var/druid
bash-5.1$ cd segment-cache/
bash-5.1$ ls -lrth
total 0
bash-5.1$ pwd
/opt/druid/var/druid/segment-cache```

can you paste the middlemanager runtime properites and jvm config and historical runtime properties and jvm config here?

Sorry, how can I get the runtime configs ?

I think you can get some of the runtime properties from the logs when the middlemanager/historical starts (please mask out any sensitive fields)

Hi. I see this location on middle manager is getting filled while I am running the reindexing.
`/dev/sdc                  3.9G      2.7G      1.1G  71% /opt/druid/var/druid`

On checking at this path, I saw all the segments are getting downloaded on the middle manager from deep store. And eventually the disk is getting out of memory. As the disk attached to middle manager is only of 4GB

```bash-5.1$ pwd
/opt/druid/var/druid/task/index_parallel_rms_logs_query_performance_kkmnpjac_2023-01-06T04:17:09.726Z/work/indexing-tmp/rms_logs_query_performance
bash-5.1$ ls -lrth
total 252K
drwxr-sr-x    3 druid    druid       4.0K Jan  6 04:17 2022-12-01T01:00:00.000Z_2022-12-01T02:00:00.000Z
drwxr-sr-x    3 druid    druid       4.0K Jan  6 04:17 2022-12-01T02:00:00.000Z_2022-12-01T03:00:00.000Z
drwxr-sr-x    3 druid    druid       4.0K Jan  6 04:17 2022-12-01T03:00:00.000Z_2022-12-01T04:00:00.000Z
drwxr-sr-x    3 druid    druid       4.0K Jan  6 04:17 2022-12-01T04:00:00.000Z_2022-12-01T05:00:00.000Z
drwxr-sr-x    3 druid    druid       4.0K Jan  6 04:17 2022-12-01T05:00:00.000Z_2022-12-01T06:00:00.000Z
drwxr-sr-x    3 druid    druid       4.0K Jan  6 04:17 2022-12-01T06:00:00.000Z_2022-12-01T07:00:00.000Z
drwxr-sr-x    3 druid    druid       4.0K Jan  6 04:18 2022-12-01T07:00:00.000Z_2022-12-01T08:00:00.000Z
drwxr-sr-x    3 druid    druid       4.0K Jan  6 04:18 2022-12-01T08:00:00.000Z_2022-12-01T09:00:00.000Z
drwxr-sr-x    3 druid    druid       4.0K Jan  6 04:18 2022-12-01T09:00:00.000Z_2022-12-01T10:00:00.000Z
drwxr-sr-x    3 druid    druid       4.0K Jan  6 04:18 2022-12-01T10:00:00.000Z_2022-12-01T11:00:00.000Z```
[9:59 AM] Does this means middle manager has to have a disk of size ~total size of the segment ?

You can split the workload across multiple middlemanager using subtasks

You can also try to reduce the time interval of your reindex to reduce the amount of data you need to download

you can change `-Djava.io.tmpdir`  to some location that has more space

<@U04CZ1Z9EUU> - Happy to help. Are you getting that error when you attempt to use the Metrics API or are you using the Prometheus emitter or something else?

More troubling is that many of our Superset queries are failing quickly with the following message:
```Error: Plan validation failed (org.apache.calcite.tools.ValidationException): org.apache.calcite.runtime.CalciteContextException: From line 6, column 7 to line 6, column 14: Column '__time' not found in any table```


The full error:
```druid-broker-12 druid-broker 2023-01-05T16:11:29,121 ERROR [DruidSchema-Cache-0] org.apache.druid.query.ResultLevelCachingQueryRunner - Error while preparing for result level caching for query e73d2ca2-c8f9-409b-8e31-b8469713b1ab with error Sequence iterator timed out waiting for data 
druid-broker-12 druid-broker org.apache.druid.query.QueryTimeoutException: Sequence iterator timed out waiting for data
druid-broker-12 druid-broker 	at org.apache.druid.java.util.common.guava.ParallelMergeCombiningSequence$1$1.hasNext(ParallelMergeCombiningSequence.java:205) ~[druid-core-0.23.0.jar:0.23.0]
druid-broker-12 druid-broker 	at org.apache.druid.java.util.common.guava.BaseSequence.makeYielder(BaseSequence.java:89) ~[druid-core-0.23.0.jar:0.23.0]
druid-broker-12 druid-broker 	at org.apache.druid.java.util.common.guava.BaseSequence.toYielder(BaseSequence.java:69) ~[druid-core-0.23.0.jar:0.23.0]
druid-broker-12 druid-broker 	at org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:88) ~[druid-core-0.23.0.jar:0.23.0]
druid-broker-12 druid-broker 	at org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:84) ~[druid-core-0.23.0.jar:0.23.0]
druid-broker-12 druid-broker 	at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-core-0.23.0.jar:0.23.0]
druid-broker-12 druid-broker 	at org.apache.druid.java.util.common.guava.WrappingSequence.toYielder(WrappingSequence.java:83) ~[druid-core-0.23.0.jar:0.23.0]
druid-broker-12 druid-broker 	at org.apache.druid.java.util.common.guava.ParallelMergeCombiningSequence.toYielder(ParallelMergeCombiningSequence.java:153) ~[druid-core-0.23.0.jar:0.23.0]
druid-broker-12 druid-broker 	at org.apache.druid.java.util.common.guava.LazySequence.toYielder(LazySequence.java:46) ~[druid-core-0.23.0.jar:0.23.0]
druid-broker-12 druid-broker 	at org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:88) ~[druid-core-0.23.0.jar:0.23.0]
druid-broker-12 druid-broker 	at org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:84) ~[druid-core-0.23.0.jar:0.23.0]
druid-broker-12 druid-broker 	at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-core-0.23.0.jar:0.23.0]
druid-broker-12 druid-broker 	at org.apache.druid.java.util.common.guava.WrappingSequence.toYielder(WrappingSequence.java:83) ~[druid-core-0.23.0.jar:0.23.0]
druid-broker-12 druid-broker 	at org.apache.druid.java.util.common.guava.MergeSequence.lambda$toYielder$1(MergeSequence.java:66) ~[druid-core-0.23.0.jar:0.23.0]
druid-broker-12 druid-broker 	at org.apache.druid.java.util.common.guava.BaseSequence.accumulate(BaseSequence.java:44) ~[druid-core-0.23.0.jar:0.23.0]
druid-broker-12 druid-broker 	at org.apache.druid.java.util.common.guava.MergeSequence.toYielder(MergeSequence.java:63) ~[druid-core-0.23.0.jar:0.23.0]
druid-broker-12 druid-broker 	at org.apache.druid.query.RetryQueryRunner$1.toYielder(RetryQueryRunner.java:133) ~[druid-server-0.23.0.jar:0.23.0]
druid-broker-12 druid-broker 	at org.apache.druid.common.guava.CombiningSequence.toYielder(CombiningSequence.java:78) ~[druid-core-0.23.0.jar:0.23.0]
druid-broker-12 druid-broker 	at org.apache.druid.java.util.common.guava.MappedSequence.toYielder(MappedSequence.java:49) ~[druid-core-0.23.0.jar:0.23.0]
druid-broker-12 druid-broker 	at org.apache.druid.java.util.common.guava.MappedSequence.toYielder(MappedSequence.java:49) ~[druid-core-0.23.0.jar:0.23.0]
druid-broker-12 druid-broker 	at org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:88) ~[druid-core-0.23.0.jar:0.23.0]
druid-broker-12 druid-broker 	at org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:84) ~[druid-core-0.23.0.jar:0.23.0]
druid-broker-12 druid-broker 	at org.apache.druid.query.CPUTimeMetricQueryRunner$1.wrap(CPUTimeMetricQueryRunner.java:77) ~[druid-processing-0.23.0.jar:0.23.0]
druid-broker-12 druid-broker 	at org.apache.druid.java.util.common.guava.WrappingSequence.toYielder(WrappingSequence.java:83) ~[druid-core-0.23.0.jar:0.23.0]
druid-broker-12 druid-broker 	at org.apache.druid.java.util.common.guava.MappedSequence.toYielder(MappedSequence.java:49) ~[druid-core-0.23.0.jar:0.23.0]
druid-broker-12 druid-broker 	at org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:88) ~[druid-core-0.23.0.jar:0.23.0]
druid-broker-12 druid-broker 	at org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:84) ~[druid-core-0.23.0.jar:0.23.0]
druid-broker-12 druid-broker 	at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-core-0.23.0.jar:0.23.0]
druid-broker-12 druid-broker 	at org.apache.druid.java.util.common.guava.WrappingSequence.toYielder(WrappingSequence.java:83) ~[druid-core-0.23.0.jar:0.23.0]
druid-broker-12 druid-broker 	at org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:88) ~[druid-core-0.23.0.jar:0.23.0]
druid-broker-12 druid-broker 	at org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:84) ~[druid-core-0.23.0.jar:0.23.0]
druid-broker-12 druid-broker 	at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-core-0.23.0.jar:0.23.0]
druid-broker-12 druid-broker 	at org.apache.druid.java.util.common.guava.WrappingSequence.toYielder(WrappingSequence.java:83) ~[druid-core-0.23.0.jar:0.23.0]
druid-broker-12 druid-broker 	at org.apache.druid.java.util.common.guava.Yielders.each(Yielders.java:32) ~[druid-core-0.23.0.jar:0.23.0]
druid-broker-12 druid-broker 	at org.apache.druid.sql.calcite.schema.DruidSchema.refreshSegmentsForDataSource(DruidSchema.java:705) ~[druid-sql-0.23.0.jar:0.23.0]
druid-broker-12 druid-broker 	at org.apache.druid.sql.calcite.schema.DruidSchema.refreshSegments(DruidSchema.java:658) ~[druid-sql-0.23.0.jar:0.23.0]
druid-broker-12 druid-broker 	at org.apache.druid.sql.calcite.schema.DruidSchema.refresh(DruidSchema.java:390) ~[druid-sql-0.23.0.jar:0.23.0]
druid-broker-12 druid-broker 	at org.apache.druid.sql.calcite.schema.DruidSchema.lambda$startCacheExec$1(DruidSchema.java:337) ~[druid-sql-0.23.0.jar:0.23.0]
druid-broker-12 druid-broker 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
druid-broker-12 druid-broker 	at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
druid-broker-12 druid-broker 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
druid-broker-12 druid-broker 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
druid-broker-12 druid-broker 	at java.lang.Thread.run(Thread.java:833) [?:?]
druid-broker-12 druid-broker 	Suppressed: java.lang.NullPointerException```
This is fascinating:
```Suppressed: java.lang.NullPointerException```

<@U03149ALHN3> - Do you have query logging turned on to see what types are queries are causing the timeout? Also, if you re-run the query (or queries) in question causing the errors do you get a consistent timeout?

we do have a real time list of all the queries. When we re-run the queries, some completed and some timed out again. So far I am not seeing any patterns (e.g. a bad full text search).

<@U03149ALHN3> - Also can you check to see if there are any other errors or warnings in the master logs or broker logs around when the query timeout occured?

so, we just finished rolling restart the brokers now. And things looked a lot better.
It looks like a lot of broker threads (in the thread pool) failed silently and they are not being refreshed/reaped properly.

This kind of explained why our cluster were not busy even though there were a lot of errors. The queries failed at broker level before they even made it to Historicals.

too bad for our uptime tho :slightly_smiling_face: the cluster were doing quite well for 77 days.

<@U03149ALHN3> - Yes :smiling_face_with_tear: for the uptime. A few questions...  Have you see this issue re-occur in the past?

I ask because looking around on my side seems to indicate a restart of the broker clear the issue, but it eventually returns.

This is actually the first time we saw this issue.

I've come across this some lately, also saw restarting brokers cleared it for a while.  Iirc, the issues I saw were related to lookups, and tuning things like druid.manager.lookups.threadPoolSize on the coordinator and druid.lookup.numLookupLoadingThreads on places lookups were loading helped.  Not sure if that applies in your case.

Interesting. We banned to use of lookups, but lookup is still turned on in our config. I’ll take a look at it.

If you don't have any lookups, then pretty sure that's not the issue here.

You have results level caching turned on in the broker? 

If you use result level cache and use cache true in the broker then the load on the broker will be large. Use cache will cache segment level results on the broker and all the merging will happen on the broker. This will overload the broker. I suggest turning off all caching on the broker and setting use cache and populate cache to true in the historicals 

We do have result level caching turned on in the broker. interesting… the cache is not large, only 5GB on each broker. Is it best to just turn off cache completely in broker?

yes. The cache is store on heap. What is the Xmx on your broker. Result level cache is useful only if you have the same queries and data again and again.

xmx is fairly generous: 24GB
this is on G1 GC

The broker cache hit rate is actually pretty good: 98%

ok. If useCache and populateCache is also true on broker then the broker will definitely get overloaded. I would set both useCache and useResultLevelCache to false on the broker

ah, useCache and populateCache are definitely false. Those two settings don’t work on large clusters. In the past we had an outage because those two were on.

We only gave it 5 CPU in Kubernetes because on normal situations, it barely used 3 CPUs.

hmm…what kind of queries do you have and what concurrency?

on average number of queries aren’t a lot. 20k-40k per day.
These came from Superset dashboards.

ok….if you see <https://github.com/apache/druid/issues/10028>

the sequence iterator error is a basically a timeout.

this could be also due to the amount of data the queries are scanning or a lot of data being returned to broker for merging.

yeah, I think there’s an underlying issue that got masked. See:
```Suppressed: java.lang.NullPointerException```
Some threads basically didn’t do anything until the 5 minutes timeout kicked in.

and that explained why on our datadog, historicals looked so free as if they are not doing anything.

There are no patterns in the failed queries. Some do groupBy, some do topN, some do nested SELECT queries, some do full text search like “abc123%“, etc.

ok..with this many historicals the merge load on the broker for groupbys will be very large.

I would look at using 50 historicals with 110 vcpu each and around 5 brokers with 35 vcpu each.

the problem with assigning a lot of CPU is that the processing thread is expensive on memory, no? The merge buffer is 2gb and based on this math:
```DRUIDHISTORICAL_PROCESSING_NUMMERGEBUFFERS + DRUIDHISTORICAL_PROCESSING_NUMTHREADS + 1) * DRUIDHISTORICAL_PROCESSING_BUFFER_SIZEBYTES```
that already consumed 66gb of DirectMemory size.

both `DRUIDHISTORICAL_PROCESSING_NUMMERGEBUFFERS` and `DRUIDHISTORICAL_PROCESSING_NUMTHREADS` are 16

this is true irrespectoive of how you split the historicals. You may slice it different ways but your overall processing threads should remain the same. If with 11 vcpu you are setting processing threads to 16 then you are over comitting the threads.

we used to set it to 16 CPU, but our average cpu usage graph showed that we almost never used &gt; 8-9 CPUs

this likely means that there are bottlenecks that prevent the historicals from taking on the full load. It may be the http connections or threads and few other things.

interesting. jetty threads are quite generous as well: 500

anyway having fewer larger servers should not increase direct memory

having too many small servers is generally not a good idea.

the aggregations coming through from each historical gets merged on the broker…with 512 servers your merge will be much larger than with 50 servers. And there is only one broker for one query. So that one broker will take that full load.

i see, can this be mitigated with more brokers?

probably not as you have 38 brokers and less that 1 query per second. Very likely each broker is doing only one query at a time.

here is another thought…by default the broker does parallel combine and can use 75% of available CPU for that. In your case the max available CPU for a query on the broker is 5. So a query that could move faster with a larger broker will slow down a lot with your approach.

the simplest change you can make is have 5 brokers with 35 vcpu each. Reduce processing threads on brokers to 1 (brokers don’t need processing threads) and set numMergeBuffers =5 on each broker (so you need only (5+1+1)*2=14GB direct memeory. The 35 vcpu will allow significantly more merge threads to come into play

oooo that’s an interesting suggestion. Thank you. We’ll play with this on our lab.

No, there isn't. Since Druid is geared towards sub-second queries people tend to take the opposite approach: issue the query straight away but be smart about it. For example: I have an app where the user can explore their data, I know that the widgets in the app will issue very 'cheap' queries because I know their shapes and what range of data the user is allowed to query - I can just send them to my main broker tier with little worry (I can add `timeout` to make sure they do not take up too many resources if something does go wrong). Next up I also have a feature in my app where the user can enter arbitrary SQL queries, I can send those to a special tier of brokers so that they do not interfere with the queries from the app, I can also set the priorities lower.

The overlord coordinates directly with each task using REST calls, it seems like it is unable to communicate with it.  How is this deployed? Could it be a network issue? Can you share the task log for task "index_kafka_sn-content-events_60625f27846ee46_aehhhela"?

Hi <@U030K4UM3H7> I thought it was some kind of network issue too. I can see that the task starts up listening on port 8100 but in the coordinator log I see that connection refused message. In order to test network connectivity I started a simple java process that listens on 8100 on the vm where the peons run and was able to telnet from the coordinator machine ok. Heres the log you asked for... if you look for the setting 8100...

```seanlaval@Seans-MBP Downloads % grep 8100 log
2023-01-06T16:24:34,847 DEBUG [main] org.apache.druid.guice.JsonConfigurator - Loaded class[class org.apache.druid.server.DruidNode] from props[druid.] as [DruidNode{serviceName='druid/middleManager', host='w3-druid-data-01-preprod', bindOnHost=false, port=-1, plaintextPort=8100, enablePlaintextPort=true, tlsPort=-1, enableTlsPort=false}]
2023-01-06T16:25:23,155 INFO [main] org.apache.druid.cli.CliPeon - * druid.plaintextPort: 8100
2023-01-06T16:26:41,489 INFO [main] org.apache.druid.server.initialization.jetty.JettyServerModule - Creating http connector with port [8100]
2023-01-06T16:26:44,893 DEBUG [main] org.eclipse.jetty.util.component.ContainerLifeCycle - ServerConnector@221cdd87{HTTP/1.1, (http/1.1)}{0.0.0.0:8100} added {JettyMonitoringConnectionFactory@5d202dc4{STOPPED},AUTO}
2023-01-06T16:26:44,893 DEBUG [main] org.eclipse.jetty.server.AbstractConnector - ServerConnector@221cdd87{HTTP/1.1, (http/1.1)}{0.0.0.0:8100} added JettyMonitoringConnectionFactory@5d202dc4{STOPPED}
2023-01-06T16:26:44,893 DEBUG [main] org.eclipse.jetty.util.component.ContainerLifeCycle - Server@451a4187{STOPPED}[9.4.48.v20220622] added {ServerConnector@221cdd87{HTTP/1.1, (http/1.1)}{0.0.0.0:8100},AUTO}
2023-01-06T16:26:59,930 DEBUG [main] org.apache.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void org.apache.druid.java.util.emitter.service.ServiceEmitter.start()] on object[ServiceEmitter{serviceDimensions={service=druid/middleManager, host=w3-druid-data-01-preprod:8100, version=24.0.1}, emitter=NoopEmitter{}}].
2023-01-06T16:27:09,450 DEBUG [task-runner-0-priority-0] org.apache.druid.indexing.overlord.TaskRunnerUtils - Task [index_kafka_sn-content-events_60625f27846ee46_aehhhela] location changed to [TaskLocation{host='w3-druid-data-01-preprod', port=8100, tlsPort=-1}].```

Another possibility is the http threads. Do you have queries running on the real-time data? The brokers also communicate with the tasks to resolve queries and use up http threads to do so. Your can increase it with  druid.indexer.fork.property.druid.server.http.numThreads 

You can control it. Take a look at parameter `replicationThrottleLimit` in the <https://druid.apache.org/docs/latest/configuration/index.html#dynamic-configuration|coordinator's dynamic configuration>.

That’s the parameter I’m playing with right now. It’s been set to 10 and I’ve increased it to 500 and it looks like it made some difference, but throttling is still happening. The symptom we are experiencing is that segments are being created and querying them return proper metadata, but data queries return no data for data ranges in question. Does that sound like a reasonable cause and effect combination?

what is the number of segments to load on the historicals when the throttling happens?

929 segments, as far as I can see:
```2023-01-06T22:02:38,785 INFO [task-runner-0-priority-0] org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexSupervisorTask - Published [929] segments
2023-01-06T22:02:38,787 INFO [task-runner-0-priority-0] org.apache.druid.indexing.worker.executor.ExecutorLifecycle - Task completed with status: {
  "id" : "index_parallel_media_events.v1.media_embed_location.monthly.v1_mndemcao_2023-01-06T21:17:46.766Z",
  "status" : "SUCCESS",
  "duration" : 2687646,
  "errorMsg" : null,
  "location" : {
    "host" : null,
    "port" : -1,
    "tlsPort" : -1
  }
}```

can you check the services tab and see the load queue in the historicals?

I can see segment metadata and I can also see them under the `Segments` tab.

But if I issue a query against the data in that time range, I’m getting `Query returned no data`.

I’m getting `20,880 results in 1.62s` for
```{
  "queryType":"segmentMetadata",
  "dataSource":"events.my_event_datasource",
  "intervals":["2022-10-01/2022-11-01"]
}```
(that’s the time range in question)

How much available disk space you have on the historicals?

Could you please try `select num_replicas, count(*) from sys.segments where datasource=&lt;datasource&gt; and "start"=&lt;intervalStart&gt; and "end"=&lt;intervalEnd&gt; group by 1`  in the druid console? It's possible a few (core) segments are still unavailable

Hmmm…my thinking is that if the source shows as 100% available in the console then the query should not be returning empty results irrespective of throttling etc 

How many historicals do you have and is this source 100% available when the query returns empty results?

And what is the query that returns empty results?

Yes, the datasource shows it’s 100% available and all historical nodes under the services tab have nothing to load.
We have 14 historical nodes.
The query is a simple Druid SQL, along the lines of:
```select sum(metric1), sum(metric2) from "datasource" where __time &gt;= &lt;daterange_start&gt; and __time &lt; &lt;daterage_end&gt;```
I know it contains non-empty results since we have the same data in a different datasource (with different segment and query granularity) After re-running the same batch ingestion spec several times, the query above returns data that matches what we have in the original datasource. So yeah, the ultimate question here is why would re-running the same task several times produce different results.

If it’s of any help, we’re on `0.21.0` release.

<@U0319HD8HEC> the output is 2 (num replicas) and 929 (count).

Have the load rules changes for the new datasource recently? maybe between runs of the job? Just a thought. Could the prior ingestion have marked segments as unused due to load rules? If this were the case, you can fix this. <@U030MMVU35Z> and I recently posted a blog on how to resolve it: <https://saydul.substack.com/p/apache-druid-recovering-dropped-segments>

<@U030K4UM3H7> we haven’t changed our load rules from their defaults (`loadForever`). So I imagine that’s not the problem.

Plus, if it was the case, wouldn’t it mean that segment metadata queries and `Segments` tab would reflect that?

I'm not sure, and like you said, it sounds like you should not have that issue. And yes, the segments view should reflect that. Do you see all the segments as "Used" in the segments tab? 100% available means that all the segments in a Used state are loaded into one or more historicals. So I wondered about the Used status as a possible cause for the "missing" data.

Yeah, every segment in that datasource for this date range is *available* and *published*. And nothing clearly odd in the logs.

I imagine without more detailed info it might be difficult to debug this issue, so maybe you folks have at least some general recommendation / suspicions about what might (with emphasis on might :slightly_smiling_face: ) be happening here and where to look for next?

On our end, we’ve been thinking about upgrading to the latest version (as I mentioned, we’re on `0.21.0` ) right now.

Here’s an example of what I see in the `Segments` tab.

Brokers also keep a map of the segments in memory in order to scatter query requests to historicals. Are there any warnings or errors in the Broker log?

Nothing obvious (like exceptions or warnings), AFAICS.

Correct me if I’m wrong, but if broker’s balancing type is left untouched, it should be set to `random` and, thus, queries should be routed to random historical nodes. So if the data is loaded (and it looks like it is - with replication factor of 2), at least one of those historicals should respond with some data. I wonder if I should try querying historicals directly to see if the problem is at the “upper” layer.

That's a good idea. Just to clarify, the random selection of historical for a given segment query processing request, is among the ones that have the corresponding segment already available, so that should not be the root cause of this. ... unless the segment map were somehow outdated on the broker.

So I pinged `POST /druid/v2/candidates/` with the query in question and got back a bunch of segments back with IP addresses of historicals.

Based on <https://druid.apache.org/docs/latest/operations/api-reference.html#queries|this> API reference.

OK, so I ran some queries against historicals direclty and I’m getting 0s for every date range in that data source.
Query:
```{
  "queryType": "timeseries",
  "dataSource": {
    "type": "table",
    "name": "my_datasource"
  },
  "intervals": {
    "type": "intervals",
    "intervals": [
      "2022-06-01T05:00:00.000Z/2023-01-01T05:00:00.000Z"
    ]
  },
  "descending": false,
  "virtualColumns": [],
  "filter": {
    "type": "selector",
    "dimension": "dim1",
    "value": null,
    "extractionFn": null
  },
  "granularity": {
    "type": "all"
  },
  "aggregations": [
    {
      "type": "longSum",
      "name": "a0",
      "fieldName": "f1",
      "expression": null
    },
    {
      "type": "longSum",
      "name": "a1",
      "fieldName": "f2",
      "expression": null
    },
    {
      "type": "longSum",
      "name": "a2",
      "fieldName": "f3",
      "expression": null
    }
  ],
  "postAggregations": [],
  "limit": 2147483647
}```
Not sure if I’m doing it correctly though - I `POST` against `&lt;historical_host_ip&gt;:8083/druid/v2?pretty` . I do get values back for the original datasource (from which I ingest data) using this method, like so:
```[ {
  "timestamp" : "2022-12-14T05:00:00.000Z",
  "result" : {
    "a1" : 313735,
    "a2" : 100597,
    "a0" : 64
  }
} ]⏎```
For this new datasource every historical returns:
```[ {
  "timestamp" : "2022-06-01T05:00:00.000Z",
  "result" : {
    "a1" : 0,
    "a2" : 0,
    "a0" : 0
  }
} ]```
However, I get back the data for every month for this problematic datasource except 1 when I query using Druid Query UI, so these empty API responses look odd (assuming I’m doing it correctly).

Maybe we should step back and look at the reindex job that you ran. Can you share the spec ?

Sure. Here’s the one that sometimes runs successfully (as in it results in new segments being created and queryable data returned), but also, from time to time, resulting in issues described above:
```{
  "type": "index_parallel",
  "spec": {
    "dataSchema": {
      "dataSource": "new_datasource",
      "timestampSpec": {
        "column": "__time",
        "format": "millis"
      },
      "dimensionsSpec": {
        "dimensions": [
          ...a_bunch_of_dimensions_here
        ]
      },
      "metricsSpec": [
        ...a_bunch_of_metrics_here
      ],
      "granularitySpec": {
        "type": "uniform",
        "segmentGranularity": {
          "type": "period",
          "period": "P1M",
          "timeZone": "EST"
        },
        "queryGranularity": {
          "type": "period",
          "period": "P1M",
          "timeZone": "EST"
        },
        "rollup": true,
        "intervals": ["2022-10-01T05:00:00.000Z/2022-11-01T05:00:00.000Z"]
      }
    },
    "ioConfig": {
      "type": "index_parallel",
      "inputSource": {
        "type": "druid",
        "dataSource": "origin_datasource",
        "interval": "2022-10-01T05:00:00.000Z/2022-11-01T05:00:00.000Z"
      },
      "appendToExisting": false
    },
    "tuningConfig": {
      "type": "index_parallel",
      "maxNumConcurrentSubTasks": 60,
      "forceGuaranteedRollup": false,
      "splitHintSpec": {
        "type": "maxSize",
        "maxNumFiles": 200
      },
      "partitionsSpec": {
        "type": "dynamic",
        "maxRowsPerSegment": 5000000,
        "targetRowsPerSegment": 5000000
      }
    }
  }
}```

hmm….the new data source is derived by rolling up the old data source? Did the old data source have metrics?

I’m wondering if the time offset of 5 hours is causing it to create two month segments…

Where the second segment only includes 5 hours of source. 

Sorry, I don’t think I was clear, the time interval is using an offset of 5 hours from the beginning and end of the interval. I’m wondering whether queryGranularity of P1M is truncating them into two different months. But the time interval mostly includes data of for one month. Then I realized, the time zone is the reason for the time shift. So probably not a good theory unless there’s a bug.

&gt; hmm….the new data source is derived by rolling up the old data source? Did the old data source have metrics?
“Yes” to both questions :slightly_smiling_face:
&gt; Then I realized, the time zone is the reason for the time shift.
Yep.

Plus, I can confirm that the same ingestion spec worked for other months (2022-08 or 2022-09, for example) and querying original datasource and the new one produce the same output.

Based on my observations, it’s relatively random. If we ingest 30 days worth of data with P1D granularity, some months might have only 15 (or 10, or 25) days worth of *queryable* data (segments are there, but they produce no data - just like that example above for that new datasource with P1M granularity). Marking those “empty” segments as unused, issuing a kill task and then re-ingesting the same data *usually* fixes the problem, but sometimes it takes 2-3 re-runs to actually see the data.
Another thing: it doesn’t seem to matter what the ingestion source is: the original DS (daily granularity) was ingesting from S3 and the new one (monthly) ingests from daily DS - both were / are having the same issue.

It’s worse for this one month (2022-10) - I tried re-ingesting it 4 times already and still don’t see any data.

In the interest of figuring out if this is a metric calculation bug or if the segment is somehow incomplete.
Do the counts match between source and rollup?  `count(*) from source` vs `sum("count") from rollup` for that period?

Large number for the first one and empty result for the second one.

(this is with the time constraint of 2022-10).

Getting back the data for previous month (that one has been re-ingested twice to make it work).

Are you familiar with the segment-dump tool? <https://druid.apache.org/docs/latest/operations/dump-segment.html>
Let's see if the segment(s) in question have data and perhaps inspect a few rows to see how they line up to the source.

Haven’t seen that tool before, but will give it a shot in a bit!

I'm also searching for bug fixes since 0.21 that might be related. This one sounds like a similar situation: <https://github.com/apache/druid/pull/11189>

I'll post any others I find. Have a look to see if they might match your situation.

I dumped one of the segments and the data seems to be there. Here’s the first few lines from the output (dump) file:
```{"__time":1664600400000,"metric1":19.905000686645508,"metric2":null,"metric3":0,"metric4":0,"domain":"<http://domain.com|domain.com>","count":1,"opportunities":0,"user_id":xxxx,"video_id":0,"loads":1,"media_type":"video"}```
(I truncated / edited the output, but I can see correct values in the original dump).

This one seems to hit the mark: <https://github.com/apache/druid/issues/11410>

It does look like a similar issue. Thank you, Sergio, I’ll try re-ingesting using suggested workaround.

Doesn’t look like it helped - still getting no data back. If we were affected by this bug, wouldn’t we see 0s for our metrics? But we can definitely see properly (as in they were created and are non-empty) ingested data that’s converted to segments. It’s possible that my interpretation of this bug is incorrect.

You're right, the segment data seems okay which is not the same as the bug. Scanning the segment is returning nothing though... does `SELECT count(*) FROM rollup WHERE &lt;time condition for that interval&gt;`  return a count?

Nope, not for the date range in question.

I SSHed into one of our historicals and confirmed that segments for this date range are there.

So I guess historicals have been “instructed” correctly by ZK to load those segments from deep storage.

Trying another angle, I'm not very familiar with how __time works under different time zone. My understanding is that it will store the UTC time for it. The value in that sample row corresponds to 2022-10-01 05:00:00 UTC which is 2022-10-01 00:00:00 in EST. What I'm not sure about is how the filter condition translates at runtime. What time filter are you using when querying?

I’m using `__time &gt;= '2022-10-01T05:00:00.000Z' and __time &lt; '2022-11-01T05:00:00.000Z'`

It is the same (well, minus the month value) as in other successfully ingested tasks.

okay, so essentially the query conditions are always on UTC.

... still thinking...  seems like a corrupt segment file. But how so? Perhaps another segment-dump with `--dump bitmaps --decompress-bitmaps` could shed some light.

Cool, will do. But yeah, it’s been a really puzzling issue for us as well.

We plan to migrate to the latest version, but we’ll have to do that gradually, with proper testing in place, so it’s going to take some time. Not that it necessarily fixes this issue, but I imagine there’s a non-zero chance that it might.

Yeah, but it would be good to know what is happening here regardless. I keep coming back to the timezone, I understand this works for other time frames , but I'm wondering if this could be a boundary condition, so I'll keep asking about this.  Do you set `sqlTimeZone` in the query context in any of these tests?

are the number of segments significantly different for this month  (2022-10)?

As compared to the previous month in that datasource? Not really: 870 for 2022-09 and 929 for 2022-10.

FWIW, I was able to successfully ingest that problematic date range. No significant changes’ been made - I increased the number of concurrent subtasks and reduced how many middle managers processes we allocate for batch ingestion. Doesn’t seem like that should affect the outcome, but still.

If I get this correct…your __time is in UTC and so are your query filters but your segment granularity and rollup are in EST?

is the druid source you are trying to rollup on also ingested with segment granularity in EST?

Hi Vijay, yes and yes (to both questions).

I am able to see this error in my setup
```2023-01-16T04:01:33,366 ERROR [task-runner-0-priority-0] org.apache.druid.indexing.common.task.IndexTask - Encountered exception in BUILD_SEGMENTS.
org.apache.druid.java.util.common.ISE: Failed to get shardSpec for interval[2019-08-19T05:00:00.000Z/2019-08-26T05:00:00.000Z]
	at org.apache.druid.indexing.common.task.ShardSpecs.getShardSpec(ShardSpecs.java:57) ~[druid-indexing-service-2022.11.0-iap.jar:2022.11.0-iap]
	at org.apache.druid.indexing.common.task.NonLinearlyPartitionedSequenceNameFunction.getSequenceName(NonLinearlyPartitionedSequenceNameFunction.java:49) ~[druid-indexing-service-2022.11.0-iap.jar:2022.11.0-iap]
	at org.apache.druid.indexing.common.task.InputSourceProcessor.process(InputSourceProcessor.java:107) ~[druid-indexing-service-2022.11.0-iap.jar:2022.11.0-iap]
	at org.apache.druid.indexing.common.task.IndexTask.generateAndPublishSegments(IndexTask.java:922) ~[druid-indexing-service-2022.11.0-iap.jar:2022.11.0-iap]
	at org.apache.druid.indexing.common.task.IndexTask.runTask(IndexTask.java:526) ~[druid-indexing-service-2022.11.0-iap.jar:2022.11.0-iap]
	at org.apache.druid.indexing.common.task.AbstractTask.run(AbstractTask.java:169) ~[druid-indexing-service-2022.11.0-iap.jar:2022.11.0-iap]
	at org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexSupervisorTask.runSequential(ParallelIndexSupervisorTask.java:1199) ~[druid-indexing-service-2022.11.0-iap.jar:2022.11.0-iap]
	at org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexSupervisorTask.runTask(ParallelIndexSupervisorTask.java:532) ~[druid-indexing-service-2022.11.0-iap.jar:2022.11.0-iap]
	at org.apache.druid.indexing.common.task.AbstractTask.run(AbstractTask.java:169) ~[druid-indexing-service-2022.11.0-iap.jar:2022.11.0-iap]
	at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:477) ~[druid-indexing-service-2022.11.0-iap.jar:2022.11.0-iap]
	at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:449) ~[druid-indexing-service-2022.11.0-iap.jar:2022.11.0-iap]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_261]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_261]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_261]
	at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_261]```


this happens only when I set rollup to EST. Not when only the segment granularity is set

<@U04HX4YNCEQ> the following works on my setup
```"granularitySpec": {
        "type": "uniform",
        "segmentGranularity": "WEEK",
        "queryGranularity": {
          "type": "period",
          "period": "P1W",
          "timeZone": "America/New_York",
          "origin": null
        },
        "rollup": true,
        "intervals": []
      }```
can you try making the segment granularity just “MONTH” and the query granularity as you have now? My thinking is if the above works for you then it would be ok as segment granularity does not affect the result

We’ve migrated our `0.21.0` cluster to the latest (`25.0.0`) version and we’re experiencing an issue where our historical processes get restarted when trying to process long-running queries. We haven’t changed anything in our Druid config.
Observations:
1. Historical pods exit with status code of either 137 (sigkill) or 143 (sigterm)
2. Datadog metrics for these pods don’t show any apparent OOM events (not very different from what we observed before the migration)
3. Pods get restarted randomly - sometimes it’s just 1 or 2 and sometimes all of them at the same time.

Just to close the loop here. This issue got sorted after increasing the number of http threads

Just to add a bit to why this happened. Druid 25 defaults to http based segment announcement, task management and coordinator/historical communication. This puts more work on the http threads and hence the need to increase them.

We really appreciate all the help we’ve received. Thank you, <@U03BJAMQ4H5> and <@U030K4UM3H7>.

Hi, that is all there AFAIK. You should know that that contrib extension is not part of the distribution and is highly experimental (see warning at the bottom of the page you linked). It requires hadoop and has a bunch of ways to shoot yourself in the foot. There is work being done on a materialized view mechanism based on the new MSQ architecture.

how small is the cluster? how small is the data? is there enough space on historicals? (see Service tab in console). How long has it been stuck for?

ah… I saw it in the log, when coordinator tried to ping 1 of the historical, it got an error. After I restarted that historical, caches are propegated immediately.

just upgraded here, like 24h ago, and no issue noticed

What problems are you encountering <@U031F9Q5D1P>? Any symptoms / messages you can share?

With the exact same specs, I’ve now got huge lag.
Even if I scale up taskCount/workers to match kafka topic partition numbers, I’ve got lag.