Apache Druid

what kind of partitioning? it happened to me, fixed if I switch to dynamic

append is possible only in dynamic partitioning….even with dynamic you will get a lock conflict if multiple tasks write to same time chunk

Looks like a very small POC cluster, you l’d probably need to make a guess without knowing additional details about data size, query concurrency requirements , ingestion troughput etc. There is a guide here : <https://druid.apache.org/docs/latest/operations/basic-cluster-tuning.html|https://druid.apache.org/docs/latest/operations/basic-cluster-tuning.html> . 
You may need to make some guesses for example if I had these constraints I’d start with requests of 2vCPU 6 GB for master, approx 4 vCPU 8 GB for query , and approx 5 vCPU 18 GB for data nodes . Note you may need to request milicpus slightly smaller than these numbers for data pods. Set only your max heap sizes based on the link above. For data nodes you may need to carv out some RAM for ingestion peons as well as direct memory buffers as well. The idea is to set max heap for everything without setting min heap then add the Prometheus emitter extension to be able to monitor your workload . Once you have this initial setup start running some ingestion and queries and monitor heap sizes for processes and iterate on fine adjustments from there. 
For example you may see the master node does not need 6 GB , or you may see lots of GCs on your peons and peon heap fully utilized . That’s a sign you may want to reduce the resources on the master and give more to data nodes. 
this may be quite time consuming, and depending on your workload may not be enough capacity for some of the processes . My recommendation would be to start with a more generous hardware setup and reduce from there based on utilization metrics. I would start with separate node pools for master , query and data nodes 4 vCPU 16 GB for master, 4 vCPU 8 GB for query and 8vCPU 32 GB for data.  The are general purpose instances you should find in azure easily. Set affinity for your pods to make sure they end up on these node pools. Set only your max heap sizes and start ingestion querying management tasks etc . Monitor usage( memory, vCPU) . This would most likely be a stable system, look at under utilized heap sizes vCPU for pods etc , then start optimizing by setting limits on pods and lowering heap sizes and compare performance.
In the 25 release of Druid there is improved startup that could self configure druid processes based on resources available on the box: <https://github.com/apache/druid/pull/13365|https://github.com/apache/druid/pull/13365> but that’s for running all processes on one node. You may find it useful in dry run mode to guess some of the configs. I did not look deep enough to see if it takes vCPU into consideration but it may be worth also looking into. 

Thank you, Cristian for the detailed reply.
We will try this.

```2023-02-08T15:59:41,750 WARN [HttpClient-Netty-Boss-0] org.jboss.netty.channel.SimpleChannelUpstreamHandler - EXCEPTION, please implement org.jboss.netty.handler.codec.http.HttpContentDecompressor.exceptionCaught() for proper handling.
java.net.ConnectException: Connection refused: /10.150.53.58:8083```

hi I was wondering if this issue got some traction? I m experiencing same issue just wanted to hear how this is sorted where to look for fix.

Heavy groupby queries could be a possibility ; need to ensure you have enough heap + direct memory on your brokers

It may be even better with range or multi_dim! <https://imply.io/blog/multi-dimensional-range-partitioning/>

And yeah: it can be pretty imperfect! It basically rolls up any rows that happen to be read by the same task; if you have a lot of tasks reading jumbled-up data then it likely won't roll up very well.

The SQL query is basically like this:
```SELECT COUNT(*) as total,
  SUM(impressions) AS impressions
FROM (
    SELECT dim1,
      dim2 SUM(impressions) AS impressions (
        (SUM(some_metric) * 1.0) / SUM(impressions)
      ) AS takeRate ---notice the alias here 
    FROM druid_datasource
    WHERE date BETWEEN '20230203' AND '20230203'
      AND country_desc IN ('Algeria')
    GROUP BY 1,
      2
    HAVING takeRate BETWEEN 0 AND 1 ---notice the usage of alias in having clause
  )
WHERE impressions BETWEEN 1000 AND 2000```


When druid console's smart query limit toggle is on, the having clause is present in the sub query.
```"postAggregations": [],
	"having": {
		"type": "filter",
		"filter": {
			"type": "and",
			"fields": [{
				"type": "expression",
				"expression": "(((\"a1\" * 1.0) / \"a0\") &gt;= 0)"
			}, {
				"type": "expression",
				"expression": "(((\"a1\" * 1.0) / \"a0\") &lt;= 1)"
			}]
		},
		"finalize": true
	},
	"limitSpec": {
		"type": "NoopLimitSpec"
	},```


When the console's smart query limit toggle is off, the having clause isn't present in the sub-query anymore.
```"postAggregations": [],
	"having": null,
	"limitSpec": {
		"type": "NoopLimitSpec"
	}```
Instead, what I  noticed was that the having clause was re-written to a where clause in the outer query.
```"filter": {
		"type": "and",
		"fields": [{
			"type": "bound",
			"dimension": "impressions",
			"lower": "1000",
			"upper": "2000",
			"lowerStrict": false,
			"upperStrict": false,
			"extractionFn": null,
			"ordering": {
				"type": "numeric"
			}
		}, {
			"type": "bound",
			"dimension": "v0",
			"lower": "0",
			"upper": "1",
			"lowerStrict": false,
			"upperStrict": false,
			"extractionFn": null,
			"ordering": {
				"type": "numeric"
			}
		}]
	}```


In our case, when the query has smart query limit toggle on, it returns some results. When the toggle is off, it doesn't return any results.

that is strange, why would rewriting a HAVING on the inner query to a WHERE on the outer query change anything? It seems like a chill thing to do...

image.png

The fact that you are calling it `smart query limit toggle` tells me that you are on an older version of Druid. In today's Druid it was renamed to "Limit inline results" and it is a checkbox now

Do you see the same behavior in Druid 25? Do you have a way of checking?

I have Druid 25 in front of me, I am trying my best to reproduce this but can not

What does the explain plan look like with and without the limit enabled?

the only diff is the limit spec (as you would expect) and the UUID

Are you able to upgrade your Druid to a later version?

We can quickly try that. The commit number on that extension is not a lot, so I didn’t check but I should have.

<https://github.com/apache/druid/blob/master/processing/src/main/java/org/apache/druid/segment/nested/StructuredData.java#L129|https://github.com/apache/druid/blob/master/processing/src/main/java/org/apache/druid/segment/nested/StructuredData.java#L129> is the current comparison we are using

it's not perfect, and has a chance of collisions

it should maybe be configurable in the case not having collision is preferred over performance

then it could always do something like keep a serialized version of the value around to do a full comparison

Hey guys,
I’ve been running native batch ingestion for a while now  and have had subtasks tasks randomly run indefinitely. The task logs indicate SUCCESS but when i go to the /druid/indexer/v1/task endpoint for the task, all the statuses there are RUNNING. Here are some extra details:
• My druid cluster is on EKS with the help of the Druid operator
• There is only one coordinator running and I noticed there’s no restarts from it
• I can ping the middlemanagers from the coordinator instance and get results back fine
• Restarting only the coordinator when this occurs does nothing to solve the issue
• This issue occurs even if both main task and subtask are on the same middlemanager
• The master task will continually log “Found [1] expired supervisor tasks” if it has a subtask running indefinitely
• The logs of an indefinite subtask mention “Starting graceful shutdown of task”. The logs between a normal subtask and indefinite running one differ a bit because an indefinitely running one does NOT log this: 
```2023-02-09T06:15:58,602 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle$CloseableHandler - Closing object[org.asynchttpclient.DefaultAsyncHttpClient@2c846d55]
2023-02-09T06:15:58,604 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [INIT]
Finished peon task```
The only resolution is to kill a task and when it restarts it finishes immediately. Is there any druid configuration I can set to automatically handle something like this?

Generally auto-compaction cannot run against a time chunk that is currently being actively ingested into.  The conflict is against the entire time chunk, not just individual segments.

I am dealing with this issue on anothe cluster right now, I have been told to be careful about mixing segment granularities with real-time ingestion.

So either:

a) set up your auto-compaction with an offset that is old enough to never operate on a time chunk that is being ingested into (as limited by lateMessageRejectionPeriod).  E.g. set the compaction offset to P31D and set lateMessageRejectionPeriod to 30D ... or set compaction offset to 2D and lateMessageRejectionPeriod to 1D

or

b) Leave your auto-compaction at HOUR granularity, with offset 2H or so, which should allow it to run when it finds a time when that hour is not being ingested into.

I may not have gotten this 100% right as I am still learning about this, hopefully others will confirm or correct.

-- John

Option b is what we had before the auto-compaction could change the granularity and indeed the compaction works. But then it is meaningless since what we want is to rollup the data in less segments as it makes queries more than a hundred times faster by our benchmarks with our real data.

Now for option a...
I haven't set anything on `lateMessageRejectionPeriod` since we deal with late messages by ourselves way before it reaches Druid. I had no idea that it could interfere with auto compaction so I'll try that. Thanks

Also, there is one more thing to keep in mind ...

Auto-compaction offset is from the LATEST TIME CHUNK in the datasource, not the CURRENT DATETIME ... so if you are ingesting any data with future event dates, this will affect which time chunk compaction tries to start with.

Check the START/END dates of your segments to see if you have any with future dates in them.  If you do, then ...

... the earlyMessageRejectionPeriod parameter allows you to limit the early data in this case.

That is the second problem. The start and end date for the Kafka ingested segments are the stars and end of the month. Even though they only have data regarding a single hour

Your auto-compaction offset is set to 0S, so no delay, it will compact the current time chunk from HOUR to MONTH.  I would try changing that to P1M so that it clears the current month and will only start compacting the data from HOUR to MONTH after you move into the next month.

I'm not exactly sure how to "fix" the current month though ... maybe run a manual compaction for the current month (using inputspec interval) to change the segment granularity back to HOUR?

But if I set the offset to one month then I'll have an entire month worth of uncompacted segments, which is exactly what I am trying to prevent

I did set it back to 2 hours to test your suggestion as I mentioned above

If you are concerned about fragmentation within the hour timechunks, then I think you can run manual compaction with HOUR granularity and specify the interval for the current month.

Currently I don't think you can create multiple levels of auto-compaction (there is an upcoming "life-cycle management" feature that is supposed to address that) ... so the manual compaction of HOUR segments for the current month you may have to set up on a cron job.

Untitled

Looks like the error is happening specifically in <https://github.com/apache/druid/blob/druid-25.0.0/extensions-core/kubernetes-extensions/src/main/java/org/apache/druid/k8s/discovery/K8sDruidNodeDiscoveryProvider.java#L227-L249|https://github.com/apache/druid/blob/druid-25.0.0/extensions-core/kubernetes-exten[…]g/apache/druid/k8s/discovery/K8sDruidNodeDiscoveryProvider.java>

I did a complete destroy/rebuild of the druid deployment including all PVCs but no luck.

also went all the way to giving the overlord service account cluster admin

Hmm. is it the full logs of overlord/coordinator?

they are separate pods.  i'm not using the combined pod

these are the logs saved from the job overlord creates

here is a full log snippet from the job save in s3

Untitled.txt

the scheduled job being run is `{ "type": "noop" }`

that's what it looks like. one option is to build a custom druid docker image with those jars inside. And then use that docker image instead.

Thanks! I thought using the init-conatiner should have the same effect as the jars are mounted in the folder before the actual druid pod is started. When add the new folder "protobuf-extensions" to plugin list (which according to the documentation you DONT have to do), you can  even see the jars getting loaded - so they must be there or?
`INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [protobuf-extensions], jars: kafka-protobuf-provider-6.0.1.jar, kotlin-stdlib-1.4.0.jar, wire-schema-3.2.2.jar`
Perhaps there is some other issue but currently im out of ideas. I tested with druid 24/25 and I checked newer version of the jars as well.

Just for reference if someone is trying something similar (despite me not having success yet) - this is the kustomize patch. The jars are downloaded in the init-script which is within a configmap:
```apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: ignore
spec:
  template:
    spec:
      initContainers:
        - name: "druid-init"
          image: "busybox:1.36.0"
          command: ["sh", "-c", "sleep 10s;/scripts/init-script.sh"]
          volumeMounts:
            - mountPath: /scripts
              name: init-script-volume
            - mountPath: /protobuf-extensions
              name: protobuf-extensions
      containers:
        - name: "druid"
          volumeMounts:
            - mountPath: /opt/druid/extensions/protobuf-extensions
              name: protobuf-extensions
      volumes:
        - name: init-script-volume
          configMap:
            name: druid-init-script
            defaultMode: 0755
        - name: protobuf-extensions
          emptyDir: {}```

yeah. the jar is getting loaded. maybe it's a different problem. You can try to reproduce this locally without k8s.

I tried directly locally (via quickstart guide) and unfortunately I still had the same issues - so probably its not supposed to be.

Context: We are using Druid 0.21.0 version with 8 data nodes, 2 master and 3 query

Broker runtime.properties
```#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#   <http://www.apache.org/licenses/LICENSE-2.0>
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied.  See the License for the
# specific language governing permissions and limitations
# under the License.
#

druid.service=druid/broker
druid.plaintextPort=8082

# HTTP server settings
# Modify this number base on Number of Data instances 350 * 2 instances &lt; 800 in historical
druid.server.http.numThreads=350
druid.server.http.enableRequestLimit = true
druid.server.http.defaultQueryTimeout = 60000
druid.server.http.queueSize = 30000

# HTTP client settings
# Modify this number base on Number of Data instances
druid.broker.http.numConnections=350
druid.broker.http.maxQueuedBytes=15MiB

# Processing threads and buffers
druid.processing.buffer.sizeBytes=1GiB
druid.processing.numMergeBuffers=8
druid.processing.numThreads=4
druid.processing.tmpDir=var/druid/processing

# Query cache disabled -- push down caching and merging instead
druid.broker.cache.useCache=false
druid.broker.cache.populateCache=false```

• Check for segment counts. Have the number of segments increased. 
• If you have metrics dumped somewhere, check jetty metrics  and broker metrics to see if you can get more clues. Metrics documentation: <https://druid.apache.org/docs/latest/operations/metrics.html#broker>
• Try to isolate the query which is causing the issue.  Generally subQueries can cause this. 
You can also take flame graph of one broker for like 5 minutes to see where the bottle neck is as well.

<https://support.imply.io/hc/en-us/articles/360033747953-Profiling-Druid-queries-using-flame-graphs>

great points, thanks a lot <@U0315MQKEHK>! very helpful - we will try it out and update! :bow:

Excellent advice.  Although I would have though segment counts would affect coordinator and historical performance more than broker, but I could be wrong.  Always good to compact and keep counts down where possible.

yeah, segments counts would impact historical performance and query would just merge the responses.. so lesser segment count would be better right?

yes ….  faster queries due to less segments to scan

broker also keeps the segmentId, rowSignature -&gt; historical in memory, so that also eats up broker heap.

understood.. so more the segments, more the broker heap usage?

yes ….as broker needs to keep track of segments in all historicals

Sorry one more question: We are thinking to scale our data nodes from 8 to 12.. does that mean we need to scale the query nodes too vertically? The reason is each query node needs to connect to all data nodes right, so need more processing power?

So, when data nodes are horizontally scaled, scale query nodes vertically is correct understanding?

do check this out : <https://druid.apache.org/docs/latest/operations/basic-cluster-tuning.html#number-of-brokers>
&gt;  So, when data nodes are horizontally scaled, scale query nodes vertically is correct understanding?
Yes, we would look at the `numThreads`  and `mergeBuffers` for groupBy queries etc

As brandon pointed out, you really donot need to scale the query servers for adding 4 more historicals

Literally was dealing with this all day yesterday.  I was getting this inside the K8s jobs that were launching as a replacement for middleManager.  Is that where you were seeing it too?
CC <@U04NHCE780G>

Here is what I was told by the `MM-less` feature developer (paraphrasing):
```oh yeah ignore those 
so it thinks its an overlord but its not, service discovery doesn't need to happen in the peon task but since its another extension I havent been able to get to it yet
...
so service discovery only needs to happen for long lived processes, the peon task…because it inherits from the overlord has the service discovery as well but its useless```


My issue is that tasks, even `{"type":"noop"}` are succeeding but reporting FAILED in the UI and I never ended up being able to figure out why

I have exactly that problem. It looks like the task is even putting data on my s3 deepstore, but never reporting successful in Druid

I did find out that the role needed "pods/log" as well to display logs in the UI, if anybody else has come across that one.

<@U04N77Z3T0X> yes that is identical to my setup/issue.  I get logs in the S3 bucket as well and thats where I see the above errors which are supposedly benign.  I had the setup working previously but when I started testing some changes to the helm chart I ended up in that state.  I went all the way to giving full cluster admin permissions to the role and it didn't help

we are using mm-less druid in production.  I did push up a few fixes for this, one has been merged the rest are not really being looked at by anyone else.  I can honestly just push up one large changeset that fixes a lot of issues next week and you guys can fork it if you want to use the feature.

I have a few more locally, like upgrading clients, working better with druid montiors etc…

i am having a hard time getting anyone in the druid community to look at them and i don’t want to keep rebasing, so ill just push up a big patch and hopefully someone in the community will take interest

Thanks <@U04NHCE780G>, I actually tried a custom built with this one <https://github.com/apache/druid/pull/13668> which didn't help sadly

so what is weird, is that the peon task is not reporting back to the overlord correctly right?  Let me look into that more today, I haven’t had much free time to work on druid this week.

Happy to try any other patches on my cloud

I wanted to go with a custom build too. At this point, not sure if going back to ZK isn't the best approach

For our deployments this all seems to work….

we run zk-less + mm-less in production here, but our build is pretty forked, ill push up our changes next week

What k8s version are you running it on? I'm on 1.25

in one big PR so at least people can use it if they want

we have all different versions of k8s, we run a lot of druid clusters

1.23 is the highest version we are running on

somewhere between 1.16 and 1.23 is what we are running druid on k8s

I was initially suspecting role permissions because I discovered it needed "pods/log" to display logs in Druid, but I'm not seeing any exceptions thrown so I guess that's not it

This one seems to be merged: <https://github.com/apache/druid/pull/13668>

i upgraded the client too, haven’t pushed that patch fully, just got it to work on our clusters yesterday….the k8s client was pretty old

but for the task success part, when the task completes, it calls a UpdateStatusAction which makes a call back to the overlord to tell it, hey I am done and i finished successfully.  Then the overlord records in the db and should have the correct state. I think something might be going on there.

if that fails would it not throw a clear exception tho? My peon logs are clean

yeah they are clean but the call is not being made for some reason successfully

<https://github.com/apache/druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/common/task/AbstractTask.java#L186-L206>

<https://github.com/apache/druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/common/task/AbstractTask.java#L194>

<https://github.com/apache/druid/blob/master/extensions-contrib/kubernetes-overlord-extensions/src/main/java/org/apache/druid/k8s/overlord/KubernetesTaskRunner.java#L257-L260|https://github.com/apache/druid/blob/master/extensions-contrib/kubernetes-overlord[…]in/java/org/apache/druid/k8s/overlord/KubernetesTaskRunner.java>

So the TaskRunnerUtils should get called, to update the status….i mean it sucks, but you could put a debug log there to show its called??? with the status

ill push up what i have on my local 25.x branch in one big PR for you guys, today / early next week.  But we are running it here (in prod) and everything is working fine.  Ours is pretty forked though.  So ill get you guys exactly what we have here.  I just don’t want to keep making individual PRs and have them rot, so I haven’t been pushing my changes upstream.

I could try that sure. Will take a bit to verify that

I'm just wondering what would happen if it's not correctly configured as an encapsulated task? Could that be it? Then first block you referenced wouldn't execute

I'm at a Beta stage now with my prod, so I can try things.

I would love to see that pr. I was thinking of making a few changes myself.

Hey <@U04NHCE780G> can you share the pr with this fix ?

1. Overlord launches peon, peon does stuff, peon reports back to overlord its state (success / failure ) when it finishes.  

2. As a backup, the overlord also checks the state of the job, and records that when it finishes.  This is to handle the case where the peon may have tried to talk back to the overlord but could not.  So we have a the overlord also register the state

I was looking at the logs <@U04MRT25PSP> had for his overlord and noticed something interesting

for a given job, it first said it recorded the status as “SUCCESS” &lt;--- from the peon talking back to the overlord

Then for that same job, later in the log lines marked it as a failure

So how does the overlord check to see if the job has successfully completed or failed.  It monitors the job phase and looks for a string :confused:

<https://github.com/apache/druid/blob/master/extensions-contrib/kubernetes-overlord-extensions/src/main/java/org/apache/druid/k8s/overlord/KubernetesTaskRunner.java#L194-L202|https://github.com/apache/druid/blob/master/extensions-contrib/kubernetes-overlord[…]in/java/org/apache/druid/k8s/overlord/KubernetesTaskRunner.java>

So if the phase does not match
```Succeeded```
it will fail.

i see it, the k8s client is not guaranteed to work with 1.25

I have a patch for the client upgrade, that will work with everything up to k8s server 1.25.3

this is exactly why we don’t see the issue, the client stops working after 1.23….  I’ll get you guys a PR + patch today

it actually worked for 1.24.6 for what its worth

Thanks a million. So I'm guessing this client upgrade won't do it? <https://github.com/apache/druid/pull/13759>

upgrading to 1.25.6 is turning out to be a huge regret

<https://github.com/fabric8io/kubernetes-client#compatibility-matrix>

`everything up to k8s server 1.25.3` --&gt; really hoping that patch versions would 'likely' maintain compatability

<@U034D4J02SG> there is more i have to add to that patch…..there were a bunch of runtime things that broke, i just got it to work yesterday on our clusters.  I’ll push up those changes into the PR as well

ill try to get you guys something today, or over the weekend.  But i think I see exactly what is wrong here.

Thanks for looking into it, it's greatly appreciated and for the original PR. This is going to be an interesting one to maintain, looks like it'll require dep updates often enough

yeah but I am hoping as more people use it, the community can get involved and we can own it together.  The thing with this upgrade in particular was it was a major version bump to the client, which means that a lot of API’s changed.  I think with minor version upgrades to the client, it should be simply bumping your pom and building.

<@U04NHCE780G>, any idea when 26.0.0 is getting released. We are facing the similar issue.

It’s corrected now had issue with metadata database

Unable to see middle manager in cluster mode

Specifically, I'm trying to achieve this:
```new TopNQueryBuilder().virtualColumns(new ExpressionVirtualColumn(
                "uniqueRecordIdentifier",
                "concat(\"comumnNameA\",\"columnNameB\")",
                ColumnType.STRING,
                ExprMacroTable.nil()))
        ...
        .build();```

I forgot to mention that I initially tried to locate the `org.hyperic:sigar:1.6.5.132` dependency in `mavenCentral()` repository.

What are you trying to do? If you're just trying to create Druid query objects so you can serialize them, you can ignore sigar (exclude the dependency). It isn't used in those code paths

As to ANTLR, I am not sure if we are tied to that older version in some way. That's an area of the code base I am not too familiar with

It's pretty workload-dependent so the information about your specific workload will be helpful!

A tip about data-gathering: in cases where the Historicals aren't dying, but are just getting a lot of allocation pressure, heap dumps (as big chunky point-in-time snapshots) don't always tell the story very well.

A couple of other options:

1) Better: heap _histograms_, from `jmap -histo &lt;pid&gt;`, taken repeatedly. These are much faster to take than heap dumps and show the same high-level info. Not the full object graph, but you get the same high-level object counts.

2) Best: a JVM allocation profiler. These hook into the JVM and track allocations continuously. Definitely the best way to figure out what's pressuring the allocator.

Another note: not sure what version you're using, but note that in recent versions (starting from 24) we improved the allocation profile of groupBys: <https://github.com/apache/druid/pull/12474>, <https://github.com/apache/druid/pull/12468>

This stuff will be helpful if you're doing a lot of groupBys and are on a version &lt; 24

Thanks Gian, that all helped! Histograms definitely were of use. and I'm working to figure out more about connecting a profiler to get some live allocation rate information

After some analysis, I'm starting to be suspicious of queries who have tons of IN filters, some of which are surprisingly large.

I'm seeing ThreadLocalMap objects that are pretty huge on query threads named as `QUERYTYPE_DATASOURCE_INTERVAL` Within this map are SpillingGroupers who have filtered aggregators and the InDimFilter seems crazy large. Measured in MBs. I'm a bit concerned that these are coming onto heap as Humongous Objects, making heap become fragmented. I see that you added a performance improvement in druid 24 that provides the filter values as UTF-8 out of the box. In my head this lines up with the timeline of this problem becoming more common for us after upgrading to Druid 24.

My question for you now is - what part of the query are these threads for? I'm thinking if we have many segments involved in a query located on a single node, if that node takes a query with these crazy IN filters, there could be a lot of these DimInFilter objects being created at the same time, causing my problems. Does that hypothesis have any merit in your opinion?

Unfortunately I'm only on the platform team, and our query services team would be the ones to put the clamps down on what kind of filters our users can come up with :sweat_smile: If I had the power I'd probably put some ground rules in place when it comes to the number of values in these filters to see if it helps at all.

another observation is that I feel like I'm seeing spilling groupers referenced by these processing threads (`QUERYTYPE_DATASOURCE_INTERVAL`) who are for what I assume are previous queries. For example, I see a reference to a grouper for a group by on datasource Y in a processing thread that is actually a timeseries for datasource x. I'm trying to wrap my head around the historical query execution code and how the processing thread pool is releasing resources, but these references stood out to me

the utf8 change does increase the memory requirement of IN filters, however, the utf8 version of the value set is shared across all segments for the query (so the # of segments on your server doesn't affect memory usage)

i expect the utf8 change would increase the memory requirement by about 50% (for IN filter)

if you wanted to test it, you could deploy a build to one server that omits the `valuesUtf8` stuff. i.e. delete the variable and also delete the code that uses it:

```      final Utf8ValueSetIndex utf8ValueSetIndex = <http://indexSupplier.as|indexSupplier.as>(Utf8ValueSetIndex.class);
      if (utf8ValueSetIndex != null) {
        return utf8ValueSetIndex.forSortedValuesUtf8(valuesUtf8);
      }```
With that code deleted, we'll fall back to using the utf16 `values`

Interesting idea! Although now that you point out that the utf8 version is shared, I'm now thinking this was a false flag due to my assumption that there was no re-use there. My search continues. I appreciate your inputs!

I think the problem is identified... there was a client out there running `none` granularity `timeseries` queries. I was so dialed in on large group by's that I never even noticed these popping up in request logs whenever we had instability. I got tipped off when looking at a heap histogram taken while a node was in the process of dying and there were millions of `DateTime` objects which got me wondering who the heck was doing insanely granular workloads. bet part is that it isn't even some type of sensor data or financial data where milliseconds even make a difference

Oh, yeah, those are gross. `none` basically means `millisecond`.

Note: if you do the queries through SQL, we plan these queries more safely since <https://github.com/apache/druid/pull/13206>

Although if you do them through native you can still foot gun

that's a great feature in #13206! We are still on native as our application team has built a whole services layer around that query language which is how our users interact with the cluster as of now. But if SQL is getting most of the attention going forward, it could be time for us to start scoping out what it would look like to migrate over. I appreciate the help you've provided in this search. The tip on histograms certainly made the difference!

glad that helped!

Certainly our way of thinking going forward (and also, just in general) is that SQL is the path for "I want the database to try to run the query the best way it can" and native is the path for "I know what I am doing"

In general we expect most new users would be using SQL, and existing users would migrate unless they really like the control they get with native

increase druid.indexer.fork.property.druid.processing.buffer.sizeBytes (max is 2 GB). If you increase this to 2GB and still see this error then reduce max rows in memory and also consider indexing only fields used in queries

Thanks! Trying now.
Out of curiosity, what's the reasoning behind that?

I don't think that increasing `druid.indexer.fork.property.druid.processing.buffer.sizeBytes` will help; if you are out of direct memory then increasing buffer size will only increase the demands on buffer memory.

Looking at your settings, your current buffer size (134217728) and count (1) is not too big relative to your direct memory size (4g). So I would guess the direct memory is being used by column merging. You can read more about that here: <https://apachedruidworkspace.slack.com/archives/C0309C9L90D/p1663951074038569?thread_ts=1663936671.637089&amp;cid=C0309C9L90D>

TLDR: if that's the issue, you can mitigate it in two ways:

1) Set `maxColumnsToMerge` using the guidelines from that thread I linked.

2) If this is batch ingest: Use MSQ / SQL based ingest instead of native batch ingest. It automatically tunes column merging to avoid running out of memory, so you don't need to worry about it

It is a batch ingest. I would use MSQ, but I'm using a custom agg function.

I'm trying with maxColumnsToMerge=1 just to see it work.
I'm surprised it's taking that much, as I only have 10columns. 5 of them are long.
I do however have a sketch in the measure.

Thank you thank you! That was it :slightly_smiling_face:

Good to hear! BTW, you can use custom agg functions in MSQ, albeit with a bit more work. You would need to define a SQL binding for your agg function

Yes, we're working on that :slightly_smiling_face:

Segments are stored on disk and memory-mapped, and are in a format that is "proprietary to Druid" in the sense that it is a format that is primarily used by Druid. It is, of course, an open format, since Druid is open source!

Data is processed in-place whenever possible, minimizing deserialization or copies. For details check out this talk: <https://www.youtube.com/watch?v=qSJ8_Lc1oAY>

Hi <@U0344FW86DD> just to clarify, mmaping a segment doesn't mean loading that file into main memory right? And druid will first identify the rows in the segment files based on meta-data and then loading only that particular portion of segment into memory and do the necessary calculations on-demand correct?

mmapping is <https://en.wikipedia.org/wiki/Memory-mapped_file>; it won't necessarily load the entire file into main memory, although it may, if you actually use the entire file and if you have free memory. the OS page cache (<https://en.wikipedia.org/wiki/Page_cache>) handles it

This can happen if you have mixed granularities in your various ingestion operators, and those mixed granularities overlap. In general it's not recommended to overlap mixed granularities.

Meaning: this error shouldn't happen if you stick to a single granularity for a datasource. Please let us know if it does, and provide any info you can about how you got into that state!

I think the problem was setting `druid_storage_storageDirectory` when i'm using S3 buckets for segments. So i think that when i deleted some segments, inside the local folder there were some leftovers. I deleted this configuration and create a new docker containers setup, and for now it seems go flawless

Another problem that i have now is this:
`DELETE /druid/coordinator/v1/datasources/{dataSourceName}/segments/{segmentId}`
When i call the following query its return me 500 http status, but the segment is deleted succefully. How can i check if the invalidation is succeded with an API call?

You could look at the `sys.segments` table (<https://druid.apache.org/docs/latest/querying/sql-metadata-tables.html#segments-table>) of Druid SQL

So is it a bug that this API return 500 and delete the segment too?

I am not sure, sometimes 500 is a bug and sometimes it is due to some transient error and you can retry on the client side. If it happens repeatedly, it is probably a bug

By default, new ingestions do start from the latest offset

if you're using an existing datasource you can reset the supervisor, that will reset the offsets to latest (or earliest, if `useEarliestOffset: true`)

What I did is made earliest offset falset

len of dimenstionSpec + metricsSpec 6513~ didnt work

Maybe need to check
Overload node XX:MaxDirectMemorySize?

Do things work if you submit your task through the web console?

The web console does use the same API, so if that works, then likely you need to adjust how you're using the API

<@U0344FW86DD> web console same...
When I crease dimensionSpec +metricsSpec count almost to 6500, tasks not started and give 404 fail.

But below some level, tasks are created well (I'm using index_hadoop) Thank you for help

Some level means dimensionSpec and metricSpec count... And I'm using hdfs

Error fixed <@U0344FW86DD>  problem was znode thank you

ah, that makes sense. the znodes can get pretty big when you have a large # of columns like that

one cool thing about the new SQL-based ingest (with MSQ) is the columns are no longer stored in znodes

so there are not limits imposed by znode size :slightly_smiling_face:

Wow I would check new SQL based ingest(with MSQ)

No they should not be any issues. 
MSQE uses ingestion tasks to process the data. 
Can you check if you are using a different tier for peon tasks. 'druid.lookup.lookupTier' is used to control that.

<https://druid.apache.org/docs/latest/querying/lookups.html#configuration|https://druid.apache.org/docs/latest/querying/lookups.html#configuration> has the configurations that are set via the middle manager

I have these configs set for all services
```  druid_lookup_lookupTierIsDatasource: "false"
  druid_lookup_lookupTier: __default```


I can see they’re the default values. Not sure why they were explicitly defined in the past.

```2023-02-14T16:33:49,202 INFO [main] org.apache.druid.cli.CliMiddleManager - * druid.lookup.enableLookupSyncOnStartup: false
2023-02-14T16:33:49,203 INFO [main] org.apache.druid.cli.CliMiddleManager - * druid.lookup.lookupTier: __default
2023-02-14T16:33:49,203 INFO [main] org.apache.druid.cli.CliMiddleManager - * druid.lookup.lookupTierIsDatasource: false```

Aah the ingestion is not waiting for lookups to load. 'enableLookupSyncOnStartup' is false.

druid.lookup.enableLookupSyncOnStartup	Enable the lookup synchronization process with Coordinator on startup. The queryable processes will fetch and load the lookups from the Coordinator instead of waiting for the Coordinator to load the lookups for them. Users may opt to disable this option if there are no lookups configured in the cluster.	true

Setting `druid.lookup.enableLookupSyncOnStartup: true` fixed, thanks <@U0315MQKEHK>

I wasn’t expecting this one to be the problem as my queries were working fine and my lookups are periodically refreshed from external sources, so the ‘sync on startup’ wasn’t looking like a solution here.

We have these changes applied in our fork for druid running mm-less k8s on our clusters on the 25.0 branch.

Also for PR 13804, I just pushed it up, might have to make small tweaks if the build breaks, but tests pass and the code is functionally correct.

You could check the coordinator logs to see if the segments for this datasource are being assigned to historicals

tail the coordinator log sometimes:
• the historical is not ping-able
• the number of segment files to move is too low

I check coordinator log not find some error logs but I try to switch coordinator:  i have two nodes to play coordinator so I kill one of then and master switch to another:at this point segments start to load  i happens serveral times. I need to trigger coordinator switch to start segments to load. this happens usually after historical nodes crash and recover

in web UI  historical's loadqueue is empty but datasource's segments  some unavailable AND it will recover by reboot coordinate

Mark the segments as unused and run a kill task

`druid.server.http.defaultQueryTimeout` is the config that controls this and it defaults to 30 seconds (30000 Millis). See: <https://druid.apache.org/docs/latest/configuration/index.html#indexer-concurrent-requests|https://druid.apache.org/docs/latest/configuration/index.html#indexer-concurrent-requests>

You probably want to set this in the common runtime configs as it applies to historicals and brokers as well as middlemanagers/indexers if using streaming. Also if you have a load balancer in front of your brokers or routers you will want to make sure it's set appropriately there as well.

You can also override it for specific queries through context settings if you don't want to override the setting globally. See: <https://druid.apache.org/docs/latest/querying/query-context.html|https://druid.apache.org/docs/latest/querying/query-context.html>

oh thank you but isn’t the default value of druid.server.http.defaultQueryTimeout is 300000 milliseconds (300 seconds)

Oh yeah it is 300 seconds (5mins, 300000 millis). Do you have any load balancers or reverse proxies in front of your queries? Do you know if you are getting a timeout from Druid or a network timeout? Sometimes it can be the network or load balancer that also has an idle connection timeout that can interfere if you queries are taking a while before returning the first/next bytes of data.

definetly i would check it out and response

I haven't seen this being set.  What I've seen normally done is to reduce heap size or direct memory settings.

now we are seeing this error in historical
```961.954: [Full GC (Allocation Failure) 961.954: [CMS965.859: [CMS-concurrent-mark: 6.859/6.866 secs] [Times: user=19.09 sys=0.41, real=6.87 secs]


1000.342: [GC (GCLocker Initiated GC) 1000.342: [ParNew: 460095K-&gt;460095K(460096K), 0.0000383 secs] 8337535K-&gt;8337535K(8337536K), 0.0001727 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
GC locker: Trying a full collection because scavenge failed
1000.342: [Full GC (GCLocker Initiated GC) 1000.342: [CMS1006.562: [CMS-concurrent-mark: 6.929/6.938 secs] [Times: user=16.74 sys=0.26, real=6.94 secs]```
<@U030KB0NXM0>

note that in Druid 25 (the latest release) you can start druid via the `bin/start-druid` script, and it will automatically determine reasonable configs given the available resources on the server

I would try this first and see if it solves your problem

if it doesn't then let us know :slightly_smiling_face: