Team - We have several dictionaries but don't want to load all of them in each realtime ingestion task as they take up unnecessary memory and we need to increase the heap size every time a new dictionary is added. Is there a way to avoid loading dictionaries or load them selectively in realtime ingestion tasks? The batch ingestion tasks (classic or MSQ) do not load the dictionaries since they don't support querying. Is this understanding correct? Thanks, AR.

Not that I am aware of. Adding it to the list of "things that don't scale well when using a middle manager for kafka intake". Query performance is another big one on it. The good news, all of this can be avoided by routing

index_kafka

tasks to an indexer instead of a middle manager: https://druid.apache.org/docs/latest/design/indexer/ - is marked as experimental, but we are using it in production for several years. An indexer is a middle manager that doesn't spawn a jvm per task, i.e. you can share lookups and the query cores between tasks. I am using a

equalDistributionWithCategorySpec

to route

index_kafka

to it exclusively, the rest stays on middle managers.

Thank you Hagen. We didn't look at indexer as it was still experimental when we started with Druid. I will read up on this in more detail. Does indexer run all the

index_kafka

jobs in a single JVM on different threads? Or do we need to run a indexer process per Kafka job? Thanks, AR.

it's a thread per task. really exactly like middle manager, but not spawn a distinct jvm. I run 24 tasks per indexer on my bare metals...

Cool. I need to dig into this. I presume MMs & Indexers running alongside on the same host, correct?

yes, that's possible. but be ultra-sure to not share storage. they will step over each other hard.

Thank you for the tip. 🙂 By storage, are you referring to the path specified by the

druid.indexer.task.baseTaskDir

config?

I sent you some helpful info via DM, considered it too long for a public thread

With this

We have several dictionaries

you mean lookups ?

lookupLoadingMode=ONLY_REQUIRED
lookupsToLoad=["foo","bar"]

Thanks Karan. Above is exactly what I was looking for. Where do I specify these in the Supervisor spec? I cannot find any info in the Druid docs. I found some details in the below Imply docs which mentions that these should be specified as

context

parameters but there is no

context

field in a Supervisor spec. Imply Docs: https://docs.imply.io/latest/druid/ingestion/tasks/ Also, from which version of Druid are these supported? We are running v27.0.0 currently. Thanks, AR.

You can put it in supervisor context

context:{ "foo":"bar"}

Druid 31 has these changes.

Thanks Karan. long way to go for v31.0.0 😞 I will give it a shot in our current version and see I get lucky. When I view the Druid supervisor spec on the web console, I can see

context: null

in it. May be it will work.. It will save us a lot of pain as we have some large lookups and we need to consider them when allocating memory for the realtime ingestion tasks..

If you do decide to upgrade which I strongly suggest you should, go directly to d32