Hi All, We have been facing issues due to the join performance & update limitation in Druid. Finally we experimented with Lookups by excluding them from the realtime ingestion tasks. We used a lookup with ~65K rows and and the join performance is promising. The idea is to store a a delimited string as the lookup value and split it up during run time to apply the necessary filters and fetch the required values using an index. We plan to have around 10-15 lookups with size varying from 50K to 3MM rows. The largest lookup would be ~2GB in size. Memory is not such a big concern as we will be able to bump up the heap size or DM size as necessary. Also, the lookups will only be loaded on demand by a manual trigger (no polling or periodic loading). There are 2 flavours of Lookups in the documentation - Globally Cached Lookups and Single Cached Lookup. Which one should be preferred in this scenario? We tested using the Globally Cached Lookups. Globally Cached Lookups We liked the fact that this lookup allows to filter the records while loading. So we can load data from a single source table into multiple lookups. The implementation seems to be a single Java Concurrent map containing a Concurrent map for each lookup. Is this understanding correct? There seems to be a limit of 10% of heap on the lookup size. Is this per lookup or total size of all the lookups in the "cachedNamespace"? We were also looking at the off heap option. In this case, would all the lookups in the "cachedNamespace" be off heap? Would

druid.lookup.namespace.numBufferedEntries

apply per lookup in the "cachedNamespace" - meaning would it create a buffer with 100K entries for each lookup? Single Cached Lookup There isn't much info on how to use this. Would the Lookup APIs work for this type of lookup as well? Finally, the max heap size for the historical process is specified as 24GB in the documentation. Is this a hard limit? As we plan to have some large lookups, can we set the max heap size > 24GB? Thanks, AR.

There are 2 flavours of Lookups in the documentation - Globally Cached Lookups and Single Cached Lookup. Which one should be preferred in this scenario?

druid-lookups-cached-global

is more commonly used, supports more sources, and should be preferred all else being equal. it always caches the full lookup in memory

druid-lookups-cached-single

is more niche, it doesn't support as many source and options, but it has one main feature that

druid-lookups-cached-global

doesn't: it has a mode where it can partially cache. this can be useful if you have a very large lookup and don't want to store the whole thing in memory at once. although, note that it will affect query performance, since it basically means druid would sometimes need to fetch lookup k/vs from a remote service during a query.

Thank you @Gian Merlino for taking the time to write the detailed reply. Regarding the heap size limitation, I am referring to the below documentation: https://druid.apache.org/docs/latest/operations/basic-cluster-tuning

A general rule-of-thumb for sizing the Historical heap is

(0.5GiB * number of CPU cores)

, with an upper limit of ~24GiB.

....

Having a heap that is too large can result in excessively long GC collection pauses, the ~24GiB upper limit is imposed to avoid this.

Is this only a recommendation and not a hard limit? Thanks, AR.

yes, that's just a recommendation, mainly to avoid long GC pauses (as the text mentions)

Thank you @Gian Merlino. We are also working on upgrading the cluster to JDK 17. Is it as simple as updating the JAVA_HOME env var on each Druid host and restarting the respective Druid services? We start up the Druid services using the bundled"`bin/run-druid` script. Can we do a rolling restart or should we shut down the entire cluster and bring it up again? Also, if we hit any issues with one or more services, can we roll back to JDK 8? Thanks, AR.

yes, it should be as simple as that. you can definitely do a rolling restart, and you can roll back to JDK 8 if needed