I could use some help thinking about how to tackle this problem. We have a table that's starting to become too big for a global lookup. We attempted to use a loading lookup, however, the initial queries are too slow. In a query I'm using to test, a fetch for a single record takes ~40ms. The problem is that, there are about 1000 distinct values, which equates to about ~30s of total runtime, because the loading lookup looks up each value independently. When I query all 1000 values together, the total time for the query is ~80ms. At first, I noticed applyAll could be overridden on the LookupExtractor and thought that may be a way to have it batch query lookups. However, I noticed that applyAll is never called. Abstractly, I'd like druid to tell the lookup to prefetch all the values it needs before joining or have the joinCursor iterate in batches and be able to make calls to the database. What are ya'll's thoughts on the best way to approach this problem?

Not only is

applyAll

never called, it also seems like it's never overridden. It might have been an idea that never quite saw the light of day Probably the easiest code path to make use it would be the

LookupExprMacro

vectorized path. Being vectorized, it's already set up to deal in batches. Currently it calls

apply

but that could change to

applyAll

. The join code path is more difficult because it isn't already set up to deal in batches. But it could possibly be done with some kind of lookahead. Btw, IMO, the signature of

applyAll

isn't ideal. In returning a

Map

it sort of forces a less efficient double-Map-lookup path for lookups backed by fully in-memory Maps. It'd probably be better for it to take a list of keys and return a list of values in the same order

I've been mainly looking at the join implementation because I assumed that'd be the easiest to optimize. The lookahead approach was the way I was leaning. I will checkout LookupExprMacro. Didn't realize that was vectorized. Thanks for the feedback.