Hi! Where can I find more info on "Single segment lookups" extension mentioned in Demystifying Druid Myths talk? Is it the same as Druid indexed tables extension? Is the broadcast rule an equivalent functionality?

I can talk about broadcast rule. If you set it to 'foreverBroadcast' you will make sure all your historicals will have a copy of all the segments from the datasource (if you set the rule in a single datasource) or every druid segment. Usually segments will have 2-3 replicas across your historicals, but with this you make sure if you have 100 historicals, all of them will have this segment. This might be particularly important if you have a table that is used all the time like a lookup table that is joined with everything else, say a 'users' table. If this table is complete in all historicals, no matter where the query runs, it can access the data from this table locally. If your dataset changes too fast and is not that big, Kafka lookups might be the best solution. If it's huge, broadcast forever might be the best you can get (and its very good). Looks like the Imply single segment lookup is the same as the indexed tables extension. I've used almost 2y. If I remember correctly it can keep this single segment table in memory, so joins with it are veeery fast. At least when I used it wasn't something Imply was working continuously back them. Maybe @Hellmar Becker or @Sergio Ferragut can talk about what were they saying in the video.

Hi Daniel, The doc page states that Broadcast rules load the segment data into the Broker, not the Historical: https://druid.apache.org/docs/latest/operations/rule-configuration#broadcast-rules Presumably this allows the Broker to do some of the query processing directly. However I am not sure this is a fully baked feature in the product. Kafka lookups work well for key-value pairs, but for multi-attribute dimension style tables I don't know if either the broadcast rules or indexed tables will ever be brought to GA status. There is an Imply/Druid only feature called Dimension Tables under development that is targeted for GA status in the coming moths, but afaik that will not be going into OS Druid. What use case do you have in mind?

Hi! Thanks for the answers.

Usually segments will have 2-3 replicas across your historicals, but with this you make sure if you have 100 historicals, all of them will have this segment.

Does it mean we can skip the part were we send results of query on the right side of join to Historical and it is instead executed locally + joined with local data of left query? Same as "Join using a lookup" in https://github.com/implydata/learn-druid/blob/main/notebooks/03-query/11-joins.ipynb?

Looks like the Imply single segment lookup is the same as the indexed tables extension. I've used almost 2y. If I remember correctly it can keep this single segment table in memory, so joins with it are veeery fast.

So the we skip loading data from local disc into memory (compared to broadcast rule)?

The doc page states that Broadcast rules load the segment data into the Broker, not the Historical:

I am also confused by this - if it loads data on Broker then is it also executing query (on the right side of join) on Broker instead of Historical?

What use case do you have in mind?

Dimensions tables basically. In this case - joining financial data with currency conversion rate table that has (date, from_currency, to_currency) as multi-value key.