This message was deleted.
# troubleshootings
Slackbot
10/23/2022, 5:08 PMThis message was deleted.
m
Mark Herrera
10/24/2022, 2:14 PMt
Tijo Thomas
10/24/2022, 7:19 PMI have seen its usage in production . Any reason you asked ?
d
Didip Kerabat
10/24/2022, 7:21 PM@Mark Herrera yes.
I am wondering if consolidating all caches to a dedicated store will lead to more efficient cache hit %.
I am thinking of using https://github.com/Snapchat/KeyDB for the “Redis” backend.
m
Mark Herrera
10/24/2022, 8:19 PMI haven't seen an efficiency consideration, but maybe @Tijo Thomas has?
I've seen Redis brought up when people want to use different, off-machine caches. For example, use one cache for one type of query, and then use another cache for a different type of query. So, your question of consolidating all caches to a dedicated store might be a novel one.
Hopefully others will chime in as well.
d
Didip Kerabat
10/24/2022, 11:06 PMQuick question, the broker does not talk to each other when it comes to cache data, is that correct?
E.g. this scenario:
• query A lands on broker A, its result is cached on broker A.
• exact same query A lands on broker B, what happens here? I am guessing that broker B will simply cache the query result again, yes?
g
Gian Merlino
10/25/2022, 4:11 AMyes, that's exactly right, unless they have a shared cache (like memcached or redis)
Gian Merlino
10/25/2022, 4:11 AMin shared-cache case, Broker B will check the cache, and be able to use it
Gian Merlino
10/25/2022, 4:12 AMHowever, watch out for Broker-based caching in general… for the per-segment cache, it can be detrimental at scale. See here: https://druid.apache.org/docs/latest/querying/caching.html#where-to-enable-caching
Avoid using per-segment cache at the Broker for large production clusters. When the Broker cache is enabled (druid.broker.cache.populateCache is true) and populateCache is not false in the query context, individual Historicals will not merge individual segment-level results, and instead pass these back to the lead Broker. The Broker must then carry out a large merge from all segments on its own.
👍 1
Gian Merlino
10/25/2022, 4:12 AMFor this reason, we generally recommend having caching enabled on Historicals but not Brokers
Gian Merlino
10/25/2022, 4:13 AMOn using shared cache (memcached/redis): it does increase hit rate, but also increases latency, due to the hop needed to the remote cache for every query. It's worth evaluating in your prod environment whether this tradeoff is worth it. Depends heavily on your users' query patterns.
d
Didip Kerabat
10/25/2022, 4:32 PMah, good point about another extra network hops to get the cache data.
t
Tijo Thomas
10/31/2022, 6:57 PM@Didip Kerabat , We are working towards removing redis cache in one of the customer who heavily uses redis cache . One of the reason is the maintainability of the redis infra. We are doing some tests before changing it to Caffeine cache and i will let u know the results.
🙏 1