Hi All! I was reading the Druid documentation for segment loading and mmap. The docs say that : "The segment cache uses memory mapping*. The cache consumes memory from the underlying operating system so Historicals can hold parts of segment files in memory to increase query performance at the data level*". Followed by "At query time*, if the required part of a segment file is available in the memory mapped cache or "page cache", the Historical re-uses it and reads it directly from memory. If it is not in the memory-mapped cache, the Historical reads that part of the segment from disk. In this case, there is potential for new data to flush other segment data from memory.*" My understanding of this is that each segment is mmap-ed and that a % of each segment is available in RAM (Address space of historical). If the part of the segment required for query is in this page cache, it will get used. If not; the newly accessed data of the same segment will replace the old page cache of the same the segment. Is this a correct understanding? Does this mean that % of each segment cached using mmap is approximately equal for each segment? Does a frequently accessed segment not get an increased mmap-ed percentage? And by increasing the RAM the % of segment cacheable simply increases and hence request speeds are faster?

I would expect recently accessed segments to remain in filesystem page cache and not-recently-accessed segments to have no presence in the filesystem page cache

k8s says it's using 8gb, /proc $ cat /proc/vmstat|grep nr_mapped nr_mapped 1296750 /proc $ cat /proc/vmstat|grep nr_file_pages nr_file_pages 1878566 Here it's says it using ~ 1 296 750 * 4 kilobytes = 5.1 gb of fmap

Right. But is that usage completely for segments that are recently accessed or is it ~ some part of all segments that keeps getting hot swapped?

oh, now I get what you are saying, technical perspective, you're absolutely right: mmap operates at the page level (typically 4kb pages) The OS can cache/uncache individual pages of a memory-mapped file So theoretically, yes, it's dealing with parts of segments.. but druid does full segment scans during queries, so in the end, even though the OS could cache partial segments, in practice it often ends up caching most of the recently accessed segments

Understood. The reason I ask is because when we performed hot and cold tier styled tiering we saw improvements in hot data access. But this was for a staging type of data. If the mmap cache is more filled with hot tiered data anyway then setting up hot and cold tiering would possibly not have that much affect on cold tier data

one benefit of tiers is avoiding thrashing the filesystem cache A scenario: • cache is filled with recent data, many queries occurring • query arrives requesting old data - cache is populated with old data to satisfy query • cache is again populated with recent data This clearing / reloading from disk of filesystem cache can be slow - separating old data from new can help to avoid this and keep frequently queried data in the cache more often

I think tiers would provide better isolation when running on different servers, as they will avoid thrashing the filesystem cache like Kyle explained. When running both tiers on the same server, if you are seeing better query times*, maybe it's possible to be due to process-level cache isolation, but probably it's performance gains from uncompressed versions that Druid keeps in heap, what exactly or how druid will place this it's not documented, as you noted initially

The way we experimented : our control was standard historical server set up and variant was same server with only hot data and cold data on a different server. Queries fired were of 3 types hot query, cold query, mixed query. We saw an improvement in hot queries but the question asked was ~ if the segment page cache is anyway occupied by only hot data (in case of control as is our hypothesis), why would we see a lot of improvement. I think this can be resolved by testing in such a way that we do all hot queries first and then mixed and then cold and then do the same but having completely shuffled the queries. We should ideally see close query times for hot tier data and hot queries used only.