This message was deleted.
# generals
Slackbot
11/28/2023, 9:15 AMThis message was deleted.
k
Kyle Hoondert
11/28/2023, 10:57 AMEnabling cache on historicals with
druid.historical.cache.useCachedruid.historical.cache.populateCachea
Anshu Makkar
11/28/2023, 11:22 AMThanks @Kyle Hoondert While I understand the
caffeineAnshu Makkar
11/28/2023, 11:25 AMMy understanding from mmap is that some of the segments in disk will be mapped and there will be less paging. If that's the case how do I control this parameter like I have TB's of data in each historical (segment cache location on disk) I have around 280 GB FREE system memory on each historical.
Copy code
Direct Memory =28G
Merge buffer =500MB
numThreads=45k
Kyle Hoondert
11/28/2023, 12:46 PMYou don’t configure it - it is managed by the OS. Any free memory will be used to cache segments which have been recently accessed https://druid.apache.org/docs/latest/design/historical/#loading-and-serving-segments-from-cache
a
Anshu Makkar
11/28/2023, 12:49 PMThis is little strange considering the fact pod memory never goes beyond 48 Gb
Each query I run queries 7 day of data, each hour has 35 segments of 500mb each.
And 28G is Maxdirectmemory
Anshu Makkar
11/28/2023, 12:50 PMIs there a possible metric which could indicate this mmap of segments
k
Kyle Hoondert
11/28/2023, 12:53 PMMaybe something like
Copy code
$ cat /proc/meminfo | grep Mapped
Mapped: 178172 kB🙌 1
a
Anshu Makkar
11/28/2023, 1:03 PMThx
Anshu Makkar
12/01/2023, 1:56 PMI checked it again, most of the RAM is consumed by
buff/cacheAnshu Makkar
12/01/2023, 1:57 PMWhereas
map_count