Hey guys, I'd like to ask a question which is not ...
# generald
Diogo Baeder
05/09/2022, 12:45 PMHey guys, I'd like to ask a question which is not really a problem, but rather just a curiosity on how an aspect of the system works: every time I spin up Pinot with my docker-compose, create the tables, add data and query it for the first time, it does't query as fast as I'd like, but then right on the second and subsequent queries it gets blazing fast, even if I change many constraints in my query. I know that Pinot doesn't do "caching", but why is there such a big difference in query times? For example, it may drop from 900ms on the first query to 40ms, 30ms or even lower on the second, third, fourth etc queries.
m
Mayank
05/09/2022, 1:41 PM
Pinot memory maps segments, so there’s an initial warmup.
d
Diogo Baeder
05/09/2022, 2:27 PMAh, got it. And that happens on the Server, I presume? And for how long does it keep them mapped?
m
Mitchell H
05/09/2022, 2:36 PMas long as possible. One of the fun parts of mmap is it doesn't cost much unless the segment is pulled into actual ram.
Mitchell H
05/09/2022, 2:37 PMthe JVM & linux kernel, and pinot its self, do a good job of bringing/keeping segments into memory via mmap and removing them when necessary.
d
Diogo Baeder
05/09/2022, 2:37 PMGot it. Nice! But what determines that the file should be kept there? Like, when should I expect it not to be there mapped anymore?
m
Mitchell H
05/09/2022, 2:38 PMhttps://www.linuxatemyram.com/ is the best explanation I've seen on this. The TLDR is "when necessary"
Mitchell H
05/09/2022, 2:39 PMnecessary meaning something else could use that ram more often than what's currently using it
d
Diogo Baeder
05/09/2022, 2:54 PMAh, so it's more determined by the OS than Pinot itself, got it. Thanks man! 🙂