Hi Team, What is the impact of having Swap memory ...
# general
a
Hi Team, What is the impact of having Swap memory enabled on the historical servers? In our org, completely disabling Swap is not allowed so we have set the "swappiness=1" which is the smallest possible value. But we see that the Swap memory is 100% utilized even though there is free memory on the host and this is raising alerts at h/w level (as 100% Swap utilization usually indicates that the host may be running out of memory). Does Druid use the Swap memory to read the memory mapped files in addition to the free main memory? Or could this be happening because the OS is swapping in all the files that Druid is reading during query execution? Is the swap actually getting thrashed? Trying to figure if we need to take any action or if this can be safely ignored. Thanks, AR.
We got the hosts checked by SAs and they found that 99% of Swap is being used by the historical process. Doesn't it use the free main memory to read the memory mapped files? Also, we have noticed a significant degradation in query performance since last week. There is in >50GB free on the host but Swap is 100% utilized as mentioned above. We marked all segments of some unused tables as "unused" and the query performance improved again. Is this in some way connected to the swap being full? We are scratching our heads right now.. would be grateful for any pointers. Thanks, AR.
g
Druid doesn't explicitly use swap. In general there's three major sources of memory use with Druid • JVM heap • JVM off-heap memory (so-called "direct buffers") • Page cache, outside the JVM, not explicitly managed by Druid Generally we recommend disabling swap because swapping JVM memory is never helpful in a correctly-configured Druid system. "Correct" configuration is when
Xmx
and
MaxDirectMemorySize
together do not exceed memory on the machine, even when they're at max size. If either of those two swap then performance is going to suffer greatly, so we recommend disabling swap rather than allowing swap to happen. I don't have any experience with running Druid with swap enabled, so I am not sure if what you're seeing is normal or not. The main suggestion I have is that you could double check that you have the "correct" configuration with regard to
Xmx
and
MaxDirectMemorySize
. That should make it so the OS never "needs" to swap out JVM memory
a
Thank you Gian. Total Memory is much higher than the Xmx + MaxDirectMemorySize. Ingestion also runs on these hosts. But we can see that free memory is ~50GB most of the time. So memory is not an issue. We checked the swap activity and found that there is no Swap thrashing. Looks like the swap is getting filled and sitting at 100% continuously. Not sure if the 100% swap utilization is affecting the Druid query performance though. Is there a way to check if it is using main memory to load the memory-mapped files? Regards, AR.
g
I'm not really sure, TBH I don't have much experience with swap on Linux since I always turn it off 🙂
a
Thanks Gian. "disable swap" means that there will be no "swap" configuration in "/etc/fstab".
free
will show "Swap" as "0". Is this understanding correct? Thanks, AR.
j
Hi, monitor the path IO - memory mapped files in windows can behave the same way - they malloc effectively but there is no IOPS against the paging files. I think the idea/engineering behind it is if you memory map a file, what is the point in allocating RAM, so allocate virtual memory instead, but don't run IOPS against it? There is also the possibility that the available system memory could not be allocated so it all goes to virtual memory. This is more of an OS behaviour thing, I'd expect to see swap allocated but few IOPS against it and as you perform RW's against the file you may see your RAM start to be used, but is this FS cache or is this physical memory being used? Who knows...
If you are memory mapping anything - I'd say you will have to have some kind of addressing against it available so virtual memory is perfect for that job???