Apache Pinot

anyone got a performance comparison of storing index in HDD/ SSD / EFS / NFS ?  As usual I assume SSD will be the high performer?

Two observations that I have seen. If data fits in memory, HDD and SSD performance is similar in terms of query latency. When data starts spilling over to SSD can make a lot of difference. We were able to push it to full capacity for read throughput for a use case to get ms latency. 

As expected of course, but good to establish that in practice as well

We typically NFS as a deep store (which is not in the read path), and not as attached disk for serving nodes 

<@U01EEQ2A7EJ> one more to add to Mayank's observation. If you have HDD, be prepared to take a latency hit when new segments are loaded. Even if segments are replaced _in situ_ and both of them fit in memory, we have seen high latency during the replacement time.

Of course, it also depends on how much data you have, the kind of queries you run, etc.

Yep, HDDs have a cold start problem, especially when all data is being refreshed 

<@UDQU92KBK> interesting to know that HDD and SSD perform similar based on memory. So that means I have to increase the xmx settings.

<@U01EEQ2A7EJ> no, the segments are not loaded on heap, so Xmx setting does not impact. It is about whether OS is paging the segments in/out of main memory

ok if too much paging is involved  HDD wont be good idea. I am trying with real time streaming data which causes the index to be appended all the time.

<@UDSRQR2GP> Thanks I assume segments keep changing based on the query right? Seems like SSD is the way to go.

<@U01EEQ2A7EJ> yes SSD would be the way to go if go if for real-time and high read throughput (thousands of qps). 

<@U01EEQ2A7EJ> you can also try using `tmpfs` for memory used by consuming segments (since these are read-write pages). Once segments are completed, they use read-only memory. Always useful to read the realtime tuning material in the documentation.  <https://docs.pinot.apache.org/operators/operating-pinot/tuning/realtime>