https://pinot.apache.org/ logo
m

Matt

03/20/2021, 12:40 AM
Hello, Just wondering is it normal MMAP going very high ? Also do this means I need to have ~1.5TB free space to hold the MMAP?
m

Mayank

03/20/2021, 12:59 AM
The servers memory map the indexes. So this should reflect the size of segments you have on the server. Is that not the case?
m

Matt

03/20/2021, 2:06 AM
If MMAP matches the total size of segments on the server then do I require 2 x size of segments disk space?
m

Mayank

03/20/2021, 2:12 AM
Why 2x?
m

Matt

03/20/2021, 2:25 AM
I might have misunderstood. I was thinking the actual segments in the disk is separate to MMAP.
m

Mayank

03/20/2021, 2:28 AM
So the way MMAP works is that OS will page in/out stuff from disk into RAM as needed. You could have 64GB RAM, but you could still MMAP 2TB of segments on disk. All you need is the disk storage.
Does that make sense?
m

Matt

03/20/2021, 2:14 PM
Thanks , So the segments stored in the disk is effectively MMAP ?.
So if I have 2TB segments then I need 2TB disk storage? no additional space is required?
s

Subbu Subramaniam

03/20/2021, 4:52 PM
@Matt you need to balance that with your QPS and latency requirements. Yes, 2TB of storage is sufficient, but then on each query, memory will be paged in depending on the amount of data needed to handle the query and where they are located in the segment. Depending on the kind of disk you have, this will negatively affect your latency.
m

Mayank

03/20/2021, 4:55 PM
Yes, 2TB of disk storage, no additional storage for mmap, to answer your original question.
m

Matt

03/21/2021, 2:39 AM
Thanks @Mayank, it is clear now.
@Subbu Subramaniam Are there any Max disk Size recommendations for an instance? I know that elastic search tends to stick at 2TB max per server.
m

Mayank

03/21/2021, 3:12 AM
It essentially boils down to cost2serve that you want to achieve. Typically if you store TB range of data size on single node, then it is recommended to have SSD storage. This will give you sub-second latency (at tens to hundred read qps). If your application is more latency critical (say < 100ms p95, at hundreds/thousands of qps), you want to store less data per node (say for a 64GB main-memory, you can go up to 200-400 GB) per node.
If you can provide more detail on your use-case, we can help recommend better configurations.
m

Matt

03/22/2021, 5:08 AM
I am using SSD and looking for cost optimized solution with ~50 QPS. I am ok with sub second latency however would like to add maximum disk on a single instance to reduce cost. Just upgraded to 9 instances reaching 20TB with each node of 2.3 TB.
m

Mayank

03/22/2021, 5:39 AM
Yeah, in that case, you can go adding 2TB per node. Of course, it also depends on how expensive your queries are.
m

Matt

03/22/2021, 2:11 PM
Thanks, are there any recommendations to improve query performance for situations like these? At the moment I am having 7 cpu, 26G ram for 2.3 TB data
Do you think I increase performance by adding more CPU and Ram ?
m

Mayank

03/22/2021, 2:22 PM
Yes, you’d need more CPU and RAM. As a reference, we have had deployments of similar nature in terms of data size and qps, and we have 32 cores and 64GB ram. If you really want to optimize for cost though, we can discuss further to better understand your use case
m

Matt

03/22/2021, 2:25 PM
That means I have to upgrade instance. My use case is for normal inverted index search and text index search
With a custom dashboard to display results
Also I am trying a multi avaliability zone deployment. Where 9 servers are deployed to 3 DC
I assume there could be a possible lag due to nw between DC
m

Mayank

03/22/2021, 2:33 PM
Is the current config working for you?
m

Matt

03/22/2021, 2:35 PM
Yes it is working, but I am noticing some ms latency
I assume it is because of nw lag, however need to trace metrics
Will it make sense to have a controller and broker per DC ?
Wondering that could make a difference
m

Mayank

03/22/2021, 2:51 PM
You can also start with this config and can upgrade if you need better latency/throughout later (without downtime).
👍 1