Hello, Just wondering is it normal MMAP going very high ? Also do this means I need to have ~1.5TB free space to hold the MMAP?

The servers memory map the indexes. So this should reflect the size of segments you have on the server. Is that not the case?

If MMAP matches the total size of segments on the server then do I require 2 x size of segments disk space?

Why 2x?

I might have misunderstood. I was thinking the actual segments in the disk is separate to MMAP.

So the way MMAP works is that OS will page in/out stuff from disk into RAM as needed. You could have 64GB RAM, but you could still MMAP 2TB of segments on disk. All you need is the disk storage.

Thanks , So the segments stored in the disk is effectively MMAP ?.

@Matt you need to balance that with your QPS and latency requirements. Yes, 2TB of storage is sufficient, but then on each query, memory will be paged in depending on the amount of data needed to handle the query and where they are located in the segment. Depending on the kind of disk you have, this will negatively affect your latency.

Yes, 2TB of disk storage, no additional storage for mmap, to answer your original question.

Thanks @Mayank, it is clear now.

It essentially boils down to cost2serve that you want to achieve. Typically if you store TB range of data size on single node, then it is recommended to have SSD storage. This will give you sub-second latency (at tens to hundred read qps). If your application is more latency critical (say < 100ms p95, at hundreds/thousands of qps), you want to store less data per node (say for a 64GB main-memory, you can go up to 200-400 GB) per node.

I am using SSD and looking for cost optimized solution with ~50 QPS. I am ok with sub second latency however would like to add maximum disk on a single instance to reduce cost. Just upgraded to 9 instances reaching 20TB with each node of 2.3 TB.

Yeah, in that case, you can go adding 2TB per node. Of course, it also depends on how expensive your queries are.

Thanks, are there any recommendations to improve query performance for situations like these? At the moment I am having 7 cpu, 26G ram for 2.3 TB data

Yes, you’d need more CPU and RAM. As a reference, we have had deployments of similar nature in terms of data size and qps, and we have 32 cores and 64GB ram. If you really want to optimize for cost though, we can discuss further to better understand your use case

That means I have to upgrade instance. My use case is for normal inverted index search and text index search

Is the current config working for you?

Yes it is working, but I am noticing some ms latency

You can also start with this config and can upgrade if you need better latency/throughout later (without downtime).