Apache Pinot

are there any limits of # of segments per server? or maybe on number of columns * indices * segments?

we hit some interesting issues with hitting `vm.max_map_count` (some detail in thread), and were just curious if there's a suggested limit

```Caused by: java.io.IOException: Map failed: MMapIndexInput(path="/var/pinot/server/data/index/d23ead09-78cc-488b-b4a5-b135439d1fb7_OFFLINE/d23ead09-78cc-488b-b4a5-b135439d1fb7_OFFLINE_1579755014000_1579873493000_772_ed37611f-b076-4901-848d-ce217aa7e863/v3/hyperarc__transactionFrom__address_category_3.lucene.index/_0.si") [this may be caused by lack of enough unfragmented virtual address space or too restrictive virtual memory limits enforced by the operating system, preventing us to map a chunk of 395 bytes. Please review 'ulimit -v', 'ulimit -m' (both should return 'unlimited'), and 'sysctl vm.max_map_count'. More information: <http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html>]```
was cleared up after we deleted a bunch of segments, and we'll follow the suggestion there to actually increase the limit, but disk usage/memory usage overall was fine, so wondering what theoretical limits are

We haven’t seen that limit.. how many segments do you have on a single node?

we have 25K segments across 4 nodes (sometimes a little more, like 30k), so ~6K segments per node

~30B docs total; ~20TB or so uncompressed

each table has somewhere between 10-50 columns, and between 1 and 3X that number of indices

Did you try what the error suggested?
```Please review 'ulimit -v', 'ulimit -m' (both should return 'unlimited'), and 'sysctl vm.max_map_count'. More information: <http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html>]```

yeah, and that should help, but we're worried we'll just run into it again

@.Kishore G suggested `native` text indices so we'll try those too!

Also check: <https://apache-pinot.slack.com/archives/C011C9JHN7R/p1687359580019639?thread_ts=1686675833.291939&amp;cid=C011C9JHN7R>

yeah, that's why we had them set to `vm.max_map_count = 524288`

What’s the total memory on each server?

And is this at restart, or after a while (fragmentation)?

`131 GiB`  (note: we weren't actually _using_ &gt;50%) of the memory at this time) we saw this after a restart

i ran a `pmap` on one of our servers as suggested by kishore; of the 400k lines in there, 200k were from lucene (conservatively)

so it's not totally surprising that if the limit is 500kish, we'd run out when adding some more tables

we can ofc raise the limit, unclear if there's any downside to making it some crazy number

probably not? guess we'd hit some other memory issue first

from experience it's worth setting up the segment roll up task if you have lots of small segment files

word; they're not too small, they range from 500MB-1GB in [uncompressed] size