There are only 3 partitions per node

What was the recommendation of the tool?

10MB. @srisudha could you confirm?

Note direct memory setting is at 10GB

Yes, 10 to 100 MB seems like an order of magnitude jump. If tool recommended 10MB then that is about what you have for memory. You have a choice of deploying more hosts, or use hosts with higher memory.

The question I am struggling with is why was it 10MB, given that there re three partitions that are configured to generate 100MB segments l, and total direct memory is 10GB

I am sure there is a reason, but would be good for the users to understand

No.. From the tool we got 100 mb only as output..

We gave 3 partitions and traffic protection as 10 million per 24 hours

And v got that output for three servers

I thought you had mentioned your initial segments were 10MB based on the tool.

But if tools says 100MB, then it is even more baffling @Subbu Subramaniam

Memory used per host numHosts --> 2 |3 | numHours 12 --------> 4.81G |2.88G | 24 --------> 8.08G |4.85G | Optimal segment size numHosts --> 2 |3 | numHours 12 --------> 49.36M |49.36M | 24 --------> 98.72M |98.72M | Consuming memory numHosts --> 2 |3 | numHours 12 --------> 3.6G |2.16G | 24 --------> 7.12G |4.27G |

this is the output for 10 million documents segment over 24 hrs and 3 partitions

They were 2 times v ran the tool for different values of traffic .. 50 mb was initial value.. But v tried with 20 mb segment size because v wanted multiple segments to b created for the run to check query perf..

Post few discussions with u , we re executed and went with higher seg size ..

So the tool actually suggested 100MB, right?

Yes

Correct

@srisudha I suppose you have set on offheap allocation to mmap?

if it is MMAP, the stack trace shouldn't probably contain calls to allocateDirect() ?

I think the loadMode is set to MMap, but offheap allocation is probably a different config?

correct, it is a different config

but using direct mem is not a probem

just increase maxDirect when stating up jvm

The maxDirect is 10g right now

in this case, the tool says how much mem we think we will need

for consuming plus completed sements

at a time there are only 3 segments each of which is 100 MB

So the confusing part is where's the gap?