Tanmay Movva
11/16/2020, 8:08 AMSize of segment directory = 239.4mb
Number of documents = 3529197
retention = 30 days
ingestion rate = 1000
numPartitions = 1
Table Replicas = 1
I am running this by building pinot from source and the results are quite surprising.active memory per host = Number of active(consuming + completed/retained) segments * Size of the segment directory.
Kishore G
Subbu Subramaniam
11/16/2020, 4:53 PMtmpfs
file system. Completed segments do not suffer writes, so you can choose to tune it mapping as much memory as you would like that does not affect your latency.Tanmay Movva
11/17/2020, 2:51 AMIf this is not the case, you can choose to reduce the ratio of mapped vs active memory by increasing the total amount of memory availableBy this do you mean
maxUsableHostMemory
argument? I’ve already set this to 90g, because anything less than the 82g(because the required server host memory as per my output is 81g) doesn’t give me any output(kinda obvious from the code).
If not this, where can I set the ratio or read about it?
The more paging you have, the latency is likely to be higher. Consuming segments are read-write memory, so mapping them on to disk can cause paging when rows are ingested since writes happenI get this part. Have no issue with consuming segments or realtime servers. Wanted to get an idea on how should I provision my offline server(which would host completed segments) so that segments are on disk with mmap and are paged in when required.