Tanmay Movva11/16/2020, 8:08 AM
I am running this by building pinot from source and the results are quite surprising.
Size of segment directory = 239.4mb Number of documents = 3529197 retention = 30 days ingestion rate = 1000 numPartitions = 1 Table Replicas = 1
active memory per host = Number of active(consuming + completed/retained) segments * Size of the segment directory.
Subbu Subramaniam11/16/2020, 4:53 PM
file system. Completed segments do not suffer writes, so you can choose to tune it mapping as much memory as you would like that does not affect your latency.
Tanmay Movva11/17/2020, 2:51 AM
If this is not the case, you can choose to reduce the ratio of mapped vs active memory by increasing the total amount of memory availableBy this do you mean
argument? I’ve already set this to 90g, because anything less than the 82g(because the required server host memory as per my output is 81g) doesn’t give me any output(kinda obvious from the code). If not this, where can I set the ratio or read about it?
The more paging you have, the latency is likely to be higher. Consuming segments are read-write memory, so mapping them on to disk can cause paging when rows are ingested since writes happenI get this part. Have no issue with consuming segments or realtime servers. Wanted to get an idea on how should I provision my offline server(which would host completed segments) so that segments are on disk with mmap and are paged in when required.