https://pinot.apache.org/ logo
Join Slack
Powered by
Hi team, I have a question regarding how to correc...
# troubleshooting
p
Hi team, I have a question regarding how to correctly configure 
maxUsableHostMemory
in 
RealtimeProvisioningHelper
. We have 6 tables being consumed by around 10 realtime servers. Each server has an upper limit of 40G of memory. Segments are also relocated to a different set of servers (not realtime) once they’re committed. The docs say “This is the total memory available in each host for hosting 
retentionHours
worth of data (i.e. “hot” data) of this table.“. How would the value be calculated in this case? Should be simply do 40/6 as a rough estimate?
Haven’t dug into why this is happening, but I run into
Copy code
Exception in thread "main" java.lang.OutOfMemoryError: Direct buffer memory
	at java.base/java.nio.Bits.reserveMemory(Bits.java:175)
	at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118)
	at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:317)
	at org.apache.pinot.segment.spi.memory.PinotByteBuffer.allocateDirect(PinotByteBuffer.java:38)
	at org.apache.pinot.segment.spi.memory.PinotDataBuffer.allocateDirect(PinotDataBuffer.java:115)
	at org.apache.pinot.segment.local.io.writer.impl.DirectMemoryManager.allocateInternal(DirectMemoryManager.java:53)
	at org.apache.pinot.segment.local.io.readerwriter.RealtimeIndexOffHeapMemoryManager.allocate(RealtimeIndexOffHeapMemoryManager.java:80)
	at org.apache.pinot.segment.local.realtime.impl.forward.FixedByteSVMutableForwardIndex.addBuffer(FixedByteSVMutableForwardIndex.java:208)
	at org.apache.pinot.segment.local.realtime.impl.forward.FixedByteSVMutableForwardIndex.<init>(FixedByteSVMutableForwardIndex.java:77)
	at org.apache.pinot.segment.local.indexsegment.mutable.MutableSegmentImpl.<init>(MutableSegmentImpl.java:308)
	at org.apache.pinot.controller.recommender.realtime.provisioning.MemoryEstimator.getMemoryForConsumingSegmentPerPartition(MemoryEstimator.java:339)
	at org.apache.pinot.controller.recommender.realtime.provisioning.MemoryEstimator.estimateMemoryUsed(MemoryEstimator.java:272)
	at org.apache.pinot.tools.admin.command.RealtimeProvisioningHelperCommand.execute(RealtimeProvisioningHelperCommand.java:286)
When 
ingestionRate
is high. For example:
Copy code
RealtimeProvisioningHelper -tableConfigFile /Users/prashant.pandey/Trash/tableConfig.json -sampleCompletedSegmentDir /Users/prashant.pandey/Downloads/segment -numHosts 10,12 -numHours 6,8 -maxUsableHostMemory 7G -numPartitions 16 -retentionHours 24 -ingestionRate 8500
m
For 1, you can use 40/6 as a rough estimate (cc: @Subbu Subramaniam for RT multi-tenancy, and OOM).
s
The prov helper tool works for single tenant use case. If you have multiple tables, one way to do this would be to use SSDs and use memory mapped consuming segments. In which case, you are limited by the numeber of paging in/out that can be done -- depending on your ingestion and query rates
@Sajjad Moradi can add some more here
s
Regarding the OOM that you're getting: The tool uses the ingestion rate parameter and the numDocs (num rows) in the provided sample and estimates how many rows there will be when the segment completes in 6 or 8 hours (numHours parameter). I think because of the high ingestion rate, the bytebuffer is huge and you get OOM. You can provide lower numHours. Also ingestion rate parameter is per partition. If you can increase the number of partitions, then you'll get lower ingestion rate per partition which leads to smaller bytebuffer size
6 Views
Open in Slack
PreviousNext
Powered by