hi all - trying to use the RealtimeProvisioningHel...
# troubleshooting
j
hi all - trying to use the RealtimeProvisioningHelper
bin/pinot-admin.sh RealtimeProvisioningHelper  -tableConfigFile /tmp/table.json -numPartitions 1  -numHosts 8,6,10 -numHours 6,12,18,24 -sampleCompletedSegmentDir /var/pinot/server/data/index/aws_flowlogs_REALTIME/aws_flowlogs__2__3__20220725T2132Z/ -ingestionRate 100000 -maxUsableHostMemory 20G -retentionHours 72
which seems to be doing it’s thing until:
Copy code
Allocating 32768 bytes for: aws_flowlogs__2__3__20220725T2132Z:srcport.dict
Allocating 32768 bytes for: aws_flowlogs__2__3__20220725T2132Z:dstport.dict
Trying to destroy segment : aws_flowlogs__2__3__20220725T2132Z
Trying to close RealtimeSegmentImpl : aws_flowlogs__2__3__20220725T2132Z
Segment used 45236440 bytes of memory for 337500 rows consumed in 4 seconds
Allocating byte array store buffer of size 71680 for: aws_flowlogs__2__3__20220725T2132Z:srcaddr.dict
Allocating -4 bytes for: aws_flowlogs__2__3__20220725T2132Z:srcaddr.sv.unsorted.fwd
java.lang.IllegalArgumentException: Illegal memory allocation -4 for segment aws_flowlogs__2__3__20220725T2132Z column aws_flowlogs__2__3__20220725T2132Z:srcaddr.sv.unsorted.fwd
	at shaded.com.google.common.base.Preconditions.checkArgument(Preconditions.java:122)
	at org.apache.pinot.segment.local.io.readerwriter.RealtimeIndexOffHeapMemoryManager.allocate(RealtimeIndexOffHeapMemoryManager.java:78)
	at org.apache.pinot.segment.local.realtime.impl.forward.FixedByteSVMutableForwardIndex.addBuffer(FixedByteSVMutableForwardIndex.java:208)
	at org.apache.pinot.segment.local.realtime.impl.forward.FixedByteSVMutableForwardIndex.<init>(FixedByteSVMutableForwardIndex.java:77)
	at org.apache.pinot.segment.local.indexsegment.mutable.MutableSegmentImpl.<init>(MutableSegmentImpl.java:310)
	at org.apache.pinot.controller.recommender.realtime.provisioning.MemoryEstimator.getMemoryForConsumingSegmentPerPartition(MemoryEstimator.java:339)
	at org.apache.pinot.controller.recommender.realtime.provisioning.MemoryEstimator.estimateMemoryUsed(MemoryEstimator.java:272)
....
I’m not clear on what the cause of this is?