https://pinot.apache.org/ logo
a

Apoorva Moghey

08/05/2020, 11:17 AM
I am getting this error while running
RealtimeProvisioningHelperCommand
n

Neha Pawar

08/05/2020, 5:11 PM
something is not right about the sample segment provided or the numHours provided. Does the sample segment have any rows? Could you share the command you ran and all the logs that followed?
a

Apoorva Moghey

08/05/2020, 5:17 PM
Copy code
bin/pinot-admin.sh RealtimeProvisioningHelper -tableConfigFile /Users/n0233/Documents/work/pinot/wallet_success_rate_table_config.json -numPartitions 3 -pushFrequency null -numHosts 3 -numHours 6,12,18,24 -sampleCompletedSegmentDir /Users/n0233/Downloads/pinot/data/d/wallet_success_rate_REALTIME/wallet_success_rate__0__3__20200805T1112Z -ingestionRate 5000 -maxUsableHostMemory 50G -retentionHours 74
Copy code
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/Users/n0233/Downloads/pinot/incubator-pinot/pinot-distribution/target/apache-pinot-incubating-0.5.0-SNAPSHOT-bin/apache-pinot-incubating-0.5.0-SNAPSHOT-bin/lib/pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/n0233/Downloads/pinot/incubator-pinot/pinot-distribution/target/apache-pinot-incubating-0.5.0-SNAPSHOT-bin/apache-pinot-incubating-0.5.0-SNAPSHOT-bin/plugins/pinot-file-system/pinot-s3/pinot-s3-0.5.0-SNAPSHOT-shaded.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See <http://www.slf4j.org/codes.html#multiple_bindings> for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.pinot.spi.plugin.PluginClassLoader (file:/Users/n0233/Downloads/pinot/incubator-pinot/pinot-distribution/target/apache-pinot-incubating-0.5.0-SNAPSHOT-bin/apache-pinot-incubating-0.5.0-SNAPSHOT-bin/lib/pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar) to method java.net.URLClassLoader.addURL(java.net.URL)
WARNING: Please consider reporting this to the maintainers of org.apache.pinot.spi.plugin.PluginClassLoader
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Executing command: RealtimeProvisioningHelperCommand -tableConfigFile /Users/n0233/Documents/work/pinot/wallet_success_rate_table_config.json -numPartitions 3 -pushFrequency null -numHosts 3 -numHours 6,12,18,24 -sampleCompletedSegmentDir /Users/n0233/Downloads/pinot/data/d/wallet_success_rate_REALTIME/wallet_success_rate__0__3__20200805T1112Z -ingestionRate 5000 -maxUsableHostMemory 50G -retentionHours 74
Allocating 4096 bytes for: wallet_success_rate__0__3__20200805T1112Z:amount.dict
Allocating 2028 bytes for: wallet_success_rate__0__3__20200805T1112Z:amount.sv.unsorted.fwd
Allocating 2048 bytes for: wallet_success_rate__0__3__20200805T1112Z:hours.dict
Allocating 2028 bytes for: wallet_success_rate__0__3__20200805T1112Z:hours.sv.unsorted.fwd
Allocating 2048 bytes for: wallet_success_rate__0__3__20200805T1112Z:months.dict
Allocating 2028 bytes for: wallet_success_rate__0__3__20200805T1112Z:months.sv.unsorted.fwd
Allocating byte array store buffer of size 18252 for: wallet_success_rate__0__3__20200805T1112Z:orderId.dict
Allocating 2028 bytes for: wallet_success_rate__0__3__20200805T1112Z:orderId.sv.unsorted.fwd
Allocating byte array store buffer of size 18252 for: wallet_success_rate__0__3__20200805T1112Z:paymentOptionType.dict
Allocating 2028 bytes for: wallet_success_rate__0__3__20200805T1112Z:paymentOptionType.sv.unsorted.fwd
Allocating 2048 bytes for: wallet_success_rate__0__3__20200805T1112Z:minutes.dict
Allocating 2028 bytes for: wallet_success_rate__0__3__20200805T1112Z:minutes.sv.unsorted.fwd
Allocating byte array store buffer of size 18252 for: wallet_success_rate__0__3__20200805T1112Z:login.dict
Allocating 2028 bytes for: wallet_success_rate__0__3__20200805T1112Z:login.sv.unsorted.fwd
Allocating 2048 bytes for: wallet_success_rate__0__3__20200805T1112Z:years.dict
Allocating 2028 bytes for: wallet_success_rate__0__3__20200805T1112Z:years.sv.unsorted.fwd
Allocating 4096 bytes for: wallet_success_rate__0__3__20200805T1112Z:createdAt.dict
Allocating 2028 bytes for: wallet_success_rate__0__3__20200805T1112Z:createdAt.sv.unsorted.fwd
Allocating byte array store buffer of size 18252 for: wallet_success_rate__0__3__20200805T1112Z:provider.dict
Allocating 2028 bytes for: wallet_success_rate__0__3__20200805T1112Z:provider.sv.unsorted.fwd
Allocating byte array store buffer of size 18252 for: wallet_success_rate__0__3__20200805T1112Z:adtf.dict
Allocating 2028 bytes for: wallet_success_rate__0__3__20200805T1112Z:adtf.sv.unsorted.fwd
Allocating 2048 bytes for: wallet_success_rate__0__3__20200805T1112Z:days.dict
Allocating 2028 bytes for: wallet_success_rate__0__3__20200805T1112Z:days.sv.unsorted.fwd
Allocating byte array store buffer of size 18252 for: wallet_success_rate__0__3__20200805T1112Z:status.dict
Allocating 2028 bytes for: wallet_success_rate__0__3__20200805T1112Z:status.sv.unsorted.fwd
Metrics aggregation is disabled.
Successfully loaded segment wallet_success_rate__0__3__20200805T1112Z with readMode: mmap
Allocating 6564 bytes for: wallet_success_rate__0__3__20200805T1112Z:orderId.dict
Allocating 6564 bytes for: wallet_success_rate__0__3__20200805T1112Z:createdAt.dict
Allocating 6564 bytes for: wallet_success_rate__0__3__20200805T1112Z:amount.dict
Allocating 6564 bytes for: wallet_success_rate__0__3__20200805T1112Z:login.dict
Allocating byte array store buffer of size 36504 for: wallet_success_rate__0__3__20200805T1112Z:login.dict
Trying to destroy segment : wallet_success_rate__0__3__20200805T1112Z
Trying to close RealtimeSegmentImpl : wallet_success_rate__0__3__20200805T1112Z
Segment used 217068 bytes of memory for 507 rows consumed in 0 seconds
Allocating 4096 bytes for: wallet_success_rate__0__3__20200805T1112Z:amount.dict
Allocating -4 bytes for: wallet_success_rate__0__3__20200805T1112Z:amount.sv.unsorted.fwd
Exception caught: 
java.lang.IllegalArgumentException: Illegal memory allocation -4 for segment wallet_success_rate__0__3__20200805T1112Z column wallet_success_rate__0__3__20200805T1112Z:amount.sv.unsorted.fwd
	at shaded.com.google.common.base.Preconditions.checkArgument(Preconditions.java:122) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-43b01ef14d6d6b6933a81077b36ab755cb4b3e95]
	at org.apache.pinot.core.io.readerwriter.RealtimeIndexOffHeapMemoryManager.allocate(RealtimeIndexOffHeapMemoryManager.java:77) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-43b01ef14d6d6b6933a81077b36ab755cb4b3e95]
	at org.apache.pinot.core.realtime.impl.forward.FixedByteSVMutableForwardIndex.addBuffer(FixedByteSVMutableForwardIndex.java:208) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-43b01ef14d6d6b6933a81077b36ab755cb4b3e95]
	at org.apache.pinot.core.realtime.impl.forward.FixedByteSVMutableForwardIndex.<init>(FixedByteSVMutableForwardIndex.java:77) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-43b01ef14d6d6b6933a81077b36ab755cb4b3e95]
	at org.apache.pinot.core.indexsegment.mutable.MutableSegmentImpl.<init>(MutableSegmentImpl.java:263) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-43b01ef14d6d6b6933a81077b36ab755cb4b3e95]
	at org.apache.pinot.tools.realtime.provisioning.MemoryEstimator.getMemoryForConsumingSegmentPerPartition(MemoryEstimator.java:280) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-43b01ef14d6d6b6933a81077b36ab755cb4b3e95]
	at org.apache.pinot.tools.realtime.provisioning.MemoryEstimator.estimateMemoryUsed(MemoryEstimator.java:223) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-43b01ef14d6d6b6933a81077b36ab755cb4b3e95]
	at org.apache.pinot.tools.admin.command.RealtimeProvisioningHelperCommand.execute(RealtimeProvisioningHelperCommand.java:227) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-43b01ef14d6d6b6933a81077b36ab755cb4b3e95]
	at org.apache.pinot.tools.admin.PinotAdministrator.execute(PinotAdministrator.java:156) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-43b01ef14d6d6b6933a81077b36ab755cb4b3e95]
	at org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:168) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-43b01ef14d6d6b6933a81077b36ab755cb4b3e95]
n

Neha Pawar

08/05/2020, 6:12 PM
sample segment has 507 rows?
could you try with a segment that has a lot more rows?
this math is breaking:
Copy code
_sampleSegmentConsumedSeconds = (int)(_totalDocsInSampleSegment/ingestionRate);
507/5000. The whole calculation is happening with sampleSegmentConsumedSeconds=0
a

Apoorva Moghey

08/06/2020, 5:21 AM
got it. thanks @Neha Pawar