Apoorva Moghey
08/05/2020, 11:17 AMRealtimeProvisioningHelperCommand
Neha Pawar
Apoorva Moghey
08/05/2020, 5:17 PMbin/pinot-admin.sh RealtimeProvisioningHelper -tableConfigFile /Users/n0233/Documents/work/pinot/wallet_success_rate_table_config.json -numPartitions 3 -pushFrequency null -numHosts 3 -numHours 6,12,18,24 -sampleCompletedSegmentDir /Users/n0233/Downloads/pinot/data/d/wallet_success_rate_REALTIME/wallet_success_rate__0__3__20200805T1112Z -ingestionRate 5000 -maxUsableHostMemory 50G -retentionHours 74
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/Users/n0233/Downloads/pinot/incubator-pinot/pinot-distribution/target/apache-pinot-incubating-0.5.0-SNAPSHOT-bin/apache-pinot-incubating-0.5.0-SNAPSHOT-bin/lib/pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/n0233/Downloads/pinot/incubator-pinot/pinot-distribution/target/apache-pinot-incubating-0.5.0-SNAPSHOT-bin/apache-pinot-incubating-0.5.0-SNAPSHOT-bin/plugins/pinot-file-system/pinot-s3/pinot-s3-0.5.0-SNAPSHOT-shaded.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See <http://www.slf4j.org/codes.html#multiple_bindings> for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.pinot.spi.plugin.PluginClassLoader (file:/Users/n0233/Downloads/pinot/incubator-pinot/pinot-distribution/target/apache-pinot-incubating-0.5.0-SNAPSHOT-bin/apache-pinot-incubating-0.5.0-SNAPSHOT-bin/lib/pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar) to method java.net.URLClassLoader.addURL(java.net.URL)
WARNING: Please consider reporting this to the maintainers of org.apache.pinot.spi.plugin.PluginClassLoader
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Executing command: RealtimeProvisioningHelperCommand -tableConfigFile /Users/n0233/Documents/work/pinot/wallet_success_rate_table_config.json -numPartitions 3 -pushFrequency null -numHosts 3 -numHours 6,12,18,24 -sampleCompletedSegmentDir /Users/n0233/Downloads/pinot/data/d/wallet_success_rate_REALTIME/wallet_success_rate__0__3__20200805T1112Z -ingestionRate 5000 -maxUsableHostMemory 50G -retentionHours 74
Allocating 4096 bytes for: wallet_success_rate__0__3__20200805T1112Z:amount.dict
Allocating 2028 bytes for: wallet_success_rate__0__3__20200805T1112Z:amount.sv.unsorted.fwd
Allocating 2048 bytes for: wallet_success_rate__0__3__20200805T1112Z:hours.dict
Allocating 2028 bytes for: wallet_success_rate__0__3__20200805T1112Z:hours.sv.unsorted.fwd
Allocating 2048 bytes for: wallet_success_rate__0__3__20200805T1112Z:months.dict
Allocating 2028 bytes for: wallet_success_rate__0__3__20200805T1112Z:months.sv.unsorted.fwd
Allocating byte array store buffer of size 18252 for: wallet_success_rate__0__3__20200805T1112Z:orderId.dict
Allocating 2028 bytes for: wallet_success_rate__0__3__20200805T1112Z:orderId.sv.unsorted.fwd
Allocating byte array store buffer of size 18252 for: wallet_success_rate__0__3__20200805T1112Z:paymentOptionType.dict
Allocating 2028 bytes for: wallet_success_rate__0__3__20200805T1112Z:paymentOptionType.sv.unsorted.fwd
Allocating 2048 bytes for: wallet_success_rate__0__3__20200805T1112Z:minutes.dict
Allocating 2028 bytes for: wallet_success_rate__0__3__20200805T1112Z:minutes.sv.unsorted.fwd
Allocating byte array store buffer of size 18252 for: wallet_success_rate__0__3__20200805T1112Z:login.dict
Allocating 2028 bytes for: wallet_success_rate__0__3__20200805T1112Z:login.sv.unsorted.fwd
Allocating 2048 bytes for: wallet_success_rate__0__3__20200805T1112Z:years.dict
Allocating 2028 bytes for: wallet_success_rate__0__3__20200805T1112Z:years.sv.unsorted.fwd
Allocating 4096 bytes for: wallet_success_rate__0__3__20200805T1112Z:createdAt.dict
Allocating 2028 bytes for: wallet_success_rate__0__3__20200805T1112Z:createdAt.sv.unsorted.fwd
Allocating byte array store buffer of size 18252 for: wallet_success_rate__0__3__20200805T1112Z:provider.dict
Allocating 2028 bytes for: wallet_success_rate__0__3__20200805T1112Z:provider.sv.unsorted.fwd
Allocating byte array store buffer of size 18252 for: wallet_success_rate__0__3__20200805T1112Z:adtf.dict
Allocating 2028 bytes for: wallet_success_rate__0__3__20200805T1112Z:adtf.sv.unsorted.fwd
Allocating 2048 bytes for: wallet_success_rate__0__3__20200805T1112Z:days.dict
Allocating 2028 bytes for: wallet_success_rate__0__3__20200805T1112Z:days.sv.unsorted.fwd
Allocating byte array store buffer of size 18252 for: wallet_success_rate__0__3__20200805T1112Z:status.dict
Allocating 2028 bytes for: wallet_success_rate__0__3__20200805T1112Z:status.sv.unsorted.fwd
Metrics aggregation is disabled.
Successfully loaded segment wallet_success_rate__0__3__20200805T1112Z with readMode: mmap
Allocating 6564 bytes for: wallet_success_rate__0__3__20200805T1112Z:orderId.dict
Allocating 6564 bytes for: wallet_success_rate__0__3__20200805T1112Z:createdAt.dict
Allocating 6564 bytes for: wallet_success_rate__0__3__20200805T1112Z:amount.dict
Allocating 6564 bytes for: wallet_success_rate__0__3__20200805T1112Z:login.dict
Allocating byte array store buffer of size 36504 for: wallet_success_rate__0__3__20200805T1112Z:login.dict
Trying to destroy segment : wallet_success_rate__0__3__20200805T1112Z
Trying to close RealtimeSegmentImpl : wallet_success_rate__0__3__20200805T1112Z
Segment used 217068 bytes of memory for 507 rows consumed in 0 seconds
Allocating 4096 bytes for: wallet_success_rate__0__3__20200805T1112Z:amount.dict
Allocating -4 bytes for: wallet_success_rate__0__3__20200805T1112Z:amount.sv.unsorted.fwd
Exception caught:
java.lang.IllegalArgumentException: Illegal memory allocation -4 for segment wallet_success_rate__0__3__20200805T1112Z column wallet_success_rate__0__3__20200805T1112Z:amount.sv.unsorted.fwd
at shaded.com.google.common.base.Preconditions.checkArgument(Preconditions.java:122) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-43b01ef14d6d6b6933a81077b36ab755cb4b3e95]
at org.apache.pinot.core.io.readerwriter.RealtimeIndexOffHeapMemoryManager.allocate(RealtimeIndexOffHeapMemoryManager.java:77) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-43b01ef14d6d6b6933a81077b36ab755cb4b3e95]
at org.apache.pinot.core.realtime.impl.forward.FixedByteSVMutableForwardIndex.addBuffer(FixedByteSVMutableForwardIndex.java:208) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-43b01ef14d6d6b6933a81077b36ab755cb4b3e95]
at org.apache.pinot.core.realtime.impl.forward.FixedByteSVMutableForwardIndex.<init>(FixedByteSVMutableForwardIndex.java:77) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-43b01ef14d6d6b6933a81077b36ab755cb4b3e95]
at org.apache.pinot.core.indexsegment.mutable.MutableSegmentImpl.<init>(MutableSegmentImpl.java:263) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-43b01ef14d6d6b6933a81077b36ab755cb4b3e95]
at org.apache.pinot.tools.realtime.provisioning.MemoryEstimator.getMemoryForConsumingSegmentPerPartition(MemoryEstimator.java:280) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-43b01ef14d6d6b6933a81077b36ab755cb4b3e95]
at org.apache.pinot.tools.realtime.provisioning.MemoryEstimator.estimateMemoryUsed(MemoryEstimator.java:223) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-43b01ef14d6d6b6933a81077b36ab755cb4b3e95]
at org.apache.pinot.tools.admin.command.RealtimeProvisioningHelperCommand.execute(RealtimeProvisioningHelperCommand.java:227) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-43b01ef14d6d6b6933a81077b36ab755cb4b3e95]
at org.apache.pinot.tools.admin.PinotAdministrator.execute(PinotAdministrator.java:156) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-43b01ef14d6d6b6933a81077b36ab755cb4b3e95]
at org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:168) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-43b01ef14d6d6b6933a81077b36ab755cb4b3e95]
Neha Pawar
_sampleSegmentConsumedSeconds = (int)(_totalDocsInSampleSegment/ingestionRate);
Apoorva Moghey
08/06/2020, 5:21 AM