Pedro Silva
05/24/2021, 1:31 PMDaniel Lavoie
05/24/2021, 1:35 PMPedro Silva
05/24/2021, 1:39 PMDaniel Lavoie
05/24/2021, 1:41 PMPedro Silva
05/24/2021, 1:42 PMDaniel Lavoie
05/24/2021, 1:43 PMPedro Silva
05/24/2021, 1:51 PMDaniel Lavoie
05/24/2021, 1:51 PMPedro Silva
05/24/2021, 1:51 PMDaniel Lavoie
05/24/2021, 1:53 PMPedro Silva
05/24/2021, 1:53 PMDaniel Lavoie
05/24/2021, 1:53 PMPedro Silva
05/24/2021, 1:53 PMDaniel Lavoie
05/24/2021, 1:54 PMPedro Silva
05/24/2021, 1:54 PMDaniel Lavoie
05/24/2021, 1:58 PMMayank
Pedro Silva
05/24/2021, 1:59 PMDaniel Lavoie
05/24/2021, 1:59 PMParts or consuming segment are on direct memory that can OOM if not enough (unlike mmap).That will cause a JVM OOM, not a K8S OOM
Mayank
Pedro Silva
05/24/2021, 2:00 PMDaniel Lavoie
05/24/2021, 2:00 PMOOM
message in the pods eventsMayank
Pedro Silva
05/24/2021, 2:06 PM#
# A fatal error has been detected by the Java Runtime Environment:
#
# [thread 140238581155584 also had an error]
SIGBUS (0x7) at pc=0x00007f8c70a013c2, pid=9, tid=0x00007f8bd167c700
#
# JRE version: OpenJDK Runtime Environment (8.0_292-b10) (build 1.8.0_292-b10)
# Java VM: OpenJDK 64-Bit Server VM (25.292-b10 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# v ~StubRoutines::jbyte_disjoint_arraycopy
#
# Core dump written. Default location: /opt/pinot/core or core.9
#
[thread 140238499997440 also had an error]
# An error report file with more information is saved as:
# /opt/pinot/hs_err_pid9.log
[thread 140237039417088 also had an error]
[thread 140238498944768 also had an error]
[thread 140238586418944 also had an error]
[thread 140238396491520 also had an error]
[thread 140238582208256 also had an error]
[thread 140238587471616 also had an error]
#
# If you would like to submit a bug report, please visit:
# <http://bugreport.java.com/bugreport/crash.jsp>
#
Aborted (core dumped)
Got this with a 3GB java heap and a pod memory request of 6GB (limit 7GB)Daniel Lavoie
05/24/2021, 2:06 PMPedro Silva
05/24/2021, 2:06 PMMayank
Pedro Silva
05/24/2021, 2:11 PMsegmentPartitionConfig
config of the pinot table?Mayank
Pedro Silva
05/24/2021, 2:15 PMMayank
Pedro Silva
05/24/2021, 2:18 PMException in thread "HitExecutionView__12__49__20210524T1141Z" java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code
Off-heap memory I assume?Exception in thread "HitExecutionView__12__49__20210524T1141Z" java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code
at org.apache.pinot.segment.local.function.InbuiltFunctionEvaluator$FunctionExecutionNode.execute(InbuiltFunctionEvaluator.java:116)
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGBUS (0x7) at pc=0x00007f51c5052416, pid=9, tid=0x00007f50a5553700
#
[thread 139984278378240 also had an error]
# JRE version: OpenJDK Runtime Environment (8.0_292-b10) (build 1.8.0_292-b10)
# Java VM: OpenJDK 64-Bit Server VM (25.292-b10 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# v ~StubRoutines::jbyte_disjoint_arraycopy
#
# Core dump written. Default location: /opt/pinot/core or core.9
#
# An error report file with more information is saved as:
# /opt/pinot/hs_err_pid9.log
at org.apache.pinot.segment.local.function.InbuiltFunctionEvaluator.evaluate(InbuiltFunctionEvaluator.java:87)
at org.apache.pinot.segment.local.recordtransformer.ExpressionTransformer.transform(ExpressionTransformer.java:95)
at org.apache.pinot.segment.local.recordtransformer.CompositeTransformer.transform(CompositeTransformer.java:82)
at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.processStreamEvents(LLRealtimeSegmentDataManager.java:509)
at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.consumeLoop(LLRealtimeSegmentDataManager.java:416)
at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:556)
at java.lang.Thread.run(Thread.java:748)
[thread 139984280483584 also had an error]
#
# If you would like to submit a bug report, please visit:
# <http://bugreport.java.com/bugreport/crash.jsp>
#
Aborted (core dumped)
LLRealtimeSegmentDataManager$PartitionConsumer.run
Appears to be in the partition consumerMayank
Pedro Silva
05/24/2021, 2:21 PMMayank
Pedro Silva
05/24/2021, 2:22 PMMayank
Pedro Silva
05/24/2021, 2:23 PMMayank
Pedro Silva
05/24/2021, 2:26 PMDaniel Lavoie
05/24/2021, 2:26 PMMayank
Pedro Silva
05/24/2021, 2:27 PMMayank
Pedro Silva
05/24/2021, 2:28 PMMayank
Mayank
Pedro Silva
05/24/2021, 2:30 PMDaniel Lavoie
05/24/2021, 2:32 PMbin
Pedro Silva
05/24/2021, 2:55 PMdocker run --rm -v /home/pedro/dev/Pinot:/tmp/volume apachepinot/pinot:release-0.7.1 RealtimeProvisioningHelper \
-ingestionRate 1000 \
-numPartitions 16 \
-retentionHours 720 \
-numRows 50000000 \
-tableConfigFile /tmp/volume/specs/tables/HitExecutionView_REALTIME.json \
-schemaWithMetadataFile /tmp/volume/specs/schemas/HitExecutionView.json
Executing command: RealtimeProvisioningHelper -tableConfigFile /tmp/volume/specs/tables/HitExecutionView_REALTIME.json -numPartitions 16 -pushFrequency null -numHosts 2,4,6,8,10,12,14,16 -numHours 2,3,4,5,6,7,8,9,10,11,12 -schemaWithMetadataFile /tmp/volume/specs/schemas/HitExecutionView.json -numRows 50000000 -ingestionRate 1000 -maxUsableHostMemory 48G -retentionHours 720