https://pinot.apache.org/ logo
Join Slack
Powered by
this is another thing i am seeing in logs ```2021/...
# troubleshooting
p
this is another thing i am seeing in logs
Copy code
2021/12/02 21:34:27.105 ERROR [LLRealtimeSegmentDataManager_km_mp_play_startree__48__1__20211202T1919Z] [km_mp_play_startree__48__1__20211202T1919Z] Holding after response from Controller: {"offset":-1,"streamPartitionMsgOffset":null,"buildTimeSec":-1,"isSplitCommitType":false,"status":"NOT_SENT"}
Killed
i don't have the part before this right now but can update once it happens again.
m
How are you running, seems like some external process is killing it (may be due to resource usage violation)?
If so, you probably need to increase resources.
Servers are not expected to die because of something that Pinot code did.
p
i am basically running the manual launches commands on multiple instances to run broker, server and controller. i have 8 servers, 3 brokers and 3 controller instances. each instance on a r4.xlarge ec2 instance ( 4 cores, 30.5 gb ram and 1024 gb ssd disk)
Copy code
[pbagrecha@pinot-startree-server-dev-1003 apache-pinot-0.9.0-bin]$ bin/pinot-admin.sh StartServer -zkAddress 10.220.10.122:2191
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/pbagrecha/apache-pinot-0.9.0-bin/lib/pinot-all-0.9.0-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/pbagrecha/apache-pinot-0.9.0-bin/plugins/pinot-file-system/pinot-s3/pinot-s3-0.9.0-shaded.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/pbagrecha/apache-pinot-0.9.0-bin/plugins/pinot-environment/pinot-azure/pinot-azure-0.9.0-shaded.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/pbagrecha/apache-pinot-0.9.0-bin/plugins/pinot-input-format/pinot-parquet/pinot-parquet-0.9.0-shaded.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/pbagrecha/apache-pinot-0.9.0-bin/plugins/pinot-metrics/pinot-yammer/pinot-yammer-0.9.0-shaded.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/pbagrecha/apache-pinot-0.9.0-bin/plugins/pinot-metrics/pinot-dropwizard/pinot-dropwizard-0.9.0-shaded.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See <http://www.slf4j.org/codes.html#multiple_bindings> for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.codehaus.groovy.reflection.CachedClass (file:/home/pbagrecha/apache-pinot-0.9.0-bin/lib/pinot-all-0.9.0-jar-with-dependencies.jar) to method java.lang.Object.finalize()
WARNING: Please consider reporting this to the maintainers of org.codehaus.groovy.reflection.CachedClass
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2021/12/02 21:33:52.382 INFO [StartServerCommand] [main] Executing command: StartServer -clusterName PinotCluster -serverHost null -serverPort 8098 -serverAdminPort 8097 -dataDir /tmp/data/pinotServerData -segmentDir /tmp/data/pinotSegments -zkAddress 10.220.10.122:2191
2021/12/02 21:33:52.415 INFO [StartServiceManagerCommand] [main] Executing command: StartServiceManager -clusterName PinotCluster -zkAddress 10.220.10.122:2191 -port -1 -bootstrapServices []
2021/12/02 21:33:52.416 INFO [StartServiceManagerCommand] [main] Starting a Pinot [SERVICE_MANAGER] at 1.032s since launch
2021/12/02 21:33:52.422 INFO [StartServiceManagerCommand] [main] Started Pinot [SERVICE_MANAGER] instance [ServiceManager_pinot-startree-server-dev-1003.localdomain_-1] at 1.037s since launch
2021/12/02 21:33:52.425 INFO [StartServiceManagerCommand] [Start a Pinot [SERVER]] Starting a Pinot [SERVER] at 1.04s since launch
Dec 02, 2021 9:34:57 PM org.glassfish.grizzly.http.server.NetworkListener start
INFO: Started listener bound to [0.0.0.0:8097]
Dec 02, 2021 9:34:57 PM org.glassfish.grizzly.http.server.HttpServer start
INFO: [HttpServer] Started.
2021/12/02 21:36:27.182 INFO [StartServiceManagerCommand] [Start a Pinot [SERVER]] Started Pinot [SERVER] instance [Server_10.220.5.143_8098] at 155.798s since launch
Killed
Copy code
[pbagrecha@pinot-startree-server-dev-1003 apache-pinot-0.9.0-bin]$ echo $JAVA_OPTS
-Xms4G -Xmx16G -Dpinot.admin.system.exit=false -Dpinot.server.instance.realtime.alloc.offheap=true
seems like both cpu and memory usage are going up. is this something that can be mitigated by increasing the number of servers? or needs a bigger server? or both?
figured out the issue. i was rolling nodes manually and chose r4.large for some of the nodes instead of r4.xlarge in the dropdown menu by mistake. r4.large has half of the memory of r4.xlarge. thanks mayank for your help.
3 Views
Open in Slack
PreviousNext
Powered by