Hi All , ...
# troubleshooting
v
Hi All , Unable to connect to SSL Enabled Zookeeper getting the following error 2025/04/24 095621.256 WARN [ClientCnxnSocketNetty] [zkNetty-EpollEventLoopGroup-3-1] future isn't success. io.netty.channel.StacklessClosedChannelException: null at io.netty.channel.AbstractChannel$AbstractUnsafe.ensureOpen(ChannelPromise)(Unknown Source) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] 2025/04/24 095622.259 INFO [ClientCnxn] [main-SendThread(localhost:2191)] Opening socket connection to server localhost/127.0.0.1:2191. 2025/04/24 095622.260 INFO [ClientCnxn] [main-SendThread(localhost:2191)] SASL config status: Will not attempt to authenticate using SASL (unknown error) 2025/04/24 095622.262 WARN [ChannelInitializer] [zkNetty-EpollEventLoopGroup-3-1] Failed to initialize a channel. Closing: [id: 0xd0530de0] org.apache.zookeeper.common.X509Exception$TrustManagerException: java.lang.IllegalArgumentException: Unable to auto-detect store file type from file name: /nfmt/instance/certificates/External/truststore.ks at org.apache.zookeeper.common.X509Util.createTrustManager(X509Util.java:612) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at org.apache.zookeeper.common.ClientX509Util.getTrustManager(ClientX509Util.java:198) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at org.apache.zookeeper.common.ClientX509Util.createNettySslContextForClient(ClientX509Util.java:77) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at org.apache.zookeeper.ClientCnxnSocketNetty$ZKClientPipelineFactory.initSSL(ClientCnxnSocketNetty.java:449) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at org.apache.zookeeper.ClientCnxnSocketNetty$ZKClientPipelineFactory.initChannel(ClientCnxnSocketNetty.java:438) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at org.apache.zookeeper.ClientCnxnSocketNetty$ZKClientPipelineFactory.initChannel(ClientCnxnSocketNetty.java:424) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at io.netty.channel.ChannelInitializer.initChannel(ChannelInitializer.java:129) [pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at io.netty.channel.ChannelInitializer.handlerAdded(ChannelInitializer.java:112) [pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at io.netty.channel.AbstractChannelHandlerContext.callHandlerAdded(AbstractChannelHandlerContext.java:1130) [pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:558) [pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at io.netty.channel.DefaultChannelPipeline.access$100(DefaultChannelPipeline.java:45) [pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at io.netty.channel.DefaultChannelPipeline$PendingHandlerAddedTask.execute(DefaultChannelPipeline.java:1410) [pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at io.netty.channel.DefaultChannelPipeline.callHandlerAddedForAllHandlers(DefaultChannelPipeline.java:1064) [pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at io.netty.channel.DefaultChannelPipeline.invokeHandlerAddedIfNeeded(DefaultChannelPipeline.java:599) [pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:513) [pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:428) [pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:485) [pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) [pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) [pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) [pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:405) [pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at java.base/java.lang.Thread.run(Thread.java:840) [?:?] Caused by: java.lang.IllegalArgumentException: Unable to auto-detect store file type from file name: /nfmt/instance/certificates/External/truststore.ks
v
seems like java not able to read "/nfmt/instance/certificates/External/truststore.ks" .ks is not standard extension of file.
x
you need to configure those certs using jvmOpts
v
even i have specified the type of files we are getting the below error
javax.net.ssl|DEBUG|44|nioEventLoopGroup-4-38|2025-04-29 165846.024 IST|HandshakeContext.java:305|No available cipher suite for TLSv1.3 javax.net.ssl|ERROR|44|nioEventLoopGroup-4-38|2025-04-29 165846.025 IST|TransportContext.java:375|Fatal (INTERNAL_ERROR): problem unwrapping net record ( "throwable" : { javax.net.ssl.SSLHandshakeException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate) at java.base/sun.security.ssl.HandshakeContext.<init>(HandshakeContext.java:172) at java.base/sun.security.ssl.ServerHandshakeContext.<init>(ServerHandshakeContext.java:63) at java.base/sun.security.ssl.TransportContext.kickstart(TransportContext.java:253) at java.base/sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:546) at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506) at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482) at java.base/javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679) at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:309) at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1441) at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1334) at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1383) at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:840)} ) javax.net.ssl|ALL|44|nioEventLoopGroup-4-38|2025-04-29 165846.025 IST|SSLSessionImpl.java:1222|Invalidated session: Session(1745926125999|SSL_NULL_WITH_NULL_NULL)
even i have specified all JAVA_OPTS like below -Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty -Djavax.net.debug=ssl:handshake \ -Djdk.tls.server.protocols=TLSv1.2,TLSv1.3 \ -Djdk.tls.client.cipherSuites=$ALLOWED_HTTP_SSL_CIPHERS_LIST \ -Djdk.tls.server.cipherSuites=$ALLOWED_HTTP_SSL_CIPHERS_LIST \ -Djdk.tls.client.protocols=TLSv1.2,TLSv1.3 \ -Dzookeeper.ssl.quorum=true \ -Dzookeeper.client.secure=true \ -Dzookeeper.ssl.keyStore.type=JKS -Dzookeeper.ssl.truststore.type=JKS \ -Dzookeeper.ssl.cipher.suites=$ALLOWED_HTTP_SSL_CIPHERS_LIST \ -Dzookeeper.ssl.keyStore.location=/nfmt/instance/certificates/External/keystore.jks \ -Dzookeeper.ssl.keyStore.password=test \ -Dzookeeper.ssl.trustStore.location=/nfmt/instance/certificates/External/truststore.jks \ -Dzookeeper.ssl.trustStore.password=test \ -Djavax.net.ssl.keyStore=/nfmt/instance/certificates/External/keystore.jks \ -Djavax.net.ssl.keyStorePassword=test \ -Djavax.net.ssl.trustStore=/nfmt/instance/certificates/External/truststore.jks \ -Djavax.net.ssl.trustStorePassword=test
x
what is this
$ALLOWED_HTTP_SSL_CIPHERS_LIST
v
ciphers list which we are supporting
TLS_RSA_WITH_AES_128_CBC_SHA
@Xiang Fu: is pinot support ssl communication between controllers and zookeeper ?
we are seeing zookeeper data logs is storing the ssl passwords which are using in realtime table creation
x
hmm, can you check with just
TLSv1.2
?
the error is
javax.net.ssl.SSLHandshakeException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate)
so it means the config for cipher is wrong
ChatGPT says TLSv1.3 doesn’t allow customized cipherSuite
I would suggest you to first configure zookeeper tls then try to start a simple zookeeper client with tls configs
v
i tried with tls1.2 as well same error
x
can you try to remove
Copy code
-Djdk.tls.client.cipherSuites=$ALLOWED_HTTP_SSL_CIPHERS_LIST \
                      -Djdk.tls.server.cipherSuites=$ALLOWED_HTTP_SSL_CIPHERS_LIST \
and
-Dzookeeper.ssl.cipher.suites=$ALLOWED_HTTP_SSL_CIPHERS_LIST \
v
without cipher list also same error
-Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty \ -Djavax.net.debug=ssl:handshake \ -Dhttps.protocols=TLSv1.2,TLSv1.3 \ -Djdk.tls.server.protocols=TLSv1.2,TLSv1.3 \ -Djdk.tls.client.protocols=TLSv1.2,TLSv1.3 \ -Dzookeeper.ssl.quorum=true \ -Dzookeeper.client.secure=true \ -Dzookeeper.ssl.keyStore.type=JKS -Dzookeeper.ssl.truststore.type=JKS \ -Dzookeeper.ssl.keyStore.location=/nfmt/instance/certificates/External/keystore.jks \ -Dzookeeper.ssl.keyStore.password=test \ -Dzookeeper.ssl.trustStore.location=/nfmt/instance/certificates/External/truststore.jks \ -Dzookeeper.ssl.trustStore.password=test \ -Djavax.net.ssl.keyStore=/nfmt/instance/certificates/External/keystore.jks \ -Djavax.net.ssl.keyStorePassword=test \ -Djavax.net.ssl.trustStore=/nfmt/instance/certificates/External/truststore.jks \ -Djavax.net.ssl.trustStorePassword=test
x
ok
for zookeeper are you using secureClientPort?
v
yes
secureClientPort=2191
x
both not working ?
v
yes
it is not working
is there a way to make a secure communication between controller and zookeeper
x
then I would suggest to write a simple application just to init zkclient and list all znode. This is to test the zookeeper connectivity
v
@Xiang Fu: is secure communication is supported in pinot ?
x
yes, we have users enabled this
v
okay can we get any sample configuration or example on this
so that i can co-relate that and achieve this
x
yeah, I haven’t configured myself, will see our SRE people chime in 🙂
v
thanks 🙂
x
I will also ask to update the docs for this
v
that will help a lot and will waiting for the same
@Xiang Fu: any update on this and if any configurations are there with you could you please provide it
@Xiang Fu / @Mayank: can we get an update on this so that we can proceed with ssl
x
sure, let me check
sorry for the delay