Hi, can you give me an example how to configure Kafka TLS/SSL on my kubernetes pinot cluster?

@Sandeep R are you asking how to mount these files in the server when deploying Pinot in k8s?

@Navina That's right.

@Xiang Fu can you help here? I think he needs to define volume mounts and add the credential files there?

You can add and mount those files as either ConfigMap or Secrets. You can modify this helmChart to add those: https://github.com/apache/pinot/blob/master/kubernetes/helm/pinot/templates/server/statefulset.yaml

thank you for your help, I was able to configure on each of pinot server,broker,controller( I was not sure which component it reads the certs so I added it in all)

for i in {0..2}; do kubectl cp /tmp/client.truststore.jks pinot-controller-$i:/opt/pinot -n pinot-quickstart; done
 for i in {0..2}; do kubectl cp /tmp/client.truststore.jks pinot-broker-$i:/opt/pinot -n pinot-quickstart; done
 for i in {0..2}; do kubectl cp /tmp/client.truststore.jks pinot-server-$i:/opt/pinot -n pinot-quickstart; done

in table I have added below settings, and this is working for me.

"security.protocol": "SSL",

"ssl.truststore.location": "/opt/pinot/client.truststore.jks",

"ssl.truststore.password": "xxxxxxxxxxxxxxxxxxx"

ah, suggest to copy it over to the data directory (your ebs) so the jks files are kept even after pod restart

Hi @Xiang Fu seeing `invalid`while converting segments any idea why this is showing in such way? is this can be ignored?

.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
	at org.apache.pinot.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
	at org.apache.pinot.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
	at org.apache.pinot.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
	at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.io.IOException: Connection reset by peer
	at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[?:?]
	at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[?:?]
	at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:276) ~[?:?]
	at sun.nio.ch.IOUtil.read(IOUtil.java:233) ~[?:?]
	at sun.nio.ch.IOUtil.read(IOUtil.java:223) ~[?:?]
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:356) ~[?:?]
	at org.apache.pinot.shaded.io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:256) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
	at org.apache.pinot.shaded.io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1132) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
	at org.apache.pinot.shaded.io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:357) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
	at org.apache.pinot.shaded.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:151) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
	... 8 more
2023/12/12 04:35:48.226 INFO [QueryScheduler] [pqr-10] Processed requestId=1904975199000000193,table=test_OFFLINE,segments(queried/processed/matched/consumingQueried/consumingProcessed/consumingMatched/invalid/limit/value)=71/1/1/-1/0/0/0/70/0,schedulerWaitMs=1,reqDeserMs=1,totalExecMs=5,resSerMs=0,totalTimeMs=8,minConsumingFreshnessMs=-1,broker=Broker_172.19.0.4_8099,numDocsScanned=10,scanInFilter=0,scanPostFilter=1380,sched=FCFS,threadCpuTimeNs(total/thread/sysActivity/resSer)=0/0/0/0
2023/12/12 04:35:49.506 INFO [QueryScheduler] [pqr-11] Processed requestId=1904975199000000194,table=test_OFFLINE,segments(queried/processed/matched/consumingQueried/consumingProcessed/consumingMatched/invalid/limit/value)=71/1/1/-1/0/0/0/70/0,schedulerWaitMs=0,reqDeserMs=1,totalExecMs=4,resSerMs=1,totalTimeMs=6,minConsumingFreshnessMs=-1,broker=Broker_172.19.0.4_8099,numDocsScanned=10,scanInFilter=0,scanPostFilter=1380,sched=FCFS,threadCpuTimeNs(total/thread/sysActivity/resSer)=0/0/0/0
2023/12/12 04:35:50.453 INFO [QueryScheduler] [pqr-12] Processed requestId=1904975199000000195,table=test_OFFLINE,segments(queried/processed/matched/consumingQueried/consumingProcessed/consumingMatched/invalid/limit/value)=71/1/1/-1/0/0/0/70/0,schedulerWaitMs=0,reqDeserMs=0,totalExecMs=5,resSerMs=0,totalTimeMs=5,minConsumingFreshnessMs=-1,broker=Broker_172.19.0.4_8099,numDocsScanned=10,scanInFilter=0,scanPostFilter=1380,sched=FCFS,threadCpuTimeNs(total/thread/sysActivity/resSer)=0/0/0/0
2023/12/12 04:58:55.762 INFO [QueryScheduler] [pqr-13] Processed requestId=1904975199000000196,table=test_OFFLINE,segments(queried/processed/matched/consumingQueried/consumingProcessed/consumingMatched/invalid/limit/value)=71/1/1/-1/0/0/0/70/0,schedulerWaitMs=1,reqDeserMs=2,totalExecMs=4,resSerMs=1,totalTimeMs=8,minConsumingFreshnessMs=-1,broker=Broker_172.19.0.4_8099,numDocsScanned=10,scanInFilter=0,scanPostFilter=1380,sched=FCFS,threadCpuTimeNs(total/thread/sysActivity/resSer)=0/0/0/0
2023/12/12 04:58:56.644 INFO [QueryScheduler] [pqr-14] Processed requestId=1904975199000000197,table=test_OFFLINE,segments(queried/processed/matched/consumingQueried/consumingProcessed/consumingMatched/invalid/limit/value)=71/1/1/-1/0/0/0/70/0,schedulerWaitMs=0,reqDeserMs=1,totalExecMs=5,resSerMs=0,totalTimeMs=6,minConsumingFreshnessMs=-1,broker=Broker_172.19.0.4_8099,numDocsScanned=10,scanInFilter=0,scanPostFilter=1380,sched=FCFS,threadCpuTimeNs(total/thread/sysActivity/resSer)=0/0/0/0
2023/12/12 04:59:18.497 INFO [QueryScheduler] [pqr-15] Processed requestId=1904975199000000198,table=test_OFFLINE,segments(queried/processed/matched/consumingQueried/consumingProcessed/consumingMatched/invalid/limit/value)=71/1/1/-1/0/0/0/70/0,schedulerWaitMs=0,reqDeserMs=1,totalExecMs=6,resSerMs=0,totalTimeMs=7,minConsumingFreshnessMs=-1,broker=Broker_172.19.0.4_8099,numDocsScanned=10,scanInFilter=0,scanPostFilter=1380,sched=FCFS,threadCpuTimeNs(total/thread/sysActivity/resSer)=0/0/0/0
2023/12/12 04:59:38.935 INFO [QueryScheduler] [pqr-16] Processed requestId=1904975199000000199,table=test_OFFLINE,segments(queried/processed/matched/consumingQueried/consumingProcessed/consumingMatched/invalid/limit/value)=71/1/1/-1/0/0/0/70/0,schedulerWaitMs=0,reqDeserMs=1,totalExecMs=5,resSerMs=1,totalTimeMs=7,minConsumingFreshnessMs=-1,broker=Broker_172.19.0.4_8099,numDocsScanned=10,scanInFilter=0,scanPostFilter=1380,sched=FCFS,threadCpuTimeNs(total/thread/sysActivity/resSer)=0/0/0/0
2023/12/12 05:00:23.965 INFO [QueryScheduler] [pqr-17] Processed requestId=1904975199000000200,table=test_OFFLINE,segments(queried/processed/matched/consumingQueried/consumingProcessed/consumingMatched/invalid/limit/value)=71/1/1/-1/0/0/0/70/0,schedulerWaitMs=1,reqDeserMs=5,totalExecMs=8,resSerMs=1,totalTimeMs=16,minConsumingFreshnessMs=-1,broker=Broker_172.19.0.4_8099,numDocsScanned=10,scanInFilter=0,scanPostFilter=1380,sched=FCFS,threadCpuTimeNs(total/thread/sysActivity/resSer)=0/0/0/0
2023/12/12 05:06:20.963 INFO [QueryScheduler] [pqr-18] Processed requestId=1904975199000000201,table=test_OFFLINE,segments(queried/processed/matched/consumingQueried/consumingProcessed/consumingMatched/invalid/limit/value)=71/1/1/-1/0/0/0/70/0,schedulerWaitMs=2,reqDeserMs=0,totalExecMs=10,resSerMs=0,totalTimeMs=13,minConsumingFreshnessMs=-1,broker=Broker_172.19.0.4_8099,numDocsScanned=10,scanInFilter=0,scanPostFilter=1380,sched=FCFS,threadCpuTimeNs(total/thread/sysActivity/resSer)=0/0/0/0
2023/12/12 05:06:22.485 INFO [QueryScheduler] [pqr-19] Processed requestId=1904975199000000202,table=test_OFFLINE,segments(queried/processed/matched/consumingQueried/consumingProcessed/consumingMatched/invalid/limit/value)=71/1/1/-1/0/0/0/70/0,schedulerWaitMs=0,reqDeserMs=1,totalExecMs=5,resSerMs=1,totalTimeMs=7,minConsumingFreshnessMs=-1,broker=Broker_172.19.0.4_8099,numDocsScanned=10,scanInFilter=0,scanPostFilter=1380,sched=FCFS,threadCpuTimeNs(total/thread/sysActivity/resSer)=0/0/0/0
2023/12/12 05:06:23.379 INFO [QueryScheduler] [pqr-20] Processed requestId=1904975199000000203,table=test_OFFLINE,segments(queried/processed/matched/consumingQueried/consumingProcessed/consumingMatched/invalid/limit/value)=71/1/1/-1/0/0/0/70/0,schedulerWaitMs=0,reqDeserMs=0,totalExecMs=6,resSerMs=0,totalTimeMs=6,minConsumingFreshnessMs=-1,broker=Broker_172.19.0.4_8099,numDocsScanned=10,scanInFilter=0,scanPostFilter=1380,sched=FCFS,threadCpuTimeNs(total/thread/sysActivity/resSer)=0/0/0/0
2023/12/12 05:06:24.189 INFO [QueryScheduler] [pqr-21] Processed requestId=1904975199000000204,table=test_OFFLINE,segments(queried/processed/matched/consumingQueried/consumingProcessed/consumingMatched/invalid/limit/value)=71/1/1/-1/0/0/0/70/0,schedulerWaitMs=0,reqDeserMs=0,totalExecMs=6,resSerMs=0,totalTimeMs=6,minConsumingFreshnessMs=-1,broker=Broker_172.19.0.4_8099,numDocsScanned=10,scanInFilter=0,scanPostFilter=1380,sched=FCFS,threadCpuTimeNs(total/thread/sysActivity/resSer)=0/0/0/0
2023/12/12 05:07:01.356 INFO [QueryScheduler] [pqr-22] Processed requestId=1904975199000000205,table=test_OFFLINE,segments(queried/processed/matched/consumingQueried/consumingProcessed/consumingMatched/invalid/limit/value)=71/71/71/-1/0/0/0/0/0,sc

what’s the issue right now? still cannot consume?

I can, but I am seeing this in logs, like

consumingMatched/invalid/limitProcessed

requestId=1904975199000000204,table=test_OFFLINE,segments(queried/processed/matched/consumingQueried/consumingProcessed/consumingMatched/invalid/limit/value)=71/1/1/-1/0/0/0/70/0,schedulerWaitMs=0,reqDeserMs=0,totalExecMs=6,resSerMs=0,totalTimeMs=6,minConsumingFreshnessMs=-1,broker=Broker_172.19.0.4_8099,numDocsScanned=10,scanInFilter=0,scanPostFilter=1380,sched=FCFS,threadCpuTimeNs(total/thread/sysActivity/resSer)=0/0/0/0
2023/12/12 05:07:01.356 INFO [QueryScheduler] [pqr-22]

yes, this is query stats, it’s normal

Ok