Hello guys, We are using Pinot 1.1 and we are cur...
# troubleshooting
k
Hello guys, We are using Pinot 1.1 and we are currently investigating an issue that we have for 3rd time for the last 2 weeks. We are getting a lot of these error messages: INFO 2025-07-16T183117.674888458Z [resource.labels.containerName: server] 2025/07/16 183117.672 ERROR [KafkaPartitionLevelConnectionHandler] [auctionsStatsRedis__5__0__20250619T0844Z] Caught exception while creating Kafka consumer, giving up ERROR [RealtimeSegmentDataManager_auctionsNew__6__4__20250614T0632Z] [auctionsNew__6__4__20250614T0632Z] Exception while in work [NetworkClient] [auctionsStatsRedis__1__22__20250716T1027Z] [Consumer clientId=auctionsStatsRedis_REALTIME-auctionsStatsRedis-1, groupId=null] Error connecting to node events-prod-cluster-kafka-0.events-prod-cluster-kafka-brokers.kafka-prod.svc:9092 (id: 0 rack: null) 2025/07/16 183137.710 ERROR [ServerSegmentCompletionProtocolHandler] [customKeys_2025_07_5d6254cd_c8e8_423d_b196_73f016e023cb__7__0__20250625T1419Z] Could not send request http://pinot-prod-controller-1.pinot-prod-controller-headless.pinot.svc.cluster.local:9000/segmentStoppedConsuming?reason=org.apache.pinot.shaded.org.apache.kafka.common.KafkaException&streamPartitionMsgOffset=0&instance=Server_pinot-prod-server-2.pinot-prod-server-headless.pinot.svc.cluster.local_8098&offset=-1&name=customKeys_2025_07_5d6254cd_c8e8_423d_b196_73f016e023cb__7__0__20250625T1419Z And after that some tables are missing segments. Right now all our currently running tables lost all their segments and can't be queried. Do you have any ideas what is going on and why?
m
This seems like connectivity to Kafka was lost. But that should only impacting consuming segments. What’s the retention on those tables? Perhaps the committed segments are deleted due to retention?
k
@Mayank Retntion is 5 years. We have lost connection to Kafka for less than a minute 2 hours ago.
I don't think it is connectivity issue at all as we are deploying all of this into one of Google's datacenters and we hit this 3 times for two weeks. Google network should be way more stable.
Whatever it is my main concern now is why all my old segments are mssing. I mean all consuming tables report that all of their segments are missing. We are talking about very small tables here. The biggest one is 1.2 GB the smallest one is 650KB.
m
The log definitely suggests Kafka connectivity issue, and there are retries. And that should have nothing to do with segments missing. Let me ping you to understand more.
k
What do you mean by ping?
m
DM you
k
Thanks Mayank. My Pinot is running again. I still need to get to the root cause of this but at least it is working again.
m
Yeah to root cause, let's look at why ingestion stopped for those consuming segments
k
@Mayank Any suggestions how to approach this?
m
We should take one of the latest segments from last 2-3 days that was OFFLINE, and then grep for it in controller and server logs
It will tell us all the things that happened to it
k
Let's hope I still have some of those segments
@Mayank Is there any log message that I can grep to see what segments I reloaded yesterday?
I found them in the logs.
m
Segment name
k
Sent 3 reload messages for segment: auctionsStats__0__295__20250703T1233Z in table: auctionsStats_REALTIME
I believe it should be something like this
m
This is from your reload call
Controller will send some state transition
k
Yes. Now I will look into the above segment name to see what I can find
m
Server will receive and try to follow: in this case create a new consuming segment.
Yes