Dear all, In some strange scenarios, Pinot Servers...
# troubleshooting
s
Dear all, In some strange scenarios, Pinot Servers(Pinot-Server-0, Pinot-Server-1) is keep on crashing with "CrashLoopBackOff" error, zookeeper is not able to communicate to the Pinot Server. Find below few observation from our side, but we could not come to a conclusion whether is this the cause of the error. We had deployed the Pinot using helm, we have currently 2 Pinot Servers, 2 Controllers, 2 Broker and 1 Zoo Keeper pods, all deployed in the Azure Kubernetes Service (AKS). 1. Number of Segments of the table was more than 30K (last time it was around 40K) 2. The disk (PVC) attached to the Pinot Servers were reached its size (2 * 8TB were fully occupied) While the logs indicates that the server is coming up and shutting down. Please do help us to getting it resolved asap. Thanks. -----logs----------- 2025/04/14 155145.303 INFO [ServerInstance] [Start a Pinot [SERVER]] Starting data manager 2025/04/14 155145.306 INFO [HelixInstanceDataManager] [Start a Pinot [SERVER]] Helix instance data manager started 2025/04/14 155145.306 INFO [ServerInstance] [Start a Pinot [SERVER]] Finish starting data manager 2025/04/14 155420.886 INFO [PinotServiceManager] [Thread-42] Shutting down Pinot Service Manager with all running Pinot instances... 2025/04/14 155420.886 INFO [PinotServiceManager] [Thread-42] Shutting down Pinot Service Manager admin application... 2025/04/14 155420.886 INFO [PinotServiceManager] [Thread-42] Deregistering service status handler ------logs--------- I am happy to connect to get this resolved, we are going into production, with no clarity on the RC for above issue (already happened twice), we need this community support..
x
for 2, you mean disk is full on pinot server?
also please check the zookeeper disk usage
s
Zookeeper disk usage was normal at that time it was below 50%.
Who is sending the shutdown signal to PinotServiceManager or why it is shutting down after 3+ minutes. Shutting down Pinot Service Manager with all running Pinot instances...
x
for your k8s, how do you configure the resources and the startup/liveness probe
s
we had helm installed with the default template provided by pinot
x
still you need to check and tune
see if the probe is setting too short time
or the resources limit got hit
s
First thing is to know what is the root cause of this continuous restart and how to avoid this proactively. 2. how to resurrect the Pinot Server without deleting the data and reinstalling the zookeeper
x
usually for k8s pod restart, it should give you a reason
or exit code
s
sorry, I forgot to note down the exit code.
x
can you put the pinot-server statefulset definition yaml
you can run
kubectl describe sts pinot-server -n pinot-namespace -o yaml
s
pinot-server-sts.yaml
x
can you try to edit the readinessProbe
give it more time, like 10 mins
right now the readiness probe will fail after 160secs
Copy code
readinessProbe:
          failureThreshold: 10
          httpGet:
            path: /health/readiness
            port: 8097
            scheme: HTTP
          initialDelaySeconds: 60
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 10
pinot server requires loading all the segments to show itself as ready for query
it that takes longer then your readiness probe will never pass
s
So what is the best timeout value for handling 8TB+ of data in each of the Pinot Server?
x
depends on your use case, I would give 1 hour
s
does it have any other impact?
x
not in your case
downside is that if your restart is not finished in 1 hour, then you will find it out in a hour then you need to increase it again
usually if there is no extra processing, then the segment loading should complete in 10 -20 mins
1 hour is what we use in prod for less noisy
s
Where is the mapping of Table to Segments is stored, how to restore it if the Pinot Server and Zookeeper crashed. One of my engineer says that he has to recreate the Zookeeper and start loading all the data from scratch to have the mapping correctly. As my disks which stores the Table segments are in Azure Disks and even the Zookeeper bitnami folder is in Azure disk. Is it not possible to just make the server up and running and do some magical operation to make the Table up and running. I don't understand what are the missing pieces here, and everytime this crash happens I have to again re-ingest the whole data which takes around 12 days. Please help.
x
-Where is the mapping of Table to Segments is stored • Zookeeper - how to restore it if the Pinot Server and Zookeeper crashed • you should have 3 zks to form a HA system as well for backup, if all zk data/disks are gone, you have data loss. Though you can periodically backup zookeeper snapshot • Server will download data from deepstore based on the metadata in ZK, it will auto recover - One of my engineer says that he has to recreate the Zookeeper and start loading all the data from scratch to have the mapping correctly. As my disks which stores the Table segments are in Azure Disks and even the Zookeeper bitnami folder is in Azure disk. Is it not possible to just make the server up and running and do some magical operation to make the Table up and running. I don’t understand what are the missing pieces here, and everytime this crash happens I have to again re-ingest the whole data which takes around 12 days • For normal ops if your zk dies but the disk is there, then you could bring back zookeeper and pinot cluster works as it is. • If it appears that pinot cluster is blank, then it means your zookeeper restart has an issue, e.g. the data directories are not the same between restarts