https://pinot.apache.org/ logo
Join Slack
Powered by
I increased more replicas in Pinot-Server from 1 t...
# troubleshooting
a
I increased more replicas in Pinot-Server from 1 to 3 https://github.com/apache/pinot/blob/master/kubernetes/helm/pinot/values.yaml#L272 After I brought it back up the the table in controller console was no longer visible.I did not see any exception in the logs I see the below message
Copy code
Instance events-pinot-k8s-controller-0.cdp-dl-pinot-k8s-controller-headless.events-pinot.svc.cluster.local_9000 is not leader of cluster events-pinot due to current session 20266278dd90003 does not match leader session │
m
Do you see the table in IdealState or ExternalView in the ZK Browser?
a
No I am not able tto see the table in etierh of these views
In External View BrokerResource
Copy code
"mapFields": {
    "events_REALTIME": {
      "Broker_events-pinot-k8s-broker-1.events-pinot-k8s-broker-headless.dp-metrics-pinot.svc.cluster.local_8099": "ERROR"
    }
But in IdealStates brokerResource
Copy code
"mapFields": {
    "events_REALTIME": {
      "Broker_events-pinot-k8s-broker-1.events-pinot-k8s-broker-headless.dp-metrics-pinot.svc.cluster.local_8099": "ONLINE"
    }
m
There is no entry for table in idealstate or external view? What operation did you perform? Seems to me you might have scaled down everything to zero (hence losing everything)?
a
I increased the replicas from 1 to 3 for servers , 1 to 2 for controller and broker and when it got deployed...I noticed there was no table
m
What does deployment do? Does it rip off existing cluster? If so, then yes, this will happen
a
No it does not do that.. I deployed it on another environment it came fine.But I decided to then create the same table again via swagger and it was successful but still I am not able to see the table
This deployment is via k8s
m
I can’t think of a reason why that would happen. Tables won’t just disappear like that. Do the logs persist across deployment? If so, anything on controller log regarding table deletion?
a
I did not see anything other than the log I posted above... but I will go and check again
@Mayank We had to teardown the entire kubernetes namespace including a pvc volume and then create a brand new pinot deployment with new table and schema with the same name.But it does have the data.We have the old data or segments stored in s3 cold storage.How can we restore those segments back in QA ? We tried "Reload Segments" but that did not work
m
If offline table, you can simply re-push the data. If realtime, currently there isn’t an easy way. For future, you should keep ZK up and running to ensure that you don’t lose tables.
Is this in production cluster?
a
Yes ... this is REALTIME upsert table
and prod data
d
if we had a ZK snapshot available, what are the steps we would take to restore the table
m
Restore the snapshot, assuming all data in S3 is available on the same path, it should just come up.
Let me know how it goes.
d
Yeah, we'll have to know for next time, right now our only options are to replay the data or see if we can recreate the zookeeper snapshot from an EBS volume snapshot
We're going to have to figure out a nice way to automate having Pinot in kubernetes snapshot the data to an object store
m
I have update the Pinot docs to indicate that the helm chart and other setup discussed in running pinot sections are to be used as reference only. And that ZK + DeepStore is needed for recovery steps.
a
@Dave Ortiz: Let us know what steps you do end up taking. We might potentially run into this problem at some point
@Mayank: Is there a doc which points to how to perform recovery
p
Is table deletion logged to logs by default? If the segments are on an ebs volume, and the ebs volume goes away during deployment, and is replaced by a new ebs volume, wouldn't the server download the segments from the deep store like S3 etc? And even if that didn't happen, shouldn't there be at least an empty table with no data? Unless of course zookeeper cluster was lost. In that case having zookeeper state snapshot would be helpful right?
m
If zk is intact and deep store is intact, then we have durability. In this case zk wasn’t, so table was lost
👍 1
2 Views
Open in Slack
PreviousNext
Powered by