https://pinot.apache.org/ logo
Join Slack
Powered by
How does fault tolerance work with servers in Pino...
# general
n
How does fault tolerance work with servers in Pinot? I.e., what happens when a server crashes? My guess would be that, as a helix participant, somehow the controller sees it has crashed, and as such sends out messages to other servers to take the segments from the crashed controller? Then, when the server reboots, the controller sees a new server is available, and starts distributing segments to it? Requiring a rebalance to truly get a bunch of segments back onto it?
t
In my understanding, yes that is how it happens. All helix participants, controllers maintain their state in zookeeper. As per the config/replication we’ve set the controller always works to maintain the ideal state of the cluster and it’s segments. In the case where a new server joins the helix cluster, controller triggers a rebalancing by redistributing segments across segments to maintain a balanced ideal state. It asks the new server to load segments from the deep store by putting messages into the server’s queue. This happens till the IDEAL state of the cluster is reached. Hope this helps. Correct me if I don’t make sense.
n
It does. So another question, I’m digging into the internals and it doesn’t look like there’s a message triggering the removal of a segment, other than from the transition from ONLINE to OFFLINE
I started to trace down how rebalances work, but it seems pretty complex. When a rebalance occurs, how are segments removed? What is that event, and how does it hook in? I would expect it to call 
HelixInstanceDataManager.removeSegment
t
From what I understand, there is a state assigned/attached to individual segments also. Which has info such as which servers to be present on etc,. So when a controller triggers rebalancing, I’m expecting it updates the state of the segments, which inturn changes the ideal state of the cluster, and to achieve that ideal state it asks servers to load/unload segments onto their disks/volumes. I haven’t dwelled into code myself. Would not be able to help with exact code blocks where this happens. But would love to know it. 🙂
k
@Noah Prince read a bit about Helix here https://helix.apache.org/Concepts.html
✔️ 1
once you get that following Pinot code will be easier.
all callbacks from Helix are already handled in Pinot, when you trigger rebalance - new segments are loaded via offlineToOnline callbacks
see SegmentOnlineOfflineStateModel for more details
Copy code
// Delete segment from local directory.
@Transition(from = "OFFLINE", to = "DROPPED")
public void onBecomeDroppedFromOffline(Message message, NotificationContext context) {
this gets triggered when a segment in removed from the idealstate for a specific server
n
So, it looks like that directly removes the directory but never removes it from the table data manager
Open in Slack
PreviousNext
Powered by