Hello everybody! I need support to set up a cluster. I followed the instructions explained by @Neha Pawar in her video. Everything works as expected but i did that test locally. Now, I should organize all the components inside a real cluster that has 2/3 servers. I need to understand how to organize the components for an high-availability architecture. Obviously, i am talking about a very small cluster, so take "high-availability" with a grain of salt :) My doubt is regarding the distribution of the components over the servers. For example, seeing the video, the Zookeeper instance is just one, we start it with

pinot-admin.sh StartZookeeper -zkPort 2181

so the first question is: what about if the server with Zookeeper goes down? Can we share two or more zookeeper instances over multiple servers? Supposing we can create multiple zookeeper instances does every machine should also have its own Controller, Broker and Server components? Because having more than one broker/controller on the same machine does not have much sense to me, maybe for very high traffic? Could someone explain it a little bit more? Thanks.

With

pinot-admin

you can only start ZooKeeper in standalone mode. If you want to start a multi-node ZooKeeper cluster, you need to configure it separately (here I find a tutorial: https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-an-apache-zookeeper-cluster-on-ubuntu-18-04)

@Jackie ok, I will follow that tutorial to create a cluster for zookeeper but if I only have 2 servers should I put all the components (zookeeper, container, broker et) on both server like a replica?

With only 2 servers, in order to get the highest availability, you need to put all the components on each server, and also configure the table with 2 replicas

How does Pinot split the segments across these two machines?

When you upload segments, the controller creates an assignment from segment to servers, uniformly distributing them across available servers. The controller has the segment tar. Each server will download the segments it is assigned, from the controller

Hi @Neha Pawar thank you! Just one more question. You said the controller creates the assignment and then each server will download its segments but what about if one server goes down? Do I lose those segments or is there a way to hold a replica of the segments of the server A inside server B and viceversa?

If you have 2 servers, and you set replication 2, then every segment will be available on both servers. Query traffic will be distributed to both servers. So if 1 server goes down, it’s okay, because other server has all the data. All traffic will be directed to the single server. When the server comes back up, it will start serving queries again. If you need to replace the down server, you start new server, untag the down server, and call Rebalance. Then new server will download the segments that belonged to the down server.

@Neha Pawar is it correct to say that when the two servers are online they serve their own segments, so they are not just clones, but when a server goes down the other will put the replica data live, right? Then.... you did and example of replication with two servers. What about if we set replication=2 with three servers instead? Does it have any sense?

if the math is 2 servers and 2 replicas, they will practically be clones. it is still true that they serve their own data - each of them has downloaded the segments they are assigned, which in case happens to be exactly same. when serving a query, broker picks segments from all available servers. So broker can pick some segments from server 1 and some from server 2. when a server goes down, the external view is updated, and the status for that segment-server combination becomes OFFLINE. So broker will only pick from the alive server. Replication 2, with 3 servers also makes sense. Say you have 10 segments, with replication 2. If you have 2 servers, then each server gets 20/2 = 10. If you have 3 servers, each server gets 20/3 = 6-7 segments. this reduces load on each server from 1/2 traffic to 1/3rd

@Neha Pawar yes having three servers will reduce load to 1/3 but what about replication=2 in this case? If the segments will be splitted across the servers in an equal/similar number what about the two replication? Does it mean that all the segments will be splitted in two servers instead of three?

For 2 and 2 example, server A will have a copy of segments 1 2 3 4 and server B will also have a copy of 1 2 3 4. At all times. Broker can pick any Segment from any server

@Neha Pawar ok but having two servers, i think that Pinot will not ask the same segments on both servers right? So i suppose a Broker ask segments in a "balanced way", no? I got that the copy of the segments (replication) live on the server togheter with the other segments but i think Pinot ask them accordingly. Neha, i can stay here asking you thousand of questions, haha. I Promise, the last two questions (for now 😄)

if you’re querying through the controller (i.e. Pinot query console), then the code will choose a broker randomly from list. but typically in production environments, users will write their own balancer module which distributes traffic amongst brokers.