Hi Pinot team, we are trying to deploy Pinot in AW...
# generala
Abhay Rawat
05/23/2022, 10:16 AMHi Pinot team, we are trying to deploy Pinot in AWS ECS.
One issue that we are facing is if one of the server instance goes down and comes back up with another instance id (in our case IP address), we lose the segments registered with that server. Is there a requirement that controller/broker/server instances have a stable identifier?
Abhay Rawat
05/23/2022, 10:24 AMjust to explain a bit more from my colleague's findings
The cluster is started with three servers. By default the hostname of the ECS task contains the IP address, and the hostname in Helix is the IP address.
servers = 1.1.1.1, 1.1.1.2, 1.1.1.3
These servers are registered with Helix (cluster manager) using the hostname or theconfiguration property. Segments are then registered using the same server identifier.pinot.server.instance.id
Now we do a deployment (or tasks are killed/started in response to ECS autoscaling or a rolling cluster restart) and the servers come up with a different hostname.
servers = 1.2.1.1, 1.2.1.2, 1.2.1.3
Since the segments in Pinot are registered to the original servers they do not get downloaded when the new servers come online.
m
Mayank
05/23/2022, 1:22 PM
From pinot server config you need to set
<http://t.set.instance.id.to|t.set.instance.id.to>.hostname=trueMayank
05/23/2022, 1:24 PM
In case of scaling up/down (by adding/removing servers), you need to run rebalance: https://docs.pinot.apache.org/operators/operating-pinot/rebalance/rebalance-servers
a
Abhay Rawat
05/23/2022, 1:54 PMthanks Mayank. so just for my understanding
1. Pinot servers should not be configured for autoscaling
2. For existing data Upscaling is fine. down scaling should be followed by rebalance
3. Service orchestration config (in our case ecs) must be aware of the different container instances of the service running, and if any goes down (in our case quite a few will as we have spot instances in the underlying cluster) must bring those up with the known identifier of that service instance (hostname)
Abhay Rawat
05/23/2022, 2:00 PMgiven that we have deep storage configured
m
Mayank
05/23/2022, 2:30 PM
Both up and down scaling require rebalance to rebalance the data on the new set of servers
thankyou 1