Hi, everyone :slightly_smiling_face: I am trying ...
# troubleshooting
s
Hi, everyone đŸ™‚ I am trying batch ingestion job. There are 5 server instances(3 is running , 2 is down) in our pinot cluster. When segments assigned, some segments assigned to down server instance. Even when all segment are assigned to down server instances, I cant use the table. Is is normal? or Any other solutions? Thank you đŸ™‚
m
What is your replication? If replication = 1, then server down = data unavailability.
Do you know why the servers are down?
s
@Mayank replication=2. I guess even if 2 server down, segment will assigned to running server instance. but after drop table and then ingest data to the table, same server(down) assigned to the table. And the reason why server instance down I changed server config (id.to.hostname=true) :-) and then restart
m
You can always run rebalance when the server comes up to evenly distribute.
c
@Mayank I'm curious about this, too. Is it normal? Why doesn't pinot's assignment strategy(segment and instance) exclude bad instance(pinot-server)?
m
Pinot’s segment assignment does exclude bad instance. I was referring to a case when the server went down after assignment.
s
@Mayank In my case, when the server went down after assignment and then rebalance server, segment again assgined to bad server instance. Is there anything else to check?
m
Rebalance is ideal state based, where as new segment assignment is external view based
Rebalance is meant for sizing the cluster up or down, not really for the case you used. For your case you need to have replication to avoid data unavailablity and set alerts in replication going down below certain threshold
If you need to replace a node, then you first untag it, add a new one and then call rebalance