https://pinot.apache.org/ logo
#troubleshooting
Title
# troubleshooting
s

sunny

02/25/2022, 12:48 AM
Hi, everyone đŸ™‚ I am trying batch ingestion job. There are 5 server instances(3 is running , 2 is down) in our pinot cluster. When segments assigned, some segments assigned to down server instance. Even when all segment are assigned to down server instances, I cant use the table. Is is normal? or Any other solutions? Thank you đŸ™‚
m

Mayank

02/25/2022, 1:18 AM
What is your replication? If replication = 1, then server down = data unavailability.
Do you know why the servers are down?
s

sunny

02/25/2022, 3:03 AM
@Mayank replication=2. I guess even if 2 server down, segment will assigned to running server instance. but after drop table and then ingest data to the table, same server(down) assigned to the table. And the reason why server instance down I changed server config (id.to.hostname=true) :-) and then restart
m

Mayank

02/25/2022, 3:10 AM
You can always run rebalance when the server comes up to evenly distribute.
c

coco

03/15/2022, 4:03 AM
@Mayank I'm curious about this, too. Is it normal? Why doesn't pinot's assignment strategy(segment and instance) exclude bad instance(pinot-server)?
m

Mayank

03/15/2022, 4:07 AM
Pinot’s segment assignment does exclude bad instance. I was referring to a case when the server went down after assignment.
s

sunny

03/15/2022, 5:27 AM
@Mayank In my case, when the server went down after assignment and then rebalance server, segment again assgined to bad server instance. Is there anything else to check?
m

Mayank

03/15/2022, 5:54 AM
Rebalance is ideal state based, where as new segment assignment is external view based
Rebalance is meant for sizing the cluster up or down, not really for the case you used. For your case you need to have replication to avoid data unavailablity and set alerts in replication going down below certain threshold
If you need to replace a node, then you first untag it, add a new one and then call rebalance