https://pinot.apache.org/ logo
Join Slack
Powered by
Hi, what’s the common reason for errorCode 235? [ ...
# troubleshooting
a
Hi, what’s the common reason for errorCode 235? [ { “message”: “ServerSegmentMissing:\n17 segments [table_name__1__22__20220527T1518Z, table_name__1__10__20220516T1300Z, table_name__1__14__20220517T1900Z, missing on server: Server_pinot-server-28.pinot-server-headless.pinot.svc.cluster.local_8098", “errorCode”: 235 } ]
m
Check ideal-state and external-view for the table for these segments. This error message means that the server did not find these segments.
a
I checked the external-view, it said the segment is on another server instance. The table status shows good.
m
Hmm, that means the broker routing table is not updated (that shouldn’t happen). Can you rebuild broker resource in swagger, that should update the routing table.
a
You mean this one?
m
yes
a
Thanks. I’ll have a try.
Copy code
{
  "status": "Broker resource is not rebuilt because ideal state is the same for table: table_test_REALTIME"
}
m
I think there is api on broker to print the routing table, you can check that to see why the broker thinks these segments are on this server
/debug/routingTable/{tableName}
Also, are you sure the IS/EV say these segments are in another sever?
a
yes,
Copy code
,
    "table_test__1__585__20220605T0324Z": {
      "Server_pinot-server-27.pinot-server-headless.pinot.svc.cluster.local_8098": "ONLINE"
    },
    "table_test__1__586__20220605T0354Z": {
      "Server_pinot-server-27.pinot-server-headless.pinot.svc.cluster.local_8098": "CONSUMING"
    },
it’s on server 27. But query shows segment not on server 28.
m
ok, then check the broker routing table
a
There is only /debug/tables/{tableName}
I can’t find /debug/routingTable/{tableName}
m
http://brokerHost:brokerPort/
It is on broker
a
It said this segment is on server 28.
Is there anything I can do to solve this error?
m
Hmm, this is not expected. What operations were done on the cluster? For now, restarting the broker should fix this. But. I haven’t really seen this behavior in the past. cc: @Jackie
a
I think I didn’t do any operation that may cause this problem.😅
edit server tag?
m
Ah, did you rebalance after that?
a
didn’t update any table config to use that new tag, so not rebalance operation.
restart broker didn’t help
m
Please run rebalance
a
rebalanced broker and server
same error there.
m
This does not make sense
Broker routing table is built from EV. If EV is correct I don’t understand why wrong server shows up in routing table
Did you confirm that routing table shows incorrect server
a
Three segments were missing from server 28. After restart broker, rebalance broker and servers, there is only one segment has this error. It’s a consuming status segment.
m
When you did rebalance, did you including consuming segment?
If not, do that.
a
just after you told me to run rebalance. I did check consuming.
image.png
Should reassign instance be checked?
m
yeah try that
a
It seems a new consuming status segment has this issue on server 28.
m
Try 
reset
segment api
a
I tried reset. reset segment success returned. Then I query this table, no error. But run the same query again, same error occurred.
😓
m
After reset segment is the consuming segment CONSUMING or ERROR in EV?
And after reset + rebalance is it on server 27 or 28? And the new error is complaning about the same consuming segment to be on incorrect server?
a
Hi, i just found server-27 and server-28 have the same hostName.
I really don’t know how is that happening.
m
I am confused, if they have 27 and 28 in the name, how is the host name same?
But sounds like that is likely the issue, Your hostnames are messed up, but that is outside of Pinot
a
restarted server 27. I’ll check if there’s still trouble.
I’m confused too.
Hi, it’s ok now. Thank you so much.
m
Ok so restart or sever fixed it?
a
restarted server 27 which has a wrong hostName.
It’s strange. Edit server tag could cause this problem?
m
I’ll need to know the exact events, but seems like it
a
😅
Open in Slack
PreviousNext
Powered by