https://pinot.apache.org/ logo
Join Slack
Powered by
Hi, we noticed that the external view and ideal st...
# troubleshooting
e
Hi, we noticed that the external view and ideal state match eachother but do not match the server to segment mapping. Is there a way to get them to match? We rebalanced the table and rebuilt the helix tags, still no change. i.e. the external view and ideal state report a segment living on 3 servers but server -> segment map shows the segments living on 3 different servers.
we are using pinot-0.8.0
x
@Jackie may have more insight of rebalance tooling
e
hey:)
x
also cc: @Mayank
e
sure, sounds good!
x
😛
e
Do you think it's dangerous to edit the ideal state and external view to point to the same servers that the segments actually exist on? First we are trying another rebalance to see if that changes anything.
j
I don't fully follow the problem. How do you find the server -> segment map?
e
Using the swagger ui:
j
Which api did you use?
e
/segments/{tableName}/servers
and for ideal state and external view I just looked in zk
we just did rebalance and it did not change the ideal state, external view or segment mapping (it was 2nd one)
is it safe to do a rebalance in bootstrap mode for a live table?
or should we edit zookeeper external view and ideal state to match the segment mapping?
j
This API reads the map from the ideal state. No idea how it returns different result
Can you please double check if you are checking the same table (also the type suffix)?
e
Yep, it's an offline only table
j
If possible, can you paste the ideal state and the server to segment map?
e
Sure:
foo.txt
foo.txt
and in ideal state and external view from zk:
Copy code
"tablexxx_2021-10-23_2021-10-23_4" : {
      "Server_pinot-xxx-server-zonal-25.pinot-xxx-server-headless.pinot.svc.cluster.local_8098" : "ONLINE",
      "Server_pinot-xxx-server-zonal-26.pinot-xxx-server-headless.pinot.svc.cluster.local_8098" : "ONLINE",
      "Server_pinot-xxx-server-zonal-27.pinot-xxx-server-headless.pinot.svc.cluster.local_8098" : "ONLINE"
    },
We are only checking that segment right now, to make it easier
it shows 3 different servers than the actual segment map
the second paste is ideal state, which also matches external view
had to substitute table names w xxx, but otherwise it's the same:)
Should rebalancing in bootstrap mode fix this?
Or would we have to modify zookeeper ideal state and external view to match segment mapping?
j
That should not be the issue
This rest API is reading the table ideal state and reverse the map, no idea how it can return different value
e
What would be the issue? The symptom is that queries return different results depending on which server is returned
We used the 
$segmentName
and 
$hostName
to verify
then we started checking idea state, external view and server to segment map
j
But this is impossible.. Both of them reading from the same ideal state, and return different result...
e
Here is an example, it happens reproducibly:
So now we see that some segments are not replicated, even after a reload segment call and server restart.
Here is some info:
This is the metadata for 1 segment on all 3 replicas. Notice the 2nd replica has a different crc and timestamp (creation.meta):
Copy code
server-3 Nov  2 02:54
0000 0000 9182 6888 0000 7c01 91de bebc
segment.total.docs = 201078
segment.start.time = 1635466658
segment.end.time = 1635548781

server-33  16 Oct 30
0000 0000 2ed2 6f09 0000 7c01 c3cf 3a79
segment.total.docs = 252241
segment.start.time = 1635470351
segment.end.time = 1635551011

server-34 16 Nov  2 02:54
0000 0000 9182 6888 0000 7c01 91de bebc
segment.total.docs = 201078
segment.start.time = 1635466658
segment.end.time = 1635548781
@Jackie @Xiang Fu @Mayank
@Mingfeng Tan
We are writing a script to detect this but it is cumbersome: go to every server, get the creation.meta for every segment and compare. Is there an easier way to do this or some way to avoid this happening?
We have code to use the segment replacement api, would that help avoid it?
x
can you try to call segment refresh api for that particular segment and see if that fix your issue
👍 1
e
Nice, retrying it now, should only take a minute...
j
Can you please check if the second server is ever disconnected from the cluster?
e
We tried reloading that segment before, do you mean
Copy code
/segments/{tableName}/{segmentName}/reload
Reload a segment
?
We tried that
what about the reset segment api, would that do it?
j
@Xiang Fu Do we have a refresh api? Or are you referring to the reload api?
e
reload api
we are using pinot-0.8.0 release
no newer commits
is that a newer api in pinot-0.9.0-SNAPSHOT?
j
Yes, reload api in the latest master can fix it, but not available in 0.8.0 yet
e
ah ok
j
reset should work, but you might lose the segment during the reset
👍 1
e
and is there an api to detect it? it seems like a few tables have just a handful of segments in that state
j
i.e. temporary inconsistency
e
this one seems like it was in that state for 2 days. Server restart and reload didn't work. We even re uploaded the segment and that didn't work either
the server with the older crc/timestamp seems to ignore the update
j
Wait.. Restart didn't work?
e
No, but a delete and restart would work. The issue is that it is cumbersome to detect with 100s of tables that have 10-100k's of segments
So we found a few, but we don't know the extent of the issue
j
Restart itself should work though
Did you delete the local segment on server then restart?
e
First we tried just restart
it seemed not to fix it
we have another segment that we didn't delete, it's been ~5 mins since the restart, maybe it fixed it. Checking...
j
That is definitely unexpected
e
yep:)
would the segment replacement api help with this?
j
No, this is nothing about how the segment is uploaded, but why the server didn't get the message to refresh
e
ah ok
j
Actually, could it be related to ZK inconsistency?
e
btw - the restart fixed it, took atleast 10 mins after restart
so that's good news:)
we didn't see any errors in the controller logs
would missing the zk message have showed up in server logs maybe?
j
If it is ZK inconsistency, that can explain the behavior
e
I can do those echo <cmd> | nc localhost:2181 for zk hosts to see if I find anything
is that how you detect zk inconsistency?
we are looking at zk cpu spikes, maybe moving to larger nodes will help
j
Sorry I don't know how to detect zk inconsistency, but still remember seeing different result from different zk servers last time
e
yep, that self healed after some time also
j
For the server with different segment, can you please check its log and see if you can find this line:
Copy code
Segment: <segmentName> of table <tableName> has crc: <crc> same as before, loading
e
sure, thanks! doing that now
j
Or
Copy code
Segment metadata same as before, loading <segmentName> of table <tableName> (crc <crc>) from disk
👍 1
I think this is the log for 
0.8.0
, we refactored it recently
e
thanks - 1 sec, checking
I don't see anything like that since today. It might be an info level message? We had to use warn log level, due to the amount of logs
But I do see that zookeeper is maxed out on cpu during restarts. I think we will use larger nodes for the zookeeper cluster
So you think the root cause of that issue is zookeeper inconsistency? Restarting definitely did fix it:)
We didn't see that, but will try scaling up the zookeeper nodes
we saw some zookeeper nodes spiking at 100% cpu - could that affect server messages not being sent?
j
@Elon Yes that is a info level log
e
So we do see zk timeouts, we're looking at scaling up zookeeper to see if that helps.
j
If zk node lags, the server will see inconsistent views, where the segment ZK metadata might be the old one
e
zk timeouts in server logs
j
Then the server finds the crc are the same, and skip downloading a new copy
👍 1
e
we lowered the tick time to 1.5seconds and the zk timeout is 20 * tick time so 30 seconds. Should we raise the zookeeper timeout? Do you have any recommendation for that?
j
The main issue is zk consistency, where different nodes get different view from zk
e
yep, and if timeout is exceeded on a server (which we see) - that could indicate lagging view? Similar to what happened when we zoomed?
j
I feel 30 seconds is long enough. I'm not a zk expert, so we might need to do some research on when zk can lag and how to prevent that
👍 1
e
On the pinot side the zk timeout is 60 seconds, on zk it's 30 seconds
should those match?
i.e. `
Copy code
HelixZkClient.DEFAULT_CONNECTION_TIMEOUT
default is 60 seconds
j
Based on my very limited understanding, these 2 should be independent
e
ok, I think I have an even more limited understanding of zk:)
j
Yeah, we definitely need to do some research 😅
😀 2
e
We will keep updating with our findings, hope it helps. thanks for all the advice!
4 Views
Open in Slack
PreviousNext
Powered by