Any guidance on how to fix a missing segment? Tabl...
# troubleshooting
p
Any guidance on how to fix a missing segment? Table shows up as bad debug endpoint gives this. See this error in table debug "Did not get any response from servers for segment:" Data is coming realtime from Kafka; so assume there must be a way to reprocess this segment.
r
upon a quick look - the consuming segment info coming from servers seem null or empty for some server. can you provide more info? 1. how many servers are you configured for this table? (also check tenant config) 2. are these servers all consuming kafka or some of them are reserved for only serving out completed segments (sharing table config will help) 3. when you said "missing" is it missing from the newest or missing inbetween some older time but newest semgents are consuming fine?
p
1. six servers, default tenants 2. all consuming from kafka 3. looks to be missing in-between old and latest data
m
Can you check the external view in ZK browser to see if a server is offline? If so, you can check server log for any errors
p
Found that this bad segment is under consuming segments endpoint but it has no server assigned also some other segments also have no consumer assigned. Queries are returning partial results.
Am trying the segment reset endpoint; referenced from an issue found on github
m
Does it appear in the ideal state?
p
yes these segments appear in the ideal state
did notice that table debug says 5 servers where we actually have 6
and about 1/6 th of the partitions are missing a server in the consuming segments end point
m
Can you try the table debug endpoint from swagger? Seems like a server may be in bad state.