Hi 👋, i retrieved a few days ago a realtime table (pinot 9.0.2 with a kafka stream config) in an unhealty ingestion status with this error :

Unable to get consuming segments info from all the servers. Reason: null

I've some segments that persist in OFFLINE even if a reset or reload them. The segment metadatas seems to be stuck with the status "IN_PROGRESS". Does anyone have any ideas on how to handle this situation ?

The consuming segment can get marked offline, if there's some stream error. There's a background task that fixes it once an hour. Could you check in the ideal state, if the status is offline there too? If yes, check the servers why it marked itself offline.

yes i guess we are in this case https://docs.pinot.apache.org/basics/getting-started/frequent-questions/ingestion-faq and the idealstate is offline too.

you’ll need to look into the pinot-server and pinot-controller logs. The pinot-server logs will show what went down with the segment which made it OFFLINE, and the pinot-controller will show the RealtimeSegmentValidationManager trying to fix it and create a new CONSUMING segment. The old segment will forever remain OFFLINE and IN_PROGRESS, and that’s expected, as long as a new CONSUMING segment gets created

pinot controller & server have been restarted few days ago (k8s upgrade). Segments was created before, so in pinot logs i've just found the server is skipping reload because segment metadata is null. If segments can remain OFFLINE, why the ingestion status stills unhealthy ?