Hi guys, good afternoon, I’m gonna need to delete a consuming segment because it got stuck and is no longer streaming data from kafka, what would be the steps to create it again?

I think you might be able to fix this problem by resetting this segment - That would force it offline and then into the consuming state, which should then create a new thread to consume from kafka

Thanks, I´m gonna try it now!

Could you confirm if it worked for your issue @Alexander Vivas

I’m afraid that’s no longer an option, someone already deleted the segment by the time I tried to reset, is there anyway (even if it’s dirty) to force pinot to create a new segment?

I think there is a periodic background thread that may do so, we probably need a rest api to invoke it (if it is not already there). In the meanwhile server restart should do it. Cc: @Neha Pawar

I’ll restart the server then, thanks

Segment deletion cannot be recovered by the restart or background task. You'll need some very manual steps now to restore it

Yes, I just tried to recreate the segment in zookeeper using an existing consuming segment from another realtime table and this is what I came up with in this zookeeper’s path: PROPERTYSTORE/SEGMENTS/dpt_video_event_captured_v2_REALTIME

{
 "id": "dpt_video_event_captured_v2__0__316__20220108T2205Z",
 "simpleFields": {
  "segment.crc": "-1",
  "segment.creation.time": "1643277577342",
  "segment.end.time": "-1",
  "segment.flush.threshold.size": "14000000",
  "segment.flush.threshold.time": null,
  "segment.index.version": null,
  "segment.name": "dpt_video_event_captured_v2__0__316__20220108T2205Z",
  "segment.realtime.download.url": null,
  "segment.realtime.endOffset": "9223372036854775807",
  "segment.realtime.numReplicas": "2",
  "segment.realtime.startOffset": "1549438226",
  "segment.realtime.status": "OFFLINE",
  "segment.start.time": "-1",
  "segment.table.name": "dpt_video_event_captured_v2",
  "segment.time.unit": "null",
  "segment.total.docs": "-1",
  "segment.type": "REALTIME"
 },
 "mapFields": {},
 "listFields": {}
}

Zk metadata should be IN PROGRESS stauts and ideal state should be CONSUMING

Thanks, I put that metadata in zookeeper and the segment appeared in state ERROR but then I used @Mark Needham suggestion and then it went back into CONSUMING

Btw, what exactly happened to the segment when you say it got stuck?

We saw a delay in our analytics dashboard that depends on pinot getting behind of schedule and when we had a look at the logs we couldn’t see anything, the segment kept on consuming state even though no new data was getting streamed into the realtime table, no new files in the deepstore since 10 days ago which is strange because pinot creates one every day

deleting and recreating fixed it?

Yes, the messages started to stream into Pinot again