ls there any way to backfill segments in a realtime only table?

Yes I think since 0.12 you can backfill real-time segments. I’ve been meaning to write this up, so this is the push to do so! I think the tricky thing will be making sure that the segment that you backfill has exactly the same records as the one being replaced. With offline that’s easier as you know exactly what the initial source was for a segment whereas for real-time Pinot is deciding when it gets committed

@Mark Needham I wonder if there could be another potential configuration option for real-time segment rollover? Currently one can configure a heuristic such as number of records, size of segment in bytes, number of hours elapsed. Adding another option like a time of day would mean that a given segment would only hold data that was ingested during a deterministic window, which could make it easier to manage real-time segment backfill (at the expense of occasionally creating segments which are smaller than the ideal). Thinking out loud 😊

For most of my use cases, I don't think that would be a problem. Would I use the SegmentCreationAndTarPush job for this?

Sorry I didn’t get back to you. I’ve been trying to adapt a guide I wrote about backfilling offline segments (https://dev.startree.ai/docs/pinot/recipes/backfill) and I’ve got stuck on a couple of things

awesome, thanks!