hi all, first time here 🙂 i have a question about segments in a realtime table. i want to repopulate our tables, so i used the rest API to delete all segments in our table, and then pushed a bunch of stuff into our kafka topic. after deleting the segments, no new data is showing up and no new segments are being created. is there something else i need to do for new segments to be created?

Did you use the recently added truncate API to remove segments?

no, i don't think we're on a version that has them

So, you probably tried to delete the consuming segment. i am unsure how that works. In general, if you want to remove all data from a realtime table, then just remove the table and recreate it. You will have to wait a bit before recreating the table with the same name. The deletion of segments happen in the background, and Helix may take some time to reflect that in all servers.

ah interesting. yes, i deleted all segments, so it would have included the consuming segment