Curious though, what kind of transformations you want to do, and is that not something that can be done in the table directly (using derived columns)?

We used to populate a topic based on ksql stream for video analytics, but since a few months ago we detected an issue with some of the metadata we were ingesting into Pinot which made the queries unreliable anymore.

So we did some changes and there is a new ksql stream capable of collecting data from different topics, aggregates them and produces to a new topic which is now our source of truth

For this new topic we also created a realtime table in Pinot and it is actively consuming the data from there

What we want now is to see if it is possible to do some backfilling

Because the table schema changed quite a bit, we removed some columns that we considered useless and added a few more based on the changes we did in ksql, that is why I was exploring the option of downloading the segment, remove the useless columns and add the new ones based on a script that retrieves them from Redis and then put back the segment into the deepstore with a different name, then the plan was to modify the segments metadata to include the backfilled segments in the servers so that a restart could make the data available again to query

To be honest, I managed to download the segment and decompress it but I found quite difficult to do such changes, so that's why I asked here, because decoding the data wasn't straightforward

@Mayank, what did you mean by derived columns?

Oh, got you, no, well, that is true for one of the new columns, because the other two are taken from a different place outside the table schema