Hi folks, we wanted to change our table names (from camel case to snake case) in Pinot. For this, we supplied existing table configs to the create table api with the changed table name (all other configs remained unchanged), and disabled the old tables. But we observed that the new tables contained data quite old (that wasn’t present in kafka). For example, our kafka retention is 2h but the new table still contained data as old as 6h old! Is there some sort of data migration happening from old segments to new segments?

Seems like you setup the new table to ingest from smallest offset available in Kafka. This will make the new table ingest all events available. And only when retention kicks in and the segment ages out it will be deleted.

Yes, indeed, it was consuming from the smallest offset. But it contained data much older than kafka’s retention period, so I am wondering where this data came from.