Could we run a minion task to backfill data so we don’t have to worry that the new segment names are identical to existing ones to avoid duplications?

there is a task to move segment from realtime to offline servers: https://docs.pinot.apache.org/basics/components/cluster/minion#realtimetoofflinesegmentstask

yes we are planning to use this task. however, occasionally there is still a need to backfill to 1) repair some cols in existing rows 2) add some missing rows. I wonder what is the best practice here to avoid data duplications and corruptions

for data repair, where is the source of truth? and how much update you foresee

for data repair, where is the source of truth? and how much update you foresee

good question. maybe we have to derive a new col from an existing one. This should happen pretty rarely though

i see, that’s more on schema evolution and data refill part