Hello Team,
We have both a real-time table and an offline table, but the real-time table contains duplicate records. Since we are not in favor of using upsert, we are considering an approach where only one row per record (eliminating duplicates) is stored in OFFLINE TABLE during data movement.
Is it possible for a Minion job to achieve this via segment replacement? If so, what techniques or configurations (e.g., MergeRollupTask, custom Minion job) would be required to handle deduplication at the segment level?
is it risky objective?