Hi Team, As part of handling duplicates in our hybrid table, we thought of using "mergeType": "dedup" for moving data from realtime to offline table. The problem we are facing is, one of our column is storing encrypted value and even for duplicate rows, this value is changing everytime. Since "dedup" works on entire row, its not removing the duplicates. Is there a way to perform "dedup" on a subset of columns for moving data to offline table via minion?

@Vibhor Jain Currently that is not supported. If the value keeps changing, we won't known which value to keep during the

dedup

. Is it possible to model the use case as

rollup

where we can merge different values into one?

Hi @Jackie, sorry for running 2 threads for this issue. @Mayank is helping us on the other thread. We'll check if rollup could help us here. Thanks.