Hi Everyone, I am trying deduplication in apache p...
# troubleshootinga
Arpita Bajpai
09/27/2021, 6:44 AMHi Everyone, I am trying deduplication in apache pinot 0.8.0. I have enabled upsert in my REALTIME table and it is working as expected. But when
data moves from realtime to offline table the duplicate data appears in offline table. Is there a way by which I can have deduplication enabled in REALTIMEtoOFFLINE flow, so that my offline table contains only distinct values.
Can anyone help me with the same?
👍 1
k
Kishore G
09/27/2021, 6:51 AM
upsert is not supported with hybrid table (realtime and offline mode). It's in the works @Yupeng Fu can provide more info.
Any reason you cannot use it in real-time only mode?
a
Arpita Bajpai
09/27/2021, 6:55 AMwe need to backfill data that is the reason
y
Yupeng Fu
09/27/2021, 3:23 PMfor backfill you can still write the data to kafka
n
Neha Pawar
09/27/2021, 3:40 PM
For rows that should dedup, will the rows have identical values across all columns(all metrics, all dimensions and time column as well) ? If yes, realtime to offline job has a config for dedup