Hi Team, We are planning to add a de-duplication flow for our solution. As part of that, we are doing 2 things: 1. Enable UPSERT in realtime table. - Flink has the key set. - Primary key defined in schema and realtime table using it. - UPSERT working fine. 2. For realtime to offline flow via minion, we found that these duplicates were coming in OFFLINE table. - so we tried with mergeType: dedup (earlier it was concat) Now, the realtime to offline flow has stopped working (no data in OFFLINE table, minion is up and running) Queries: -------- 1. Is our dedup flow proper? UPSERT for realtime and mergeType: dedup for realtime to offline flow? 2. Any pointers around why this realtime to offline flow stopped working after adding these configs?

Upsert feature currently works for realtime only tables, afaik

Hi @Mayank, we are using UPSERT (i.e. "primaryKeyColumns": [] defined in schema) for realtime table and its working fine. Are we saying hybrid table flow is not supported here via mergeType: dedup?

Hi @Vibhor Jain I am saying UPSERT does not support hybrid table at the moment, it is being worked on afaik.

mergeType: "dedup" should work there? Although we are doing a full row comparison, that should be ok for deduplication?

However, if let's say an upsert comes for the same primary key, I don't think the code will take care of what is already moved to offline, right?

Hi @Mayank We are not looking for UPSERT across realtime and offline table. UPSERT is limited to realtime table and its working as expected for us. Now, when we move this realtime data to offline, these duplicate records are showing up again (UPSERT keeps all copies). So, we tried mergeType config with value "dedup" to handle duplicates here and we see the realtime to offline flow has suddenly stopped working.

Ok, may be check the dedup config, and also see if errors in minion log

you might also find logs about realtimeToOffline job in controller. First some lines should appear in controller about task scheduling, then you will see corresponding lines in minion logs about task execution

Realtime to offline flow will move records to the offline table (managed separately from the realtime table), and UPSERT cannot be applied to the offline side. Dedup won't help with this setup