Hi Team, i am pushing realtime to offline data and...
# general
v
Hi Team, i am pushing realtime to offline data and trying to implement upsert but its returning error saying that we can not use usersert with realtime to offline table.
Copy code
"error": "RealtimeToOfflineTask doesn't support upsert table!"
can't we use upsert with the reltime to offline flow?
s
What was the blog post that you tried to follow?
I think that the upsert feature in today requires you to use realtime table only
v
https://dev.startree.ai/docs/pinot/recipes/upserts-real-time-offline-job but here its saying that its will work for realtime to offline as well
s
@Rong R It looks that the validation is added from https://github.com/apache/pinot/pull/7921 and it would make the above documentation invalid. Can you double check on this?
@vishal can you add a bit more context on your use case on why u would like to move data to offline for an upsert table?
v
@Seunghyun I am giving you one simple use case here. I have more than 100 million customer information which contains their address and phone no. which I want to store in the RealtimeTooffline tables. Now, the customer has updated either the address or phone no. We want to update that information in offline tables.
s
@vishal After checking the above PR, it actually adds a way to bypass the validation. If you really want to try out realtime to offline + upsert, you can create the table by enabling the following query parameter
Copy code
validationTypesToSkip=ALL
Copy code
The main thing to keep in mind when combining these features is that upsert functionality only applies to real-time tables.

As soon as those segments are moved to an offline table, the upsert logic is no longer applied at query time. We will need to backfill the offline segments created by the real-time to offline job to achieve upsert-like queries.
The above is the caveat from the documentation