Hi, We have a use case where we want to setup a hybrid table with upsert support for the realtime table of the hybrid setup. When i try to setup a RealtimeToOfflineTask, I get the following error:

Invalid table config: table_REALTIME with error: RealtimeToOfflineTask doesn't support upsert table!

. Any recommendations on how to go about this setup? Our data is more like this - we will have update use cases for data for current and future quarters while earlier quarters' data does not require upsert support. So considering moving the older quarters' data to the offline table while still supporting update for the previous data. cc: @Mayank

At the moment this is not supported. We do plan to address the limitation in near future though. @Jackie

@Mithun Vigneswar Gunasekaran Trying to understand the use case and requirement better. Do you want to move segments to OFFLINE side to save the cost of upsert metadata? Is the goal to only store the upsert metadata for the current quarter (e.g. past 90 days), and stop tracking the metadata once the time passed?

Yes @Jackie that is precisely the use case.

We are planning to introduce retention to the upsert metadata for this scenario. @Qiaochu Liu will be working on that. Qiaochu do you have a timeline for that?

@Jackie We planned to completed snapshot feature in Q3 and then prioritize the deepstore and TTL changes for this feature.

@Qiaochu Liu @Jackie is this feature already available? Can you share some details on the same?

Hi @Mithun Vigneswar Gunasekaran The implementation of this feature is still in progress. will let you know once it’s ready

Sure @Qiaochu Liu