Hi all! Is there a way to maintain primary/composite key in an offline only table? I know upsert/dedup only work for realtime tables. Or is there something planned along these lines?

No plans for upserts in offline for now. What’s the use case?

We have several data feeds where for a given composite key we receive daily/weekly metrics. The most common query for such data is the latest value of the metric given the composite key. For now, we have added an extra timestamp column and using

LASTWITHTIME(dataColumn, timeColumn, dataType)

function

SELECT pk1, pk2, LASTWITHTIME(m1, processing_time, ‚‘INT‘) , LASTWITHTIME(m2, processing_time, ‚‘INT‘) 
FROM myTable
GROUP BY pk1, pk2

where pk1 and pk2 together makes the unique composite key and m1 and m2 are metric columns It works fine but 1) it makes the query verbose especially if there are a lot of metric columns. 2) I am not sure how the performance will be for LASTWITHTIME function as the query scale. 3) I am not sure if the metrics returned would be really really latest as the query scale. We would like to avoid inconsistent results here. It would have been really nice to have upset functionality where we can just make a select query with composite key and get the latest values of metrics.

I have the same usecase! I was planning to use the LASTWITHTIME function but I'm also concerned about performance (i have around 10 billion rows)

I also found another way to do it with ROW_NUMBER() but unfortunately it only works with multistage engine. Don’t know which one would scale better,ROW_NUMBER or LASTWITHTIME.