@User gave a very interesting Kafka Summit APAC talk recently entitled "Real-time Analytics with Upsert Using Apache Kafka and Apache Pinot"

Thanks Yupeng! Amazing work on upsert /cc @User

I’m curious about why upsert isn’t compatible with a few index types (ie StarTree).

That’s because startree index is built with preaggregation

Oh, right. What about Kafka’s log compaction? Would that help at all?

Not really. Think of it this way, there are indexes stored in a certain way (say pre-materialized form for star-tree) that upsert is either not possible or difficult.

I didn't think so. I was justing considering a potential workaround, similar to what I've done with Redshift (which doesn't have upsert).

startree index is more for preaggregation, so the info is lost by the time you have upsert. i.e. you dont know how much preaggregated value shall be discounted

That makes sense. I've only heard of StarTree at a very high level. Planning to review https://docs.pinot.apache.org/basics/indexing/star-tree-index

btw, @User and @User had some cool ideas of supporting star-tree index for upsert as well.