A quick question about Flink Pinot connector: Is this only for offline table? Does it support upsert table?

@Weixiang Sun Iiuc, flink pinot connector works with upsert tables as well. Here are some things to keep in mind while operating flink connector with upserts - https://github.com/apache/pinot/blob/master/pinot-connectors/pinot-flink-connector/README.md#notes-for-backfilling-upsert-table

Thanks @Navina. Is this only for offline table? Do we have any design document about how Pinot Connector works with upsert table? As for our use case, our table is hybrid upsert table. I am wondering if flink pinot connector works for us or not.

@Weixiang Sun hmm. here is the design doc - https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177045634

But it does not talk about upsert table

I am actually not sure now that you mention "hybrid upsert table". I thought upserts only work with realtime tables 🙂

hybrid table and upsert cannot be enabled simultaneously unless you explicitly skip the validation check

hybrid upsert table means upsert realtime table + offline table.

correct, and it is not allowed unless when the table is created it explicitly skip validation for upsert config

Without flink pinot connector, we can create upsert realtime table + offline table. Are you saying

upsert realtime table + offline table

is not allowed under flink pinot connector?

no it is not allowed at all

are you talking about the minion job of converting the realtime table to offline table?

e.g. if this is simultaneously enabled. query results might not be correct across the time boundary. thus it is user's responsibility to ensure the query results are correct

In my use case, the data will have the fixed timestamp. The upsert will not create two different rows for the same key across the time boundary.

upsert is not supported on offline table LOL. that was my point

@Weixiang Sun I think the pinot flink connector can be used to "backfill" data in the "offline" part of the hybrid table. But I am not even sure how a hybrid table can exist with an upsert enabled realtime table. what is your use case for upsert , if I may ask? How do you guarantee that you will not create 2 different rows for the same key across time boundary?

In our use case, the timestamp for the same key will be fixed. One of similar examples to ours is that when you place an order, the order creation time will be fixed and never be changed. But the order can be updated. The primary key is order id. The customer can make the change to old orders which are inside the offline table.

ah I see. you would still to ensure that no order info gets split across the offline/realtime boundaries. I am assuming you have way of doing that? if that is the case, then pinot flink connector can be used to populate the offline table. I don't think we have tried it with upsert enabled realtime table.

If upsert tables have backfill support, is the offline table still needed?

flink connector is for backfilling upsert table

@Yupeng Fu do you mean that flink connector is the replacement for streaming ingestion?

The Flink connector appears to create and upload segments to the Pinot Controller according to the connector’s buffer-flush settings. Due to delay associated with buffering, it’s more appropriate for historical records.