A quick question about Flink Pinot connector: Is t...
# pinot-dev
w
A quick question about Flink Pinot connector: Is this only for offline table? Does it support upsert table?
n
@Weixiang Sun Iiuc, flink pinot connector works with upsert tables as well. Here are some things to keep in mind while operating flink connector with upserts - https://github.com/apache/pinot/blob/master/pinot-connectors/pinot-flink-connector/README.md#notes-for-backfilling-upsert-table
w
Thanks @Navina. Is this only for offline table? Do we have any design document about how Pinot Connector works with upsert table? As for our use case, our table is hybrid upsert table. I am wondering if flink pinot connector works for us or not.
n
w
But it does not talk about upsert table
n
I am actually not sure now that you mention "hybrid upsert table". I thought upserts only work with realtime tables 🙂
yes. that is the only design doc there is. Perhaps @Yupeng Fu / @Xiang Fu can answer your questions better
r
hybrid table and upsert cannot be enabled simultaneously unless you explicitly skip the validation check
👋 1
w
hybrid upsert table means upsert realtime table + offline table.
r
correct, and it is not allowed unless when the table is created it explicitly skip validation for upsert config
w
Without flink pinot connector, we can create upsert realtime table + offline table. Are you saying
upsert realtime table + offline table
is not allowed under flink pinot connector?
r
no it is not allowed at all
w
are you talking about the minion job of converting the realtime table to offline table?
r
e.g. if this is simultaneously enabled. query results might not be correct across the time boundary. thus it is user's responsibility to ensure the query results are correct
but back to your original question. • you can use flink connector to ingest into the "offline" part of a hybrid table, regardless of upsert or not. • but if you enabled upsert. it is your responsibility to ensure there's no data overlap issue across the time boundary
does that answer your question ( i felt like i might've misunderstood your intent, so please correct me if i were wrong )
w
In my use case, the data will have the fixed timestamp. The upsert will not create two different rows for the same key across the time boundary.
How does the flink pinot connector deal with the upserts to the old offline segments?
r
upsert is not supported on offline table LOL. that was my point
✔️ 1
e.g. the metadata map is only maintained for realtime table
n
@Weixiang Sun I think the pinot flink connector can be used to "backfill" data in the "offline" part of the hybrid table. But I am not even sure how a hybrid table can exist with an upsert enabled realtime table. what is your use case for upsert , if I may ask? How do you guarantee that you will not create 2 different rows for the same key across time boundary?
w
In our use case, the timestamp for the same key will be fixed. One of similar examples to ours is that when you place an order, the order creation time will be fixed and never be changed. But the order can be updated. The primary key is order id. The customer can make the change to old orders which are inside the offline table.
n
ah I see. you would still to ensure that no order info gets split across the offline/realtime boundaries. I am assuming you have way of doing that? if that is the case, then pinot flink connector can be used to populate the offline table. I don't think we have tried it with upsert enabled realtime table.
r
If upsert tables have backfill support, is the offline table still needed?
y
flink connector is for backfilling upsert table
upsert is realtime only
and flink connector can push the real-time segment directly
👍 2
w
@Yupeng Fu do you mean that flink connector is the replacement for streaming ingestion?
r
The Flink connector appears to create and upload segments to the Pinot Controller according to the connector’s buffer-flush settings. Due to delay associated with buffering, it’s more appropriate for historical records.