Apache Pinot

but just a heads up, that i wont have something ready for you immediately. I’ll need a week or so to wrap up some other things i’m working on

I prefer collect. Collect and flush may give user the impression that this thing won’t be persisted until you call flush

I feel this is the impression we should give to the user

Xiang, the actual behavior could depend on implementation

for example, we want the mutableSegmentImpl to implement this

then it can serve right after write is called, right?

true, but still it’s not persisted until you call flush

meaning if user don’t call flush, then the data are not guaranteed to be persisted.

this is also why i want the default behavior of flush to be in sync mode

Hmm I see the difference we are seeing. Data serving vs data persistence

i meant to have more extra buffer for better parallelism, the error handling would be similar to a single buffer

if there is a single buffer, how do we recover?

flink connector will be waiting for success response from the segment build. If it fails, the connector knows the last point of flush, and will try write again. Say last flush was at 100, and now it is 250. But the flush at 250 failed. Flink connector will write from 100.
But if the connector has moved onto writing new records, i.e. called flush at 250 and began writing, and is now at 330 when it received response that the flush at 250 was a failure. Then what does the flink connector do for records 100-250?

btw, i think like what spark connector behaves today, an error during the segment load will fail the entire job

i think similar behavior will be there in flink connector

Alright, if you think the flink connector can handle that, then I have no concerns about doing the 2 burgers design

Do you mind getting on a call with me next week and explaining it from the flink side some more? I've never worked with flink, so I'm probably not seeing what you are

hey, i'm ooo for the week. may take a look this weekend