Good morning @Felix GV @Koorous Vargha @Zac Policzer, I was looking at the realtime-ness and hybrid stores setup for Venice. It all made sense at first sight (new batch version is created, RT events are replayed up certain rewind time), but then the fundamental question is: how to handle the batch lag problem in practice. Consider the following example when batch and RT write the same field, where it seems like there is no clean resolution:

1:00pm  VPJ1 pushes purchase_count = 3
1:10pm  RT update: purchase_count = 4
1:15pm  RT update: purchase_count = 5
1:30pm  RT update: purchase_count = 6
1:35pm  VPJ2 pushes, but batch data lags by 15min: purchase_count = 5

As VPJ2's new version is being built, RT events from 110 130 are also being applied to it. But DCR resolves conflicts by timestamp: • Batch sets purchase_count = 5 (with timestamp 1:35pm) • RT event from 1:30pm (purchase_count = 6) has an earlier timestamp • Timestamp-based resolution: batch wins. Value regresses to 5. So far, my understanding is that achieving good consistency with Batch + RT updates is possible only if RT updates separate set of fields through write-compute. Am I missing something? I put up a doc on Venice Hybrid store considerations as a reference. cc @Hubert Puszklewicz @Amre Shakim

So a qq, what does good consistency mean in this case? For version 2, you just don't want to serve a read such that it shows up as purchase_count =5 if you already consumed up to that point on version 1?

So a qq, what does good consistency mean in this case? For version 2, you just don't want to serve a read such that it shows up as purchase_count =5 if you already consumed up to that point on version 1?

Good consistency here means: if a key's value was updated to 6 by RT at 1:30pm, and a batch push happens at 1:35pm with stale data (5), the store should not regress to 5. The most recent correct value should win regardless of which pipeline wrote it.

Ah, so DCR w.r.t to batch writes is such that I believe RT writes always win? @Koorous Vargha could keep me honest here. If it didn't then that would make rewind's pretty problematic since we'd just be applying a bunch of dropped writes.

Yes

Just to make sure I'm understanding correctly, if the timestamp of the RT event > batch event you want the RT event to win DCR?

Just to make sure I'm understanding correctly, if the timestamp of the RT event > batch event you want the RT event to win DCR?

no, it's opposite - RT even can have more recent data, even though it's timestamp is 5-10min earlier than Batch kafka event.

Oh I see

I've found this: BatchConflictResolutionPolicy has THREE options:

BATCH_WRITE_LOSES(0)      // RT always wins over batch
  USE_START_OF_PUSH_TIMESTAMP(1)  // Batch uses SOP timestamp, RT wins if its timestamp is higher
  USE_PER_RECORD_LOGICAL_TIMESTAMP(2)  // Full per-record timestamp comparison

Let me check what is the default policy we have internally

These are partial updates from RT?

Looks like it's clearing all the confusion @Koorous Vargha if the RT events always win.

Confusion being cleared is good, but uh, does it also do what you want it to do? 😅

Still, we'd have to be careful with how we configure rewind time, but I guess for a case when nothing goes wrong, RT winning all the time does the right job. When smth goes wrong (and they always do), however, it could lead to a situation like: if an RT event is missed (Flink crash, Kafka hiccup, producer bug) and the rewind window doesn't cover it, the stale RT value from before the gap gets replayed and wins over the newer batch value. You're stuck with the stale RT value until either a new RT event corrects it or the next batch push happens with a rewind window that doesn't replay that stale event.

When Batch and RT update disjoint sets of fields - then things are clean, no interference between the two.

This use case is what Write Compute's Partial Update is meant to solve. I would definitely recommend using that operation in the RT pipeline.

The confusion happens for the fields that are updated from both - and so far it looks like it's not a good thing to do at all.

In theory that should be fine as long as the timestamps are properly set (which admittedly is easier said than done). Note that by default the RT pipeline will use the producer process’ clock as the TS, but this can be overridden with a logical time of your own if you have access to one… again, easier said than done, but if the logical TS is wired in correctly, then the system should work fine even in the case of processing lag, or duplicate data, etc, which are always possible scenarios in stream processing…

Thanks Felix.

which admittedly is easier said than done

yeah, it can be very nuanced, but I assume if updates are possible on same fields from an RT job, than whatever is pushed from the batch should not be too far from the source, making it feasible to populate a proper logical timestamp. Another place where things can get interesting is if we have incremental pushes + RT updates, because both are going to the _rt topic. So, for example, setup like "do full push daily, incremental push hourly, and support RT updates for freshness in between incremental pushes" can be tricky to do. In that sense, having one of "full push daily + incremental hourly" or "full push daily and RT updates through Flink" may be cleaner.

In theory Venice supports ingesting all of these inputs into the same store. But the inc push part might not be as feature-complete regarding logical timestamp support, etc.

delete from inc push was missing maybe

that's interesting. Feel like it should be just about the writing tombstone rows with NULL value. If there are any limitations to the incremental logic, would be great to know about those (I don't see anything in the documentation https://venicedb.org/user-guide/write-apis/batch-push/?h=incremental#incremental-push)

I 'think' @Jialin Liu would know the current state.