Someone asked me a few questions about Da Vinci (the stateful client option for Venice, which preloads and then queries local state, rather than doing requests across the network). I’ll share the answers below in case anyone else is interested: > (1) For the non-partitioned case, where the dataset can fit all-in-RAM or SSD, what’s a typically size limit you support? For dataset less than 10GB that may seem straightforward I assume with larger dataset that’s 100GB - 1TB, bootstraping the data from network may take time. Your intuition is right, and most of these use cases are <10 GB. We are currently working with a partner team on a roughly 1 TB / node use case, but they are not in prod with this yet, so no concrete experience to share so far. On the Venice servers, most clusters have >1 TB / node of state, and so we do need to rebootstrap this much in certain scenarios. But the servers are a more controlled use case, since they are treated as stateful infrastructure and so there is host stickiness and more control in general over maintenance operations such as OS upgrades. Whereas Da Vinci client applications tend to be treated as stateless (even though they’re not) and so the host assignments are more volatile and bootstrap can come into play more often. In the current architecture of Venice (and by the same token Da Vinci), all writes always come through Kafka, no matter if they’re batch pushes or single row updates. And while we have optimized the shit out of it as best we can, it remains the case that Kafka’s per-message overhead is a bottleneck. To address this, we are currently working on the next-gen of this part of the architecture, which we call blob transfer. We are planning to have several iterations of this, but the first one (which is pretty much code complete and nearing production try out quite soon) is a fairly narrow scope which we call “Da Vinci P2P blob transfer”. As the name implies, Da Vinci instances will serve the raw blobs of the RocksDB databases among peers, thus optionally skipping the Kafka part, then resume subscribing to Kafka for further updates, using the correct checkpoint info which is maintained atomically alongside the data. Eventually, the scope will expand to support servers blob transfer, and also to support alternatives to P2P (e.g. putting the blobs on S3/ABS/GCS or whatever). We don’t have numbers to share yet on the blob transfer vs Kafka performance for full bootstrap, but we expect that to be a good boost.

Thanks Felix for the prompt reply, really appreciate it. And really happy to join this community. Thanks for sharing the dataset size limit for the current applications, and the exciting future improvements. Just to share our experience as well: We actually started the other way around, by supporting server-side bulk transfer first, and are exploring doing something similar on the client-based mode (nothing exist in prod yet, just early explorations). We do see more and more use cases that want near-real-time updates as well.

Glad to have you here! Just to clarify, there isn't really any hard limit per say, the practical limit is only based on whatever reaources are available on the local node... So you currently have a solution that supports batch pushes / read-only datasets, but not streaming writes?

We also support writes, but not like Venice that takes writes from Kafka, we made reads and writes generic just like regular RPCs for an online datastore, and hence it's more heavy lifting (with replication, versioning, etc), there has been some growing adoption on this method and in general it's much more expensive than the batch-push+read-only option.

RPC? Does it mean it’s strongly consistent? Or how is it setup?

Eventually consistent --- we are a little different, it's a generic key-value store today that also store some source-of-truth data, in addition to the derived data use cases (probably 70% of use cases).

Nice, looking forward to it! Please share here when ready.

Definitely some ideas were inspired from Venice, including the Venice hybrid mode