> (3) For the partitioned-cases, is that a popular use case, the sharding part is a bit clear to me from reading the blog, and I imagine it’ll require tight integration to make it work --- the sharding on the data preparation & push side, may need to agree with the sharding on the data access layer? The partitioned case is definitely a power user scenario, and in that sense it is “not that popular”. I would say that it only makes sense when used as part of a “framework” or “environment” which already has a notion of partitioning. IOW, it doesn’t make much sense to use partitioned Da Vinci just like that, out of the box, because you wouldn’t know what partition to subscribe to on each host… At LI, we’ve had partitioned Da Vinci integrated in our search stack as well as our stream processing stack, both of which have built-in notions of partitioning. In those cases, we configure the relevant Venice stores with a “custom partitioner”, which results in the Venice data being “co-partitioned” with the other system (e.g. search or stream processing), and therefore the other system can subscribe to the correct shard in each of its own hosts (or tasks, or whatever). As an aside, this is where the name Da Vinci really shines… did you know that Leonardo Da Vinci left many of his works, including his masterpieces, unfinished? For example, he kept tweaking the Mona Lisa up until his death, many years after having started it. Likewise, while Venice is a full-fledged system that one can use directly without much scaffolding around it, Da Vinci is more of a building block, which you can integrate into other systems.

Thanks for confirming and sharing your integration story at LinkedIn, makes sense. This is exactly the set of use cases we are exploring as well, where the application has its own notion of partitioning, and may even integrate its own in-memory cache in front of the backend datastore to reduce load and enhance performance.