🧵 Venice Writes for non-JVM languages Hi team, Sooner or later, we'll need to support writes into Venice outside of VPJ / Flink. We're a Python shop, and while Venice routers provide decent performance for non-JVM languages through HTTP endpoints, supporting ad-hoc writes outside of VPJ/FLink is not trivial, yet it's an obvious ask for Venice which covers the functionality of KV stores. We've evaluated the following options: 1. Python-native Producer - too complex (KME envelope serialization, the DIV state machine (segment numbers, sequence numbers, SOS control messages), partitioner byte-compatibility, schema fetching). Risks of divergence from Java implementation 2. New component -

venice-writer-router

, alongside

venice-router

,

venice-controller

, etc- that will be a Netty HTTP server on top of the

venice-producer

client • pros: ◦ not that hard to implement - we can reuse a bunch of code from router ◦ doesn't have to handle complex use-cases like

write compute

right away - we can start with simple PUT/DELETE for the entire key(s) 3. proxy solution, shipped as a DaemonSet • can be extended further to support fast client and Da-Vinci client for reads. It's very likely we'll explore faster clients down the road. Questions to the community: • I heard from @Ankit that for their C++ service, they used go-proxy (overseeing the Java Da-vinci and Producer), and the writes and reads went through this proxy. Curious if there is any chance this part could be open-sourced? It should be generic enough, doesn't even have to compile, we can pick it up and make it work. Even design docs/proposal would be helpful. • how common is the RT-only pattern for Venice, where writes happen outside of streaming/batch path? cc @Koorous Vargha and @Zac Policzer would appreciate your input and recommendations on the direction.

If @Ankit remembers the name of the MP that was used for that then @Koorous Vargha could probably ping the team that owns it and asks

slight correction. our C++ "service" used the Java client under the hood

I think DaemonSet with the OnlineProducer would be the best approach here.

how common is the RT-only pattern for Venice, where writes happen outside of streaming/batch path?

It's pretty common, but our recommendation to our customers now is to still have a batch job to push the full dataset in the event of data loss.

The one in go(haproxy) was supposed to be one. not the Java client

haproxy looks to be archived...

basically ATS -> QCS-> Davinci

ok, so ATS (C++) was colocated with QCS (Java) and avoided network hops?

I think #2 would be a hard sell to the team. The common pattern in LinkedIn is a sidecar running the JVM process. I think working towards this would be best since it can be extended to support DaVinci and Fast client as you mentioned.

My 2 cents (not important at all so feel free to disregard): Having an online write endpoint that is easy to use (e.g., built with HTTP and/or gRPC) makes sense. Whether that endpoint is served by the standard router as just one more endpoint there, or run inside a sidecar process, that seems like a minor (almost cosmetic) deployment topology detail. Tradeoffs being that the sidecar skips a network hop, but incurs the overhead (both performance and operability wise) of the sidecar itself, and the reverse for the router. Which is better is likely pretty org-dependent. But as previously mentioned, it can be pretty much the same impl under the hood so it’s just the packaging around it that differs. More importantly than the above (IMHO)… is the fact that the Venice write path is async by design. That needs to be made as clear as possible as part of the API design in order to avoid tech debt pile up from users (and let’s face it, LLMs) who misunderstand or completely skip reading the docs. Or else, if async writes are unacceptable, then you can build sync writes, but that’s a fair bit of scope creep, and you will end up with either: high latency while maintaining Venice write throughput capability, or low latency at the cost of degraded throughput.

Thanks for chiming in @Felix GV. I guess my hesitancy to move forward with the router approach is that the router is on deprecation path, and I don't think I can recommend building more on top of it as the team wants to remove router entirely in the future.

Thanks for chiming in Felix!

More importantly than the above (IMHO)… is the fact that the Venice write path is async by design. That needs to be made as clear as possible as part of the API design in order to avoid tech debt pile up from users (and let’s face it, LLMs) who misunderstand or completely skip reading the docs

Yes, that's a great point. We write through kafka. so there are no real "PUT/DELETE", more like "submit". By "if async writes are unacceptable, then you can build sync writes", do you mean writing to Servers directly?

just to clarify, extending existing router was not an option - it would bring unnecessary and useless complexity for JVM-based client. The router proposed was specifically for write path only.

So it would be a separate deployment that operates as an API service for writes? I guess I'm confused on this part

not that hard to implement - we can reuse a bunch of code from router

We write through kafka. so there are no real "PUT/DELETE", more like "submit". By "if async writes are unacceptable, then you can build sync writes", do you mean writing to Servers directly?

Well in order to make it truly sync the server needs to get involved, yes. There are many different design variants, but essentially, it boils down to waiting on an ack from 1, some or all server replicas before returning the write API call as successful.

Sounds like read after write consistency?

@Koorous Vargha yes, #2 is a separate component. By reuse - I meant that we already have a good example of HTTP-endpoint based router (venice-router), so we could take its skeleton and stitch it to OnlineProducer. Sorry for the confusion

@Jialin Liu @Sushant Mane Do you mind reading over this thread? I'm curious what your take is on #2 here.

but essentially, it boils down to waiting on an ack from 1, some or all server replicas before returning the write API call as successful.

turning it into Cassandra essentially 😄

The variants could include: 1. Let the client open a long-lived connection to the servers (could be the same one that is used for reads by the FC), and have the server "push" its committed offset periodically back to all clients that are connected in such way. Then on the client-side you write to Kafka directly using the Online Producer client, and note down the offset of the write, then block on getting the server’s committed watermark at least up to that offset or higher. If you wait on less than all replicas before acking, then you need to route the reads accordingly to the freshest replica (based on this watermark pushed from servers to clients), whereas if you wait for al’ you can still read from all, but the write latency will be slower. 2. Another approach is more traditional: write to the server rather than to Kafka, and let the server handle the persistence and replication details (no matter if it uses the current write path impl or a different one). Simpler to implement from the client lib’s perspective, maybe slower as there is possibly an extra hop (client -> server -> kafka -> server -> client).

I agree with Felix that the placement of the service is a cosmetic detail. IMO servers themselves could host this endpoint (just register listener in servers startup sequence) if the write rate is relatively low, meaning it does not impact reads or cause excessive GC. It can be a relatively lowkey service that chains the futures returned by online producers with the client response future. We get the same set of guarantees: once the future is completed, the write is durable, but it does not necessarily mean it has been applied or that your next request will see it.

Thank you all. Very insightful discussion

> but essentially, it boils down to waiting on an ack from 1, some or all server replicas before returning the write API call as successful.

>

turning it into Cassandra essentially 😄

Bah… that could be one of the options, though not the one I was envisioning… that being said, in the words of the great philosopher zwu: given enough code changes, any system can become any other system. The concept of tunable ack quantity is very widespread in dist sys. Kafka producers have the same knob while being a leader-follower system, and Cassandra as well while being a quorum consistency based system. In the case of Venice, since the write path is heavily reliant on Kafka, it would still remain mostly like a leader-follower system, though that depends what part of the architecture are you zooming in on (within a region, the single Kafka there acts as leader, and all Venice servers are replicas applying writes in the same order; but in a multi-region system, there are independent Kafkas in each region, and independent Venice leader servers in each region as well, and those can consume writes in different orders, which starts to become a little more similar to Cassandra’s eventual consistency model).

given enough code changes, any system can become any other system

Ah, the old "YOLOship of Theseus"