> (2) Have you considered running Venice as a sidecar container? As compared to your current client-library based approach. They are pros and cons but would like to understand your experience on maintaining that client-facing library, especially you cannot quite control the binary releases.
Yes we did… but so far the sidecar story at LinkedIn is a bit immature, so we’re sitting on the sidelines for now. There is work on this (at LI, not in Venice specifically) so we may jump on that bandwagon eventually…
I will say, however, that the current architecture of Da Vinci is to have the RocksDB database accessed directly from the host application’s process, via JNI, and the overhead of that is quite low (we’ve benchmarked single digit microseconds including everything: JNI overhead, RocksDB lookup [assuming the PlainTable / all-in-RAM config], and Avro deserialization). Whereas with a sidecar we would hit the network stack (even though it would be on loopback) and it seems like it might be challenging to be quite as fast. Still, the maintainability benefits may warrant pursuing that approach.
Regarding client releases and uptake, I acknowledge that it has been historically painful, and still is to some extent, but I will say that it has gotten better, at least internally at LI… Through a mix of tooling and processes, we now have much better dependency hygiene than before and our dependents do pick up new library versions much more quickly than before… That is not Venice-specific and does nothing for folks outside LI who likely have the same challenges, but I just wanted to share that data point nonetheless.