This message was deleted.

Hey @Suman Karumuri. Answers in line: • Not today no. Though maybe we have some answers depending on what you're trying to achieve? The davinci client today is a read only interface. • This is also not supported today but we have had internal discussion about something like this frequently. Again it'd be good to understand what you're after. Today data transfer is all based off of our PubSubConsumer interface, and it's fast enough that we haven't had much urgency to rework it (always something to do around here 😅 ) • I'm sure I could find some public examples of this. It'd be a comparison between rocksdbjni and straight rocksdn microbenchmark. I'll get back to you.

What Zac said is correct. Some extra details: 1. Da Vinci consumes from Kafka (soon it could be from other non-Kafka pub subs) and writes the data to RocksDB. The end result is some SST files in the Da Vinci data directory, though those are fully managed by the library. Currently, we support 2 RocksDB formats: block-based (ideal for SSD+RAM) and PlainTable (ideal for all-in-RAM). 2. We recently saw an internal use case clocking in at 800K k/v consumed and written per second on a single node (it's fast). And we are continuously optimizing it further. Personally, I think we'll get to 1M writes/sec/node in the not too distant future. Blob-based transfer is on our radar but so far we've resisted adding extra moving parts as it didn't seem quite necessary. Still open to considering it in the future. 3. The JNI layer doesn't add too much penalty from my understanding, though I don't have concrete numbers in terms of the delta it adds. In absolute numbers, I've seen Da Vinci configured in the PlainTable format perform lookup + deserializarion in ~2 microseconds for a 10 KB payload. In the block-based format, on SSD, about 200 microseconds.

Thanks for a quick response. I am looking for a good backend for persistent store backend for yuvi. Currently, it’s an in-memory store but we are looking to persist it to disk. Kaldb already has the bits to persist data to s3, so we are looking for a good local KV store like rocksdb but that is also cloud native.

RocksDB is really solid IMHO. If you decide to use it, you may find that a lot of other things aren't needed anymore... e.g. you might not need to manage the in-memory part anymore, as RocksDB is already super fast if you give it RAM to hold the hot portion of the data. Regading making it cloud native, you might want to check out https://github.com/rockset/rocksdb-cloud

Can I replace rocksdb in Venice with rocksdb cloud?

Good question... I suppose you could. I don't know the differences well enough, but I imagine it's pretty similar and so should be possible. There may be minimal effort to integrate it.