venice


<https://github.com/diegopacheco/rust-playground/tree/master/jni-fun>


<https://medium.com/@brian_90925/llms-and-the-emerging-ml-tech-stack-6fa66ee4561a>


<https://github.com/facebookresearch/faiss>


<https://twitter.com/thetaph1/status/1662148334134476835?s=12&amp;t=m6BQM90Va2-_ZODxR6K7FQ>


<https://github.com/ozangunalp/kafka-native>


<https://twitter.com/criccomini/status/1669002835038728192?s=12&amp;t=m6BQM90Va2-_ZODxR6K7FQ>


<https://twitter.com/apachekafka/status/1669385613697769472?s=12&amp;t=m6BQM90Va2-_ZODxR6K7FQ>


<https://twitter.com/stereosky/status/1671174970687250432?s=12&amp;t=m6BQM90Va2-_ZODxR6K7FQ>


<https://news.apache.org/foundation/entry/the-apache-software-foundation-announces-new-top-level-project-apache-kvrocks>

Very cool and congratulations. Curious where exactly is Venice in this eco-system?

We could implement Sinks and Query agents to read/write on Venice.
It is a matter of just writing the adapter code (probably less than 1k lines of code).
Look at the Pinecone agents

I think you might be interested in this, <@U048Z66EUA0>

Would it be valuable to support plugging LanceDB as the storage engine, as another option, alongside RocksDB?

yeah that would be awesome,  How easy is it to integrate? LanceDB is pretty interesting with its Lance data format an  innovative and new columnar data format for ML. Lot of bets on lanceDB

Thinking out more on this, I think having the embedded storage engine in a way like our DaVinci client seems to be a good use-case (still needs some validation from real users). They can still use avro based rocksDB Venice to store the features and vectors (float is good with avro). Have lanceDB serving their application which ingests the required vectors and metadata as an embedded vectorDB. I guess a niche usecase to start with.

I wonder if there would be use cases for WASM in Venice :thinking_face:

Happy for Timeplus! I worked closely with these founding people during my Splunk days!

<https://www.linkedin.com/posts/nikhilgarg28_key-architectural-ideas-powering-fennels-activity-7108203961689522176-pcuE|https://www.linkedin.com/posts/nikhilgarg28_key-architectural-ideas-powering-fennels-activity-7108203961689522176-pcuE>

Could be interesting…
<https://db.cs.cmu.edu/events/ml-db-2023-featureform-yeah-your-database-can-do-that-simba-khadder/>

This might be interesting…
<https://github.com/TileDB-Inc/TileDB>

it is a new OSS project that is joining the Apache Software Foundation

Interesting… first because the concept of a "merry commitmas" post sounds fun :joy: but also because they’re working on similar challenges as us… a CRDT implementation that is slow to catch up on a large amount of updates. We could get an inspiration or two from them :smile:

Merry commitmas everyone :christmas_tree:!
<https://automerge.github.io/blog/2023/12/21/merry-commitmas/>


<https://www.linkedin.com/posts/ryanlpeterman_stop-using-dated-cli-tools-here-are-9-better-activity-7157217662845046784-w8_f?utm_source=share&amp;utm_medium=member_ios>


<https://github.com/NVIDIA/cuda-checkpoint>


<https://x.com/mamboitaliano__/status/1819777135227142454?s=12&amp;t=m6BQM90Va2-_ZODxR6K7FQ>

Nice: <https://duckdb.org/2024/12/09/duckdb-avro-extension.html|https://duckdb.org/2024/12/09/duckdb-avro-extension.html>

<@U067HQNDH0A>, <@U03USK369J5>: would this be useful for a DVRT plugin?

i guess it would make it easier.  Then we would need to do very little to DuckDB venice data

Is the idea we provide an interface that already implements DuckDB as the storage engine?

Also, what use-case would DuckDB apply to?

Would users being doing OLAP on a subset of partitions, or would they need to subscribe to all partitions?


<https://github.com/facebook/rocksdb/pull/13197>

I guess step one is learn: <https://github.com/facebookresearch/faiss>

So.  Been thinking about consumer groups/auto sharding of davinci.  Went down the rabbit hole a bit on a few approaches and came across this: <https://ratis.apache.org/>

Pretty active project.  I'm considering it as a thought exercise of how this would work if we didn't need the Venice server/system store to drive the assignment/load balancing.  We might even be able to build a POC just around Davinci to start.

I stumbled upon this a while back… though I was looking at it for controller A/A (i.e. getting rid of parent controllers).

I think it’d be good to kick the tires on it and get acquainted! If it’s in a code path that isn’t used ultra intensively and where we can afford to make a mistake or two, that may be an added bonus :joy:

I think I can just do it as client code on top of Davinci/DVRT.  First POC would be just to rebalance the subscriptions

And then if that works, take it a step further and add it into the interface.  Essentially add a new method to Davinci that's subscribe(String groupId)

and probably a getAssignment for being able to publish routing state information.

This might be overly ambitious, but could we take the KRaft approach to get off ZK, and use RocksDB as the storage? This is what <https://github.com/tikv/raft-rs|TiKV> is doing.

Potentially.  That is a huge gutting.  We'd need to replace all that helix does on top of the metadata stuff.

<https://dl.acm.org/doi/pdf/10.1145/3341301.3359656|https://dl.acm.org/doi/pdf/10.1145/3341301.3359656>


<https://www.vldb.org/pvldb/vol18/p1509-miao.pdf>