Thanks a lot for the detailed reply @Felix GV. A follow up question. For Da Vinci mode, do you keep a copy of the data both on the client side and on the Venice server side? Do you really need the server-side copy?

We have a Venice talk lined up at J on the Beach in Spain next month! Who's coming 😁 ? https://www.linkedin.com/posts/j-on-the-beach_jotb25-duckdb-rocksdb-activity-7318664574696632322-nsbo

I've ran into issues while running

./gradlew check --continue

with Ubuntu 24.04.1 LTS on WSL2 (Windows 11). In a nutshell: numerous integration/e2e tests would fail due to timeouts. I'll setup the work environment on my Macbook Pro instead. Further details in reply

@Koorous Vargha I just noticed dark mode got enabled on the docs 👀 very nice

Hey folks, question regarding tests. In this current PR, I'm adding unit tests where each tests takes ~6 seconds due to exponential backoff. When running

:internal:venice-common:test --tests "com.linkedin.venice.utils.TestHelixUtils"

, the tests run sequentially, which takes some time. I was wondering if you knew of any way to run multiple tests in parallel. I remember that

./gradlew check --continue

would spawn x threads when running the full suite, and thought perhaps I would be able to do the same thing here, or similar. Thanks! EDIT: it doesn't actually take that long, but thought it might be helpful to know in the future.

Interesting talk by @Jia about Pinterest's KV Store, used for ML feature serving:

All checks passing on this new PR 🍀 However, I think much discussion will be required in order to arrive at a proper solution we can all agree on 😅 Exciting! Have a great week everyone! 👋

One of the things we were talking about today @Gabriel Drouin

@Felix GV A very important question I wish to solve today... Would you be able to tell me why the tests in the CI are named in french? 😆

I have added the welcome bot to our slack channel! Anyone who join #C03SLQWRSLF will trigger the workflow. Any extra info you want to add?

Hey folks, is there a working example of fast client wiring? The closest thing I found to a guide is

com.linkedin.venice.fastclient.utils.TestClientSimulator

. Is it an appropriate thing to follow (minus the mocks and dummies 🙂 )? Thanks!

The latest conference talk is now posted online! Reshare and like this social media post, if you'd like, to promote Venice to your networks!

Direct YouTube link here:

HyBRID: Hybrid Batch/Real-time Ingested Data

New functionality, new video! Watch it to learn about: Facet Counting in Venice

If you would like, please re-share our announcement to your networks! • LinkedIn • Bluesky • Twitter

VPJ.log

Hi, we are slowly but steadily continue with testing Venice. We are trying now with the bigger setup and trying to push 4TB store but we fail. We use 0.4.613 version with the k8s setup through EKS. We are pushing with 100 partitions and 12 venice-servers. It seems that some partitions fail to allocate replicas. The root cause seems to be the failure of opening RocksDB metadata database on some venice-server pods which manifests in such exceptions:

Caused by: org.rocksdb.RocksDBException: While lock file: /opt/venice/rocksdb/rocksdb/client-id-to-email-hash_v26/client-id-to-email-hash_v26_1000000000/LOCK: Resource temporarily unavailable

I am attaching VPJ, leader controler and one of the venice-server logs. It seems that venice-server tries to open the metadata database from multiple threads and the locking failures are not retried. Is it a known issue? Are we doing anything wrong? 🤔

The 2nd edition of Designing Data-Intensive Applications mentions Venice!

For those in the Bay Area, this could be an interesting meetup: https://luma.com/e7feg2i6

Has anyone looked into integrating Venice with Snowflake as a batch data source or batch ingestion layer? Since we already use Snowflake for transformations, I’m wondering what the main challenges or limitations might be.

This is quite similar to how large values get chunked inside the Venice server. If instead of appending a flag to the key, Venice used a separate table (column family) to store chunks, then it would essentially be the same. https://www.linkedin.com/posts/ben-dicken-78797a73_postgres-uses-toast-to-store-large-variable-sized-activity-7396908837485727744-dG76?utm_source=share&utm_medium=member_ios&rcm=ACoAAAEk238BXlhz1s5hXc96bKIJJ-eWwXUnlas

@Felix GV The latest Venice blog post is out today: https://www.linkedin.com/blog/engineering/infrastructure/evolution-of-the-venice-ingestion-pipeline

Great to see that post finally come to light! I just re-read it and it’s even better than the first time I did! Congrats @Gaojie Liu and team!

Just posted on the VeniceDB LinkedIn page.

idk why, but using a social media account and posting as VeniceDB feels like this:

Blue sky post up as well: https://bsky.app/profile/venicedb.org/post/3m5yzjby7tk2t

🎥 Proud to share our two latest ASQ videos! • Data Integrity Validation by @Lei Lu: • Stateful CDC Client by @Koorous Vargha: Check them out and let us know your thoughts! 🚀

Is the RocksDB implementation located in

com.linkedin.davinci.store.rocksdb

considered the canonical implementation for local storage across both the Venice Server and the Da Vinci Client?