3. Would you recommend D2 or "router-based" routing? Would router become a bottleneck over time?

With D2 it also goes through the router, so the performance is the same. The upside of D2 is that it provides service discovery, so that if some router instances go up and down the clients can be aware of this. The downside is that D2 does this via Zookeeper, which at very large scale can be a bottleneck, and is an extra moving part (you could use a single ZK for Venice, Kafka and D2 if you wanted to, or spread these ZK workloads across two or even three different ZK deployments, with different cost/complexity/isolation tradeoffs). There is also a client lib which skips the routers and goes straight to the servers. We call it the fast-client (as opposed to the thin-client). We use the fast-client in production today but only in a limited way, with some of our most committed internal users that can work with us to debug issues and take patches rapidly. I would recommend starting off with the thin-client as it is the simplest and by far the most battle-tested. The Da Vinci client is also quite well tested at this point but inherently more complex to operate due to making the client service stateful. Eventually, we would probably make the fast-client into the default client lib, but we’re not there yet…

4. Could we use Kafka only for informing server nodes about changes while keeping the data in object storage?

That is not supported today but I think we would be open to exploring that design change.

5. During the last sync session, you mentioned that a push job creates a new topic each time it runs. If we understood that correctly, what's the motivation for it?

Yes, that is correct. There are a few motivations for this design choice… 1. Each dataset version having its own "version topic" simplifies the follower replica and Da Vinci code paths: rehydration requires just scanning the topic of the current version from beginning to end. 2. Also, this allows us to delete the old version topic to be retired (thus reclaiming space faster), rather than relying on time retention (which provides much less control). 3. Finally, in a hybrid (batch+stream) store, the outcome of merging a streaming write on top of whatever was there before could lead to a different outcome if the value that was pushed from batch is different. This complexity is encapsulated in the leader replica, which writes to its version topic. Hope that makes sense…!

6. How does Venice ensure data redundancy (e.g., in case of a node failure)?

Kafka and Venice are both replicated, and they each have the ability to self-heal if a replica is down for some period of time. Not sure if you want to drill deeper than that into the details…?

7. Are there any recommendations regarding the server upgrades?

The Venice backend is made up of three services: controllers, routers and servers. Typically, we recommend upgrading controllers first, then the rest. The reason for that is that servers have versioned protocols to replicate data between them and if there is a change to that protocol (which is not that common… maybe once per year or so…), then old servers can fetch the schema of the newer protocol from the upgraded controllers. As is usually the case for distributed (replicated, partitioned) systems, we perform upgrades in a rolling fashion, one (or few) nodes at a time.

8. Are there any recommendations about cluster sizing/capacity planning?

Hmm, good question… I guess it depends on the workload… but for us anyway, the majority of our Venice clusters are storage-bound. The default server config is optimized for that and will work well for a node that has a local SSD mounted on it, and a little RAM to cache the warm part of the data. In this common case, we would leave replication factor at the default of 3. For workloads that are throughput-bound, we enable RocksDB’s PlainTable format, which works best for a node sized to fit the whole dataset in RAM (in which case the SSD is not necessarily needed). In this case, we may scale the read capacity of the cluster by increasing replication factor. Regarding throughput capacity per node, that will likely vary a lot based on the number of cores available, workload details and possibly other factors… we can chat about it further if you’re interested.

Are you interested in doing the sync today at 5 PM Central Europe time, @Dejan Mijic, @Ilya Esin, @Dmytro Prokhorenkov?

If you end up using the Venice Helm chart and have any questions, feel free to bug @Enrico Olivelli about it 😄 … you folks are in the same time zone, so it might be even easier to get questions answered 😄

Hey folks, Today I have started to rebase VIP-1 implementation https://github.com/linkedin/venice/pull/471 It will take some time, but this week I have time to clear all the comments and add the tests that are missing to be compliant with the code-coverage checks

Anyone know how to update the bug/feature request format? We should add more tick boxes for the online producer and the change capture client.

Hello everyone, my name is Alex, we help top-notch developers and startups find each other. 🙂 Through our platform, founders and CTOs can quickly build their engineering dream teams. And top IT specialists can find long-term remote work with competitive rates. Today we've launched on Product Hunt and would appreciate your feedback! https://www.producthunt.com/posts/expert-remote P.S. if anyone lost jobs in the current market, feel free to leave your CV here: https://developers.expertremote.io 😉

FYI, next community sync up is tomorrow, in case anyone’s interested.

Hi people, as we discussed before, we (LiveIntent) prefer using S3 as the data source, and it should be relatively straightforward to adjust the

VenicePushJob

to support it. Since we started to work on this, would you be willing to accept this modification to the main codebase, or should we keep it on our fork?

New social media post mentioning Venice! Feel free to like or reshare with your networks if you’d like! • LinkedIn • Twitter

Shout out to @Moditha Hewasinghage for the S3 support contrib 😄 ! • LinkedIn • Twitter

Hello everyone

Hey guys, I am currently working on the issue:

[BUG] venice-admin-tool doesn't parse map arguments correctly #647

and was wondering how I could recreate the bug and test my code for JSON argument inputs. On the issue a command is cited

java -jar venice-admin-tool-all.jar --update-store --url <http://ltx1-app6326.stg.linkedin.com:1576> --cluster mt-0 --store TestGpuJobsDataStoreV1 -storage-view-configs '{"changeCaptureView": {"viewClassName": "com.linkedin.venice.views.ChangeCaptureView","params": {}}}'

To reproduce the bug, but I don't have the same jar in my build.

Hello, I'm curious about the main advantages of using "venice-thin-client" as opposed to "da-vinci-client," and vice versa. I understand the "thin" client is stateless, so it skips the overheard of maintaining a local repository of the upstream store. Is the idea that there is a performance boost in the davinci client from only having to query the local repository rather than sending a request to the backend servers, however one may choose not to use it because of complexity introduced by the client?

Another quick question about davinci, I was wondering since the it specializes in reading data from Venice, are there issues with the local data on the davinci client becoming stale and how is this dealt with? How often does the client have to update its local storage and how does this effect performance if at all?

"So the idea Zac was pitching is basically to extend this ability to change storage config at start time and make it even more flexible. Let the user do more stuff with the data that Da Vinci ingests on their behalf. Instead of letting the data go into RocksDB, it might go into something else, such as SQLite, DuckDB, Lucene or KuzuDB." In order for this to be an option, there would have to be derived classes of StorageEngineFactory for each of these options right? Also wondering if for example there is some boolean flag in the davinci client config, should there be a new derived class of the DaVinciClient that is instantiated if that flag is true or should it be defined in the DaVinciClient interface to be used by all derived classes? I know you guys haven't finalized the details yet, but I thought it would be good practice for me to try to keep reading through the codebase and try to think about ways to implement the idea.

For the part about different DB options, I guess my question is are we exposing more predefined options to be changed in the configuration or are you thinking that it should be able to be fully customized if that makes sense

I think I got a little confused at the end but to confirm the first steps we talked about were writing up an interface (similar to the ObjectCacheBackend) that would allow for operations on the davinci backend in-memory storage?

Or did I get it wrong?

What is the best way to run a local instance of DaVinci for testing?

Hello! 👋 My name is Koorous and I'm looking forward to becoming a contributor.

Hi @Zac Policzer, regarding this issue, what was the regex you had in mind to find all of the system stores within a cluster? Thanks! I have this so far:

// Check the migration status for all source and destination cluster stores
Pattern systemStorePattern = Pattern.compile(storeName);
Stream<String> allSrcStoreNames = srcControllerClient.getClusterStores(srcClusterName).stream().filter(srcStoreName -> systemStorePattern.matcher(srcStoreName).matches());
Stream<String> allDestStoreNames = destControllerClient.getClusterStores(destClusterName).stream().filter(destStoreName -> systemStorePattern.matcher(destStoreName).matches());

allSrcStoreNames.forEach(srcStorename -> printMigrationStatus(srcControllerClient, srcStorename));
allDestStoreNames.forEach(destStorename -> printMigrationStatus(destControllerClient, destStorename));

Ah right. There should be a constant for it. Something like "venice_system_store_" is the prefix I believe

I'd need to pull out my IDE. I'm away from my keyboard just mow

Take your time. No rush.

@Zac Policzer is there a way to view this talk without previously being registered for the event? I'm unable to register since the event has passed.