Felix GV
09/05/2023, 12:18 AM3. Would you recommend D2 or "router-based" routing? Would router become a bottleneck over time?With D2 it also goes through the router, so the performance is the same. The upside of D2 is that it provides service discovery, so that if some router instances go up and down the clients can be aware of this. The downside is that D2 does this via Zookeeper, which at very large scale can be a bottleneck, and is an extra moving part (you could use a single ZK for Venice, Kafka and D2 if you wanted to, or spread these ZK workloads across two or even three different ZK deployments, with different cost/complexity/isolation tradeoffs). There is also a client lib which skips the routers and goes straight to the servers. We call it the fast-client (as opposed to the thin-client). We use the fast-client in production today but only in a limited way, with some of our most committed internal users that can work with us to debug issues and take patches rapidly. I would recommend starting off with the thin-client as it is the simplest and by far the most battle-tested. The Da Vinci client is also quite well tested at this point but inherently more complex to operate due to making the client service stateful. Eventually, we would probably make the fast-client into the default client lib, but we’re not there yet…
Felix GV
09/05/2023, 12:18 AM4. Could we use Kafka only for informing server nodes about changes while keeping the data in object storage?That is not supported today but I think we would be open to exploring that design change.
Felix GV
09/05/2023, 12:19 AM5. During the last sync session, you mentioned that a push job creates a new topic each time it runs. If we understood that correctly, what's the motivation for it?Yes, that is correct. There are a few motivations for this design choice… 1. Each dataset version having its own "version topic" simplifies the follower replica and Da Vinci code paths: rehydration requires just scanning the topic of the current version from beginning to end. 2. Also, this allows us to delete the old version topic to be retired (thus reclaiming space faster), rather than relying on time retention (which provides much less control). 3. Finally, in a hybrid (batch+stream) store, the outcome of merging a streaming write on top of whatever was there before could lead to a different outcome if the value that was pushed from batch is different. This complexity is encapsulated in the leader replica, which writes to its version topic. Hope that makes sense…!
Felix GV
09/05/2023, 12:19 AM6. How does Venice ensure data redundancy (e.g., in case of a node failure)?Kafka and Venice are both replicated, and they each have the ability to self-heal if a replica is down for some period of time. Not sure if you want to drill deeper than that into the details…?
Felix GV
09/05/2023, 12:19 AM7. Are there any recommendations regarding the server upgrades?
The Venice backend is made up of three services: controllers, routers and servers. Typically, we recommend upgrading controllers first, then the rest. The reason for that is that servers have versioned protocols to replicate data between them and if there is a change to that protocol (which is not that common… maybe once per year or so…), then old servers can fetch the schema of the newer protocol from the upgraded controllers. As is usually the case for distributed (replicated, partitioned) systems, we perform upgrades in a rolling fashion, one (or few) nodes at a time.
Felix GV
09/05/2023, 12:20 AM8. Are there any recommendations about cluster sizing/capacity planning?Hmm, good question… I guess it depends on the workload… but for us anyway, the majority of our Venice clusters are storage-bound. The default server config is optimized for that and will work well for a node that has a local SSD mounted on it, and a little RAM to cache the warm part of the data. In this common case, we would leave replication factor at the default of 3. For workloads that are throughput-bound, we enable RocksDB’s PlainTable format, which works best for a node sized to fit the whole dataset in RAM (in which case the SSD is not necessarily needed). In this case, we may scale the read capacity of the cluster by increasing replication factor. Regarding throughput capacity per node, that will likely vary a lot based on the number of cores available, workload details and possibly other factors… we can chat about it further if you’re interested.
Felix GV
09/06/2023, 2:55 PMFelix GV
09/06/2023, 3:48 PMEnrico Olivelli
09/11/2023, 6:41 AMZac Policzer
09/21/2023, 10:13 PMAleksey Chepovoi
10/03/2023, 1:12 PMFelix GV
10/03/2023, 7:18 PMDejan Mijic
10/09/2023, 7:12 AMVenicePushJob
to support it. Since we started to work on this, would you be willing to accept this modification to the main codebase, or should we keep it on our fork?Elijah Grimaldi
10/31/2023, 8:24 PMElijah Grimaldi
11/02/2023, 7:01 PM[BUG] venice-admin-tool doesn't parse map arguments correctly #647
and was wondering how I could recreate the bug and test my code for JSON argument inputs. On the issue a command is cited
java -jar venice-admin-tool-all.jar --update-store --url <http://ltx1-app6326.stg.linkedin.com:1576> --cluster mt-0 --store TestGpuJobsDataStoreV1 -storage-view-configs '{"changeCaptureView": {"viewClassName": "com.linkedin.venice.views.ChangeCaptureView","params": {}}}'
To reproduce the bug, but I don't have the same jar in my build.Elijah Grimaldi
11/08/2023, 9:04 PMElijah Grimaldi
11/10/2023, 1:58 AMElijah Grimaldi
11/11/2023, 6:27 PMElijah Grimaldi
11/11/2023, 6:28 PMElijah Grimaldi
11/15/2023, 5:22 PMElijah Grimaldi
11/15/2023, 5:23 PMElijah Grimaldi
11/17/2023, 4:58 PMKoorous Vargha
11/27/2023, 11:45 PMKoorous Vargha
12/01/2023, 4:42 AM// Check the migration status for all source and destination cluster stores
Pattern systemStorePattern = Pattern.compile(storeName);
Stream<String> allSrcStoreNames = srcControllerClient.getClusterStores(srcClusterName).stream().filter(srcStoreName -> systemStorePattern.matcher(srcStoreName).matches());
Stream<String> allDestStoreNames = destControllerClient.getClusterStores(destClusterName).stream().filter(destStoreName -> systemStorePattern.matcher(destStoreName).matches());
allSrcStoreNames.forEach(srcStorename -> printMigrationStatus(srcControllerClient, srcStorename));
allDestStoreNames.forEach(destStorename -> printMigrationStatus(destControllerClient, destStorename));
Zac Policzer
12/01/2023, 4:44 AMZac Policzer
12/01/2023, 4:44 AMKoorous Vargha
12/01/2023, 4:45 AMKoorous Vargha
12/06/2023, 11:00 PM