This message was deleted.
# general
s
This message was deleted.
i
Yup, also interested in this topic. Currently we are running several RO1 stores on voldemort. Over the time we came to the deployment schema where one store is deployed on one cluster. This does not scale well but at least this approach gives us some performance guaranties. We have several different stores and Data-to-RAM ratio from 14 (data does not fit but local SSDs can handle the load) to 171%(data fits into RAM, but CPU and Network is a bottleneck).
Real question is: how (and if) can we estimate/derive required cluster capacity from current number of requests and compute, network and disk capacity used?
f
So... looking at storage, one way to go about it is to assume you'll have a replication factor of 3, and 2 versions of each dataset (the current one + either a past backup or a future version in the process of being loaded), and finally tack on some RocksDB-related overhead of let's say 1/3rd. So that means if one logical copy of your data (meaning
record count * (average key size + average value size)
) weighs N bytes, then the physical size will be
3 * 2 * 1.33 * N = 8N
You can certainly pack multiple stores in a single cluster, and that will almost certainly increase your efficiency, especially if you have a lot of small and medium-sized stores. One of our multi-tenant clusters has ~500 stores in it.
For the request throughput capacity, it's a bit less straightforward and may require some experimentation with the SKUs available to you...
For both of these dimensions, storage and throughput, Venice offers per-store quotas, so that multi-tenancy can be more predictable. i.e. if one store tries to take more than their quota they can be capped, so the other stores are not impacted. There are various strategies you can consider there, with the two main ones being: 1. The sum of the quota you allocate is equal to or less than the capacity of the cluster, or 2. The sum is higher, in which case you are over-provisioning The first one is safer, while the second one is more efficient. Which one is safe enough depends on whether your use cases tend to spike in correlated fashion or independently... if independently, then the over-provisioning approach might be safe enough.
✅ 1