Reminder: Monday is our community syncup meeting at 9AM PST/12 PM Eastern/6 PM Central Europe. Here is the Microsoft Teams Link to join the meeting.

@Zac Policzer @Koorous Vargha hope you guys can join on Monday. From the previous follow-up items: https://venicedb.slack.com/archives/C0567KESECQ/p1773075655601829?thread_ts=1772816420.761369&cid=C0567KESECQ As we prepare to run Venice in production in the next few weeks, would be great if you could share practical guide on the capacity requirements to maximize performance (routers vs servers vs controller) as well as guide on how to use the plain table for in-memory clusters.

I’ll be there. Let me reach out to our POC for those questions since they are back

Be sure to get a bit of Alis time w.r.t. capacity

Yup. He just joined the channel

Sweet

Meeting notes: Higher priority items 1. Venice to open-source oncall runbook and dashboard setup 2. How to setup plain table for servers 3. How to do capacity planning @Ali will you be able to help here? 4. Is Venice primarily still using JDK17 or did we migrate to JDK21? a. I checked our server setup and it looks like it's using JDK17 still. @Ali did you do any performance benchmarking on JDK21 and if there were any gains over JDK17?

we have hit some limitations using jdk21 for runtime, we hope to resume that soon, running things on jdk 17 for now .

Found smth concerning - we mostly used default configs in the benchmarks, but looking at the defaults that are used for the single-dc servers, plain table is already enabled: https://github.com/search?q=repo%3Alinkedin%2Fvenice%20rocksdb.plain.table.format.enabled&type=code Maybe we should consider disabling it there, as it's not really the default behavior that is expected. If I understand it correctly,

<http://rocksdb.block.cache.size.in|rocksdb.block.cache.size.in>.bytes=2147483648

is not even used, and the Venice servers will try to greedily use the available memory. @Ali @Koorous Vargha could you help confirm?

Yes that's true. I was not aware that the docker examples have plain table enabled by default. Plain table will put the full dataset in memory

In some sense, plain table looks pretty similar to the block cache based - if the data is not there yet, it still needs to load it from SSD. It's just that plain table is not limited, but with block cache we have hard limits with LRU eviction policy. I guess if we have 1TB of data in SSD, and only 100GB available for plain table - when data is not in memory, it would result in a page fault, subsequently read from SSD and load into page cache. Correct me if I'm wrong.

Claude's take. @Ali can you please verify if the plain table part is correct? > Plain table relies entirely on the OS page cache. It uses mmap to access SST files — when a page isn't in memory, it triggers a page fault and the OS loads it from SSD into page cache. The "limit" is whatever RAM the OS decides to give page cache (which is everything not used by the process heap, and subject to pressure from other processes). There's no application-level control over what stays in memory. > > Block-based with block cache gives you an explicit, application-managed cache with a hard size limit and LRU eviction. When a block isn't in the cache, RocksDB does a direct SSD read (or uses use.direct.reads=true to bypass page cache entirely) and loads just that block into the block cache. > > So you're right that both end up reading from SSD on a miss. The real differences are: > > 1. Predictability: Block cache gives you a guaranteed memory budget. Plain table's effective cache size fluctuates with OS memory pressure — other processes can silently evict your pages. > 2. Control: With block cache you can tune cache size, use tiered caches (compressed secondary cache), and get cache hit/miss metrics. With plain table you're at the mercy of the kernel's page cache heuristics. > 3. Production suitability: Plain table was designed for read-only, in-memory datasets where the entire dataset fits in RAM. If 1TB of data fits in your available memory — great. If not, the random page fault behavior under memory pressure is hard to reason about and tune.

the summary is correct, Block-based works best if we need explicit control in user-space, and can pay small latency fee for it. Plain table puts us completely at the mercy of OS memory management, so it needs extra care in making sure there are no noisy neighbor processes and memory usage is properly managed . It in return gives us kernel layer memory management and a measurable reduction in latency specially in highly concurrent reads and writes .

@Ali @Koorous Vargha by any chance, could you share your server configs (nothing sensitive) for plain table and standard setup? Was wondering if we are missing on some other stuff before we go live in Prod.

it might work better if you share your configs and we can explore together .

Btw @Sergey Makagonov for the oncall runbook, the PR is still in progress. But if you need to reference it now, you will be able to use the links included in the PR description that's pointing to my fork's website deployment.

Amazing! Thanks @Koorous Vargha!

left some comments 🙂

@Koorous Vargha would it be possible to know how many connections from clients your routers are able to handle? Context: since we're a Python shop, we end up spawning 10k+ small pods in our main backend, and due to continuous deployment, they live on average for 30-40minutes. Even with http2, there are serious concerns about the number of connections as well as the connections churn. Knowing how many connections you guys can handle at the moment would give some hopes whether we should proceed with our Python HTTP client or go straight into an approach of having a "java client(s) per host" and let the python pods talk to them locally.

10K connection we can handle using the resources mentioned above .

I think what'd probably chew you up first is high wave of turns for ssl session establishment. So if all 10k of those pods turned at once you'd probably see latency spikes that correlate with cpu turn to make the connection. We should probably be able to get a rough estimate of the number we use in production with current metrics. Is it correct to assume the 10k pods is 1 connection per pod, or is there some multiplier?

Correct. One connection per pod

@Zac Policzer @Ali Can you one of please take a look at this very small PR

what's the default block-based cache size ? should we tweak that as well ? ( 2G ? 4G by default )

2GB in single-dc-configs. The multi-dc ones will resolve to the default in the code base which is 16GB, as it is not overriden there.

yeah, 16G is too much for experimental cases . might break smaller docker setups

PR is updated, please take another look.

I think what'd probably chew you up first is high wave of turns for ssl session establishment. So if all 10k of those pods turned at once you'd probably see latency spikes that correlate with cpu turn to make the connection.

thanks for the input folks. We'll go in the direction of talking to the routers through an HTTP reverse proxy (envoy proxy). We run 20 pods on a single node on average, so the connections to routers would be from the proxy instances rather than from each backend pods. Ideal scenario: have a DaemonSet, where we have a single proxy instance on each node. This would eliminate any extra hops (see screenshot), pods would be talking to the colocated proxy instance. However, we've seen Envoy Proxy perform great even as a Deployment with limited number of instances. Earlier it helped us reduce connections to Redis without much of a latency penalty. We'll stat with this and will evaluate how much the actual penalty is.