Hi, We've been using SpiceDB for almost two years now, and I would say we have a fairly large schema. We've run into an issue twice where the namespace cache hit ratio suddenly starts dropping, and then two out of our four SpiceDB pods continuously increase their heap memory usage until they eventually stop responding to authorization requests. After the first occurrence, we increased the namespace cache size from 256 MiB to 512 MiB, but the issue happened again. What's interesting is that we haven't been able to identify any clear trigger. One time it started on a Sunday, and the other time it started during a regular weekday morning. Our relevant configuration is: SPICEDB_DATASTORE_REVISION_QUANTIZATION_MAX_STALENESS_PERCENT: "1" SPICEDB_DATASTORE_REVISION_QUANTIZATION_INTERVAL: "5s" SPICEDB_ENABLE_EXPERIMENTAL_WATCHABLE_SCHEMA_CACHE: "true" SPICEDB_DATASTORE_SCHEMA_WATCH_HEARTBEAT: "60s" SPICEDB_NS_CACHE_MAX_COST: "512MiB" SPICEDB_DISPATCH_CACHE_MAX_COST: "256MiB" SPICEDB_DISPATCH_CLUSTER_CACHE_MAX_COST: "512MiB" We are using PostgreSQL as the datastore. Another important detail: our schema is essentially static. We only deploy schema changes together with a new application version and after restarts. The schema itself is roughly 4,000 lines long. I've attached a screenshot showing the namespace cache hit ratio dropping. Under normal conditions it stays close to 100%. After restarting the affected pods, it returned to normal. It also looks like cache entries may have started getting evicted overnight, although unfortunately the screenshot doesn't include the very beginning of the issue. Has anyone seen similar behavior or has ideas about what could cause the namespace cache hit ratio to suddenly drop and memory usage to keep growing on only a subset of pods? https://cdn.discordapp.com/attachments/844600078948630559/1536684965142732900/image.png?ex=6a7c4ce5&is=6a7afb65&hm=3d5c660ca04c593003f547389955c83ae7ab0e72e53ec2c279d38f04c4029588&

what version of SpiceDB are you on?

This issue occurred on v1.54.0. The first occurrence happened on a version prior to v1.50.0.

in 1.56 we fixed an issue with how we were doing cache accounting in 1.54. would you be able to upgrade to 1.56 and compare? we also moved to a different caching library in 1.54 which has better performance characteristics but is also more likely to hold onto entries/less likely to evict them, which can manifest as higher overall memory usage without necessarily reflecting a problem. also, where are you running SpiceDB? k8s? ECS? bare metal?

We are running SpiceDB on Kubernetes. We can upgrade and compare the behavior. As I mentioned, this issue has only occurred twice so far, and we still don't know the exact cause, so it is hard to compare. I've attached a better screenshot that includes a legend.

ahhh you're talking about the ratio dropping a bit

Another Screenshot where the drop is not hidden. https://cdn.discordapp.com/attachments/1536684965415231582/1536761462796718120/image.png?ex=6a7c9424&is=6a7b42a4&hm=bd56760c078044e907d47a8ad081d240c2b538013c86eafaa643995478c333d0&

i'm not sure what to think of that yet. i'll have a think on it. let me know if y'all upgrade to 1.56 and if the behavior is any different.

Ok, I will give it a try and test this.

let us know how it goes!

From a performance perspective, we haven't seen any noticeable differences so far, so it looks like the unified schema cache is working well. We'll keep an eye on it and see whether the memory leak occurs again over the longer term.

awesome. glad to hear it!