https://authzed.com logo
Join Discord
Powered by
Hey everyone! Our SpiceDB cluster (`v1.
# spicedb
p
Hey everyone! Our SpiceDB cluster (
v1.39.0
) is having some performance issues and it seems like our cache hit/miss ratio is low. 
dispatch
cache sits at around 5-7% and 
cluster_dispatch
at around 3-5%. We've tried increasing the size of the cache but the actual heap used by the process is hardly ever above 400MB. We also increased the num_counters for Ristretto, but the cache performance was unchanged. I'm wondering if there's any guidance on how to best configure the cache max hit ratio.
y
cache hit ratio depends much more on the workload than on the configuration of the cache
if you're not seeing lots of cache eviction pressure, it's more likely that your workload just isn't making good use of the cache
are you using 
fully_consistent
anywhere? do you have a consistent write workload? how much read throughput do you have?
p
Thanks for getting back to me @yetitwo Most of our traffic (~80%) uses 'minimize latency', with some using 'at least' (~17%) and some using full consistency. Our cache bytes added/removed ratio is 1:1 and at about 9MiB per minute
y
and the write workload?
my sense is that if your subproblems are not overlapping and your read throughput isn't very high, the thing driving the ratio that you're seeing is basically just that every ~5s or so spicedb is selecting a new optimized datastore revision and that means the cache entries associated with that revision need to be populated
if your clients can tolerate a higher level of staleness, one experiment to run would be setting 
--datastore-revision-quantization-interval
to something higher (30s?) and seeing if your cache hit ratio goes up
but there's also a chance that cache hit ratio is the wrong thing to be chasing
when you say "performance issues," what are you talking about
p
Most of our load is CheckBulkPermission and ReadRelationships. Writes are negligible Our performance problems are because too much traffic is going between nodes, and it lands in the DB and the nodes drain the connection pools and everything slows down
y
if you've tuned your connection pool size according to the number of nodes and the number of database connections available on the datastore, the number of nodes in the pool shouldn't matter
are you seeing positive values for 
pgxpool_empty_acquire
?
p
@yetitwo yeah pgxpool_empty_acquire spikes up and stays up
y
i take it this is on postgres?
p
Yeah
y
then yeah when we run into this condition in our own clusters the first thing we do is scale up postgres
and get more connections
if the datastore is otherwise unstressed, it might be worth looking at fronting it with pgbouncer
p
pgbouncer is in place already
y
then yeah i'd do what you need to do to make more database connections available to spicedb
p
Thanks @yetitwo - in your experience, for a medium size cluster, how many connections should each SpiceDB node have access to?
y
it depends a lot on your workload 🤷 number of relations written, shape of your schema, query throughput, query shape
4 Views
PreviousNext
Powered by