https://authzed.com logo
Join Discord
Powered by
you might also want to try LR3 (its an
# spicedb
j
you might also want to try LR3 (its an experimental flag) to improve LR performance
h
Thank you for the honest gut check! That makes perfect sense. We will fix the connection pool limit first to eliminate the wait time bottleneck, and then see where our actual compute limits are. We are totally prepared to scale up/out if needed. Also, thank you for the tip about LR3! I didn't know about that experimental flag. Since it is experimental, we will proceed with some caution. However, if the standard tuning doesn't give us the improvements we need, we will positively consider enabling it. I really appreciate all your help! I'll report back to the community if we hit any other interesting roadblocks.
j
sounds good
h
thank you!!
Thanks for your help the other day. Since then, I tried removing the intersection from the schema and increasing the 
datastore-conn-pool-read-max-open
. - 4vCPU 8GB RAM ECS Task * 6 - --dispatch-concurrency-limit=100 - --dispatch-lookup-resources-concurrency-limit=100 - --dispatch-cache-max-cost=512MiB - --dispatch-cache-num-counters=500000 - --dispatch-cluster-cache-max-cost=1GiB - --dispatch-cluster-cache-num-counters=1000000 - --ns-cache-max-cost=64MiB - --datastore-conn-pool-read-min-open=20 - --datastore-conn-pool-read-max-open=50 - DB: Aurora Serverless v2 for PostgreSQL (ACU: 10~50) With the above configuration, we are now seeing the following throughput for 200 concurrent requests:
Copy code
mean   : 2401.47 ms
median : 2550.78 ms
p95    : 2670.66 ms
p99    : 2675.55 ms
min    : 200.63 ms
max    : 2676.74 ms
However, our goal is to achieve this throughput at 600 concurrent requests, so we still need to improve it further. I checked the Otel metrics, and since the requests in our performance tests tend to have a high cache hit rate (which is not an issue as this matches our production traffic), increasing 
datastore-conn-pool-read-max-open
didn't seem to provide much benefit. Also, while LR3 further reduced the minimum latency, it didn't significantly improve the overall throughput for the 200 concurrent requests. By the way, as a baseline, is it theoretically possible for SpiceDB to process 600 concurrent LookupResources requests (with 500 resources each) within 1~3 seconds? I understand that it naturally depends on the infrastructure specs, but is it feasible by design in SpiceDB? I'm asking this to help us decide whether we should continue moving forward with this standard approach.
j
you need to increase the min open as well
and it completely depends on the size of the cluster
but easily
h
Thanks > and it completely depends on the size of the cluster > but easily By the way, as a general rule of thumb, if we want to handle 600 concurrent LookupResources requests for 500 resources (15,000 subjects (200 members/team * 75 teams)) within 1~3 seconds, roughly what kind of scale or specs would we be looking at?
j
no idea; depends heavily on schema shape
and caching
you'd have to test it
why are you making 600 concurrent LR calls?
h
This is because, in our production environment, although there might be a few seconds of jitter, we can currently experience up to 600 concurrent list view requests. Ideally, we would like to handle these requests without relying on pre-computation via the watch API or using intermediate tables.
j
you're going to need much larger SpiceDB machines then
LR has to walk all the various portions of the graph for the user
h
I've been experimenting with some tuning, and to handle 600 concurrent LookupResources in about 3 seconds, I ended up needing around 20 ECS tasks (8 vCPU / 16GB RAM each). Generally speaking, with the schema below, is this amount of compute expected to handle this level of concurrency in such a short time? I'm asking because I'd like to figure out if my tuning is just lacking, or if this kind of heavy resource footprint is generally standard for a Zanzibar-based product. (Note: The 16GB RAM is just because of AWS Fargate's spec requirements to get the higher CPU flavor. The actual memory usage isn't that high.)
Copy code
definition team {
  relation member: user with login_team_scope
  permission is_member = member
}

definition contents {  
  relation allowed: team#member with login_team_scope
  relation enabled: contents#allowed
  
  permission member_view = enabled
}
Best Regards,
12 Views
PreviousNext
Powered by