Hi, I've read the performance evaluation article f...
# general
h
Hi, I've read the performance evaluation article from the team at https://docs.permit.io/concepts/pdp/cloud-pdp-benchmarks/ and I also noticed that the PDP code has a fixed value of 10 for pool_max_idle_per_host. My goal is to achieve 100-200 concurrent users (CCU) per PDP, but when I try to increase it to 100 CCU, the PDP starts returning the error: "ERROR cannot send to opa," even though direct calls to the opa don't cause errors. I suspect this is due to pool_max_idle_per_host. Can you guide me on how to adjust these parameters to achieve that goal?
@Ask Inkeep
a
Thanks for tagging me @Huy. I'll look through relevant sources and get back to you shortly.
The "ERROR cannot send to opa" you're experiencing is likely related to connection pooling limits when scaling to 100+ concurrent users. While the specific
pool_max_idle_per_host
parameter you mentioned isn't directly configurable through the documented PDP environment variables, there are several configuration options that can help you achieve 100-200 CCU per PDP. (1) Based on the Cloud PDP benchmarks, the PDP has been tested to handle approximately 54-56 req/s with 10 concurrent requests across different policy models (RBAC and ReBAC) . (2) To scale to your target of 100-200 CCU, consider these approaches: ​*1. Adjust Worker Count*​ The
PDP_PORT
configuration (added in PDP v0.9.0) works alongside the worker configuration. For earlier versions, use
UVICORN_PORT
. (1) While the exact worker count configuration isn't explicitly documented in the available sources, scaling workers can help distribute the load. ​*2. Enable Caching*​ The PDP includes a configurable caching mechanism that can significantly improve performance by reducing redundant policy evaluations. This cache supports both in-memory store for single instances and Redis for distributed caching across multiple PDP instances. (3) This is particularly helpful for frequently evaluated permissions at high concurrency. ​*3. Deploy Multiple PDP Instances*​ For truly high concurrency, consider deploying multiple PDP instances behind a load balancer. The documentation shows this as a "Centralized Edge PDP" deployment model where multiple PDP instances sit behind a load balancer. ​*4. Monitor Health and Performance*​ Use the health check endpoints to monitor PDP performance: • Liveness probe:
/health
• Readiness probe:
/healthy
• Startup probe:
/ready
(4) To help investigate this issue further, could you provide: 1. Which Workspace / Project / Env you are using 2. What calls you are performing (as SDK code or CURL) 3. Which PDP version you are running 4. Relevant logs from your PDP showing the "ERROR cannot send to opa" message This information will help our engineers investigate the connection pooling issue you're experiencing.
To learn more, see the sources I considered: [[1] Configurations](https://docs.permit.io/concepts/pdp/configuration) [[2] Cloud PDP Benchmarks](https://docs.permit.io/concepts/pdp/cloud-pdp-benchmarks) [[3] Permit's PDP (Policy Decision Point)](https://docs.permit.io/concepts/pdp/overview) [[4] Deploying PDP using raw Kubernetes YAMLs](https://docs.permit.io/how-to/deploy/cloud-hosts/kubernetes-raw)
h
I would like to ask if the value of pool_max_idle_per_host should be modified to allow for more flexible changes. Thank you.
o
1. please ask questions in #C02RTML2WNN 2. notice these are the benchmarks for the cloud-pdp, not the edge pdp (they use a completely different architecture) 3. the answer the AI gave is still good for the edge PDP
also seems you're confusing concurrent users and concurrent requests. 100 active users don't usually send a 100 concurrent requests.
h
thanks