Hi! We've had some issues with our PDP deployment ...
# permit-saas
r
Hi! We've had some issues with our PDP deployment on the last couple of weeks. We are currently using PDP V0.9.3. In summary, the permission checks stops working after a random period of time (around once a week). Below I describe two instances of this issue: Last Friday, one of our users was receiving an empty array on the permissions bulkCheck. We made sure this user had the proper roles assigned in Permit. To resolve the issue, we took 2 actions. Unsure which one was actually effective. • Restarted the PDP deployment • Rotated the user roles. i.e. we changed its role to something else and then back to the correct one. I have the logs around the time this issue happened and can share in private if needed. Today, we also saw issues with permission checks. The following error caught my eye in the logs:
Copy code
[25/Aug/2025:14:28:16] [2025-08-25T14:28:16Z ERROR pdp_server::api::authz::user_permissions] Failed to send request to OPA: Failed to send request to OPA: error sending request for url (<http://localhost:8181/v1/data/permit/user_permissions>)
[25/Aug/2025:14:28:16] [2025-08-25T14:28:16Z ERROR pdp_server::api::authz::allowed_bulk] Failed to send request to OPA: Failed to send request to OPA: error sending request for url (<http://localhost:8181/v1/data/permit/bulk>)
We restarted the PDP deployment and the issue was fixed. Could you help us understand what's causing this issue and how to fix it? Thanks in advance.
a
Hi @Rodrigo Mayer, I'm an AI assistant for Permit.io. I'll analyze available sources and get back to you shortly.
I understand you're experiencing intermittent issues with your PDP where permission checks fail after running for about a week. The errors you're seeing indicate that the PDP server is unable to communicate with the internal OPA (Open Policy Agent) component (1) . Based on the error messages showing failures to reach
<http://localhost:8181>
, this appears to be an issue with the OPA service that runs inside the PDP container . The PDP uses OPA internally to evaluate policies, and when OPA becomes unresponsive or crashes, permission checks will fail . Here are some configuration changes that might help stabilize your deployment: 1. ​*Adjust OPA timeout settings*​ (2) : •
PDP_OPA_CLIENT_QUERY_TIMEOUT
- Default is 1 second, you might want to increase this •
PDP_OPA_CLIENT_FAILURE_THRESHOLD
- Default is 0.1 (10% failure rate triggers health check failure) •
PDP_OPA_CLIENT_FAILURE_THRESHOLD_INTERVAL
- Default is 60 seconds 2. ​*Monitor OPA health more closely*​ : • Enable
PDP_OPA_DECISION_LOG_CONSOLE=True
to see more detailed OPA logs • Set
OPAL_INLINE_OPA_LOG_FORMAT
to
full
or
http
for better debugging 3. ​*Consider resource limits*​ - The PDP might be running out of memory over time, causing OPA to crash (3) . To help our engineers investigate this issue further, could you please provide: 1. Which Workspace / Project / Env you are using 2. What specific SDK calls are you performing (as SDK code or CURL examples) 3. The exact PDP version (you mentioned v0.9.3 - can you confirm with
docker images
or your deployment manifest?) 4. The relevant PDP logs from when the issue occurs, including: • Logs showing the OPA connection errors • Any memory/CPU usage metrics from your monitoring • Logs from PDP startup showing the configuration being applied The logs you mentioned having from last Friday would be particularly helpful for diagnosis.
To learn more, see the sources I considered: [[1] Permit's PDP (Policy Decision Point)](https://docs.permit.io/concepts/pdp/overview) [[2] Configurations](https://docs.permit.io/concepts/pdp/configuration) [[3] How to properly scale out OPAL servers for data updates directly from Kafka](https://github.com/permitio/opal/discussions/671)
a
Hey @Rodrigo Mayer , We will be right back to you
👍 1
t
Hey Rodrigo, we'll need to get the logs. It would also be good to get any performance metrics you have around the time of the failures. Most of the time
Failed to send request to OPA: Failed to send request to OPA: error sending request for url
is a because of lack of CPU or memory resources, but won't be able to tell until we get into the logs.
r
Ok, will share in private @Thomas