Hi team, following up on the ticket with @Or Weis, we're sharing PDP logs and config for the persistent communication issues. Our Setup: • PDP Type: Local PDP (most basic setup) • Deployment: Helm chart

pdp

version :`0.0.5` • Configuration ( Basic):

pdp:
  ApiKey: "[OUR_API_KEY]"
  port: 7000
  logs_forwarder:
    enabled: true
    debug_mode: false

The Core Problem: Our PDP is experiencing repeated failures to communicate with your control plane, resulting in HTTP 503 Service Unavailable errors and timeouts when trying to reach

<https://opal-relay.api.permit.io/v2/pdp/callback>

. Key Log Evidence: We've attached two log files from the incident period `[from Nov 17 at 12:30 PM to Nov 18 at 11:30 AM (IST)]`: 1.

logs_by_time.csv

- Shows the sequence of events. 2.

logs_by_group.csv

- Groups the repeated messages, showing the high frequency. The most repeating errors look like this:

- server replied with HTTP 503 Service Unavailable
- Timeout while fetching url: <https://opal-relay.api.permit.io/v2/pdp/callback?retry=0>

This is causing a complete breakdown in our authorization flows. We'd appreciate your help in reviewing these logs and our configuration to understand why our local PDP cannot maintain a stable connection to your servers. Thanks in advance for your help. Cyclops security team.

Hey @Matan Benjio Thanks for sharing the details. A few clarifications: • As noted in your logs:

“This does not affect the PDP’s operational state or data updates.”

So, the local PDP does not rely on Opal Relay to function, and these 503 errors should not affect your authorization flows. Few questions to better understand the root cause for the authorization flow issue: • Is this issue still happening since Nov 18? Can it be reproduced reliably? • Could you share: ◦ Exact problem observed ◦ Environment details (workspace, project, environment) ◦ All relevant PDP logs around the time of the issue ◦ Which user authorization failed? ( can you provide the permit.check you did? ) ◦ Did you get a false negative or an exception/error? Once we have this info, we can analyze the incident and suggest next steps. Thanks

Thanks for your quick response. After further investigation, we can confirm: 1. Issue Status: The PDP synchronization issue has not occurred since November 18th and cannot be reproduced currently. 2. Our Current Flow: ◦ On user login: We call GetAssignedRoles + GetRole via direct Permit API ◦ For API protection: We use Permit.check via the local PDP 3. Exact Problem Description: ◦ Users received correct permissions from the direct API (GetAssignedRoles and GetRole) ◦ But PDP.check returned false due to outdated PDP cache ◦ Error Type: False negative - users were authorized but received denial ◦ Duration: Persistent for over 6 hours, resolving only after manual intervention via Permit UI triggered PDP synchronization 4. Technical Questions: ◦ PDP Update Frequency: Documentation says hourly - is this configurable? ◦ Cache TTL: Can we configure cache time-to-live? How? ◦ Force Update: Is there a way to force PDP update when user permissions change? ◦ Regarding Enforcer.GetUserPermissions: Could you clarify its implementation? Specifically, we need to know if it uses the locally cached PDP data or performs a live query to the Permit API. Answers to these would help us better manage authorization and minimize customer impact. Thanks

I’ll let @Alon Boshi and the team address further investigation here. And will just address the questions

• PDP Update Frequency: Documentation says hourly - is this configurable?

Where does that say that? The PDP updates on the fly within less than a 100ms usually

• Cache TTL: Can we configure cache time-to-live? How?

There’s no cache TTL - data updates on the fly (Assuming the PDP is connected to the control plane)

• Force Update: Is there a way to force PDP update when user permissions change?

As explained before; It’s automatic and on the fly. Restarting a PDP forces it to download policy and data from scratch

• Regarding Enforcer.GetUserPermissions: Could you clarify its implementation? Specifically, we need to know if it uses the locally cached PDP data or performs a live query to the Permit API.

GetUserPermission like all other PDP APIs that don’t mention proxying to the API explicitly is always local

Thanks, we tried to use Enforcer.GetUserPermissions and got null in the Result and the status is WaitingForActivation, Why? and what we can do about it?

Please: • include which SDK version you’re using • which PDP version you’re using • which workspace / project / environment • what is your input to getUserPermissions • the PDP log capturing this call I’d also recommend 1. Making sure the PDP is ready before sending it a query (via health-checks) to avoid any races 2. using the API directly to remove potential noise