Slackbot
09/30/2026, 9:44 PMAsk Inkeep
09/30/2026, 9:44 PMAsk Inkeep
09/30/2026, 9:44 PM503 with Retry-After. The PDP client retries five times with backoff (roughly 20-40s of coverage) and then waits until the next policy update or reconnect. A clone that takes longer than that can leave the PDP without policy. (1)
Things to check and tune on the OPAL server:
• *Clone wait time*: OPAL_SCOPES_POLICY_CLONE_WAIT_SECONDS (default 20.0) controls how long the policy request is held while the clone is in progress. Values above 55s are clamped, so the hold stays under typical load balancer idle timeouts. (1)
• *Concurrent requests held*: OPAL_SCOPES_POLICY_CLONE_WAIT_MAX_INFLIGHT (default 64) caps how many requests one worker holds. Requests beyond the cap get an immediate 503 with Retry-After 30. (1)
• *Git fetch timeout*: OPAL_SCOPES_GIT_FETCH_TIMEOUT (default 120.0) is the soft timeout for a scope clone/fetch. If the clone or fetch is slow or the remote is unreachable, the op is skipped and retried next cycle. (1)
• *Failed sources and backoff*: if the repo clone/fetch is failing (unreachable host, revoked creAsk Inkeep
09/30/2026, 9:44 PMBradley Herrup
09/30/2026, 9:54 PMBradley Herrup
09/30/2026, 9:56 PMBradley Herrup
09/30/2026, 10:05 PMBradley Herrup
09/30/2026, 10:05 PMAsk Inkeep
09/30/2026, 10:05 PMAsk Inkeep
09/30/2026, 10:06 PMd7d5c7a78f9e4d33a1b53bb2cf6fc5d7
• *Calls*: the SDK check calls going to the PDP (share the SDK code or CURL you use, if you can)
• *PDP logs*: the excerpt you posted, showing the 503 from <https://opal-v2.permit.io/scopes/d7d5c7a78f9e4d33a1b53bb2cf6fc5d7/policy>, the failed bundle fetch, the failed OPA transaction, and the /healthy endpoint returning 503
• *Timestamps*: the logs start at 2026-09-30T214017Z
Since the PDP can't load its policy bundle, /healthy keeps returning 503 and the watchdog reports repeated health check failures. Checks against this PDP will not be served until the bundle loads. If you have a second PDP on the same environment, check whether it loads the policy, and include that in your email.Ask Inkeep
09/30/2026, 10:06 PMBradley Herrup
09/30/2026, 10:06 PMAsk Inkeep
09/30/2026, 10:06 PMAsk Inkeep
09/30/2026, 10:06 PMd7d5c7a78f9e4d33a1b53bb2cf6fc5d7
• *Calls*: the SDK check calls going to the PDP (add the SDK code or CURL you use, if you can)
• *PDP logs*: the excerpt you posted, starting at 2026-09-30T214017Z, showing the 503 from the scope policy endpoint, the failed bundle fetch, the failed OPA transaction, and /healthy returning 503
• *Current impact*: the PDP can't load its policy bundle, so it reports unhealthy and checks against it aren't being served
While you wait, you can check the PDP's health endpoint to confirm whether it has recovered, using the health check documented for a local Edge PDP: run curl -i <http://localhost:7766/healthy>. A healthy PDP returns HTTP 200 with a top-level status of ok, while an unhealthy one returns HTTP 503 with status set to error. (1)
If you have another PDP running against the same environment, check whether it loads the policy, and mention the result in your email. That helps the team tell whether the problem is specific to this PDP or to the scope.Ask Inkeep
09/30/2026, 10:06 PM