This message was deleted.
# permit-saas
s
This message was deleted.
a
Hi @Bradley Herrup, I'm an AI assistant for Permit.io. I'll analyze available sources and get back to you shortly.
That 503 comes from the OPAL server's scope policy endpoint. It means the git clone backing that scope is still being populated, so the server can't build a policy bundle yet. The OPAL server holds the request for a configurable time (default 20s) while the clone completes, then falls back to
503
with
Retry-After
. The PDP client retries five times with backoff (roughly 20-40s of coverage) and then waits until the next policy update or reconnect. A clone that takes longer than that can leave the PDP without policy. (1) Things to check and tune on the OPAL server: • ​*Clone wait time*​:
OPAL_SCOPES_POLICY_CLONE_WAIT_SECONDS
(default
20.0
) controls how long the policy request is held while the clone is in progress. Values above 55s are clamped, so the hold stays under typical load balancer idle timeouts. (1) • ​*Concurrent requests held*​:
OPAL_SCOPES_POLICY_CLONE_WAIT_MAX_INFLIGHT
(default
64
) caps how many requests one worker holds. Requests beyond the cap get an immediate
503
with
Retry-After 30
. (1) • ​*Git fetch timeout*​:
OPAL_SCOPES_GIT_FETCH_TIMEOUT
(default
120.0
) is the soft timeout for a scope clone/fetch. If the clone or fetch is slow or the remote is unreachable, the op is skipped and retried next cycle. (1) • ​*Failed sources and backoff*​: if the repo clone/fetch is failing (unreachable host, revoked cre
To learn more, see the sources I considered: [[1] OPAL Configuration Guide | OPAL](https://docs.opal.ac/getting-started/configuration)
b
buckstopcom/Staging. SDK to the PDP check PDP 0.9.16
you run the OPAL server not me
"[2026-09-30T214017Z INFO pdp_server::horizon] opal_client.policy.fetcher | INFO | Fetching policy bundle from https://opal-v2.permit. io/scopes/d7d5c7a78f9e4d33a1b53bb2cf6fc5d7/policy [2026-09-30T214017Z INFO pdp_server::horizon] opal_common.utils | WARNING | Unexpected response code 503: {'detail': 'Policy clone for scope d7d5c7a78f9e4d33a1b53bb2cf6fc5d7 is temporarily unavailable, retry shortly'} [2026-09-30T214024Z INFO pdp_server::horizon] opal_client.policy.fetcher | WARNING | Failed all attempts to fetch bundle, got error: ValueError('unexpected response code while fetching bundle: 503') [2026-09-30T214024Z INFO pdp_server::horizon] opal_client.policy_store | ERROR | OPA transaction failed, transaction id=b80abc8ce7884e4a81b9f8e83af77a3a, actions=[], error=ValueError('unexpected response code while fetching bundle: 503') [2026-09-30T214029Z INFO pdp_server::horizon] logging | INFO | 127.0.0.1:44740 - "GET /healthy HTTP/1.1" 503 [2026-09-30T214034Z WARN watchdog::service] Service 'python3' health check failed: Unhealthy status code: 503 (expected 200) (consecutive failures: 6/12)"
@Ask Inkeep please escalate this
a
Thanks for tagging me @Bradley Herrup. I'll look through relevant sources and get back to you shortly.
I can't escalate from here, but the details you shared are enough for the team to investigate. Reach out over Email with the information below so an engineer can pick this up directly: • ​*Workspace / Project / Env*​: buckstopcom / Staging (please confirm the exact project name) • ​*PDP version*​: 0.9.16 • ​*Failing scope*​:
d7d5c7a78f9e4d33a1b53bb2cf6fc5d7
• ​*Calls*​: the SDK check calls going to the PDP (share the SDK code or CURL you use, if you can) • ​*PDP logs*​: the excerpt you posted, showing the
503
from
<https://opal-v2.permit.io/scopes/d7d5c7a78f9e4d33a1b53bb2cf6fc5d7/policy>
, the failed bundle fetch, the failed OPA transaction, and the
/healthy
endpoint returning
503
• ​*Timestamps*​: the logs start at 2026-09-30T214017Z Since the PDP can't load its policy bundle,
/healthy
keeps returning
503
and the watchdog reports repeated health check failures. Checks against this PDP will not be served until the bundle loads. If you have a second PDP on the same environment, check whether it loads the policy, and include that in your email.
Feel free to tag me with additional questions. I also appreciate feedback, it helps me improve. 👍 button 👎 button
b
@Ask Inkeep the system is currently erroring, I would like support
a
Thanks for tagging me @Bradley Herrup. I'll look through relevant sources and get back to you shortly.
I can't open a support ticket or page an engineer from this chat, so please contact the team directly. Since the system is currently erroring, email help@permit.io now so an engineer can pick it up. Include these details from your thread: • ​*Workspace / Project / Env*​: buckstopcom / Staging (please confirm the exact project name) • ​*PDP version*​: 0.9.16 • ​*Failing scope*​:
d7d5c7a78f9e4d33a1b53bb2cf6fc5d7
• ​*Calls*​: the SDK check calls going to the PDP (add the SDK code or CURL you use, if you can) • ​*PDP logs*​: the excerpt you posted, starting at 2026-09-30T214017Z, showing the
503
from the scope policy endpoint, the failed bundle fetch, the failed OPA transaction, and
/healthy
returning
503
• ​*Current impact*​: the PDP can't load its policy bundle, so it reports unhealthy and checks against it aren't being served While you wait, you can check the PDP's health endpoint to confirm whether it has recovered, using the health check documented for a local Edge PDP: run
curl -i <http://localhost:7766/healthy>
. A healthy PDP returns HTTP
200
with a top-level
status
of
ok
, while an unhealthy one returns HTTP
503
with
status
set to
error
. (1) If you have another PDP running against the same environment, check whether it loads the policy, and mention the result in your email. That helps the team tell whether the problem is specific to this PDP or to the scope.
To learn more, see the sources I considered: [[1] PDP overview](https://docs.permit.io/concepts/pdp/overview)