Vivek Garg
03/23/2026, 4:00 PMAsk Inkeep
03/23/2026, 4:00 PMAsk Inkeep
03/23/2026, 4:00 PM/healthy endpoint has a configurable timeout that controls how long it waits for health check responses before timing out (1) . The default PDP_HEALTHCHECK_TIMEOUT is 3 seconds .
For more advanced policy-level health checks and callbacks, OPAL provides additional capabilities. These advanced health checks apply to Edge PDPs (self-hosted PDPs) and are not available on the Managed Cloud PDP (2) . You can read more about them as part of OPAL's health check policy and update callbacks .
Kubernetes Readiness Probe
When deploying the PDP in Kubernetes, the documentation recommends using the /healthy endpoint for readiness checks . The health check endpoint is /healthy, and you can use wget for health checks :
wget --no-verbose --tries=1 --spider <http://127.0.0.1:7000/healthy> || exit 1
For Kubernetes readiness probes, there's a specific guide available in the deployment documentation .
Regarding the Outage
I don't have information in the available sources about specific outages or service status for the timeframe you mentioned. For real-time incident information and to report this issue, I recommend reaching out over Email .
*
To help our engineers investigate your specific issue further, could you please provide:
1. Which Workspace / Project / Env you are using
2. What calls you were performing (as SDK code or CURL)
3. Which PDP version you are running (you mentioned permitio/pdp-v2:latest, but the specific version tag would help)
4. The relevant logs from your PDP during the outage window (especially around 1422 1425 UTC)
The DNS resolution loop issue ([Errno -5] Name has no usable address) and the health check behavior you observed warrant deeper investigation with your specific configuration and logs.Ask Inkeep
03/23/2026, 4:01 PMOr Weis
03/23/2026, 4:02 PMOr Weis
03/23/2026, 4:02 PMVivek Garg
03/23/2026, 4:06 PMWhy /healthy returned 200 with OPA dead
The /healthy endpoint has two code paths:
async def healthy():
healthy = await self.policy_store.is_healthy() # checks OPA transaction state
if healthy:
return 200
if self.offline_mode_enabled and await self._is_ready():
return 200 # <-- THIS IS WHY
return 503
Your sidecar has PDP_ENABLE_OFFLINE_MODE=true. When offline mode is enabled, the
health check bypasses OPA health and returns 200 as long as the backup config was
loaded (_is_ready()). The comment in the source even says:
▎ "Client would only report unhealthy if server → policy-store transactions
failed, but not if server connection gets disconnected."
So the flow was:
1. OPA lost policies → is_healthy() returns False
2. But offline mode is enabled and backup exists → /healthy returns 200 anyway
3. K8s thinks pod is fine
The fix
Two options:
Option 1: Set OPAL_OPA_HEALTH_CHECK_POLICY_ENABLED=True on the permit-sidecar.
This enables a rego policy at system.opal.healthy that checks actual policy/data
sync state. The /healthy endpoint would then reflect real OPA health.
Option 2: Remove PDP_ENABLE_OFFLINE_MODE=true — but this would mean the PDP
reports unhealthy during any transient OPAL disconnection, which might be too
aggressive.
Option 3 (best): Keep offline mode but add
OPAL_OPA_HEALTH_CHECK_POLICY_ENABLED=True, and use /ready or query
system.opal.ready as the K8s readiness probe instead of /healthy. The ready rule
checks that policy bundle synced at least once AND initial data sources loaded —
which is what you actually want for readiness.