Hey Permit team — we saw a transient outage this m...
# opal
v
Hey Permit team — we saw a transient outage this morning (~622 625 AM PST) where both our PDP sidecars (permitio/pdp-v2:latest) lost their OPAL websocket connection to wss://opal.permit.io/ws. OPA went to health: False (policy: False, data: False) on both pods, causing 502s on all permission checks. Two questions: 1. Was there a Permit.io / OPAL outage this morning around 1422 1425 UTC? Both our PDP instances failed simultaneously. 2. Why does the PDP /healthy endpoint return 200 when OPA is completely unhealthy? We observed OPA client health: False (policy: False, data: False) being logged every 5 seconds, yet /healthy kept returning 200. This prevented Kubernetes from detecting the failure and restarting the pod. Is there a config option to make /healthy reflect OPA readiness, or is this expected behavior? One pod self-recovered, the other got stuck in a DNS resolution loop for opal.permit.io ([Errno -5] Name has no usable address) for over an hour until we manually killed it.
a
Hi @Vivek Garg, I'm an AI assistant for Permit.io. I'll analyze available sources and get back to you shortly.
\Based on the available documentation, I can address your questions about the PDP health check behavior: Health Check Configuration The PDP's
/healthy
endpoint has a configurable timeout that controls how long it waits for health check responses before timing out (1) . The default
PDP_HEALTHCHECK_TIMEOUT
is 3 seconds . For more advanced policy-level health checks and callbacks, OPAL provides additional capabilities. These advanced health checks apply to Edge PDPs (self-hosted PDPs) and are not available on the Managed Cloud PDP (2) . You can read more about them as part of OPAL's health check policy and update callbacks . Kubernetes Readiness Probe When deploying the PDP in Kubernetes, the documentation recommends using the
/healthy
endpoint for readiness checks . The health check endpoint is
/healthy
, and you can use wget for health checks :
Copy code
wget --no-verbose --tries=1 --spider <http://127.0.0.1:7000/healthy> || exit 1
For Kubernetes readiness probes, there's a specific guide available in the deployment documentation . Regarding the Outage I don't have information in the available sources about specific outages or service status for the timeframe you mentioned. For real-time incident information and to report this issue, I recommend reaching out over Email . * To help our engineers investigate your specific issue further, could you please provide: 1. Which Workspace / Project / Env you are using 2. What calls you were performing (as SDK code or CURL) 3. Which PDP version you are running (you mentioned
permitio/pdp-v2:latest
, but the specific version tag would help) 4. The relevant logs from your PDP during the outage window (especially around 1422 1425 UTC) The DNS resolution loop issue (
[Errno -5] Name has no usable address
) and the health check behavior you observed warrant deeper investigation with your specific configuration and logs.
To learn more, see the sources I considered: [[1] Configurations](https://docs.permit.io/concepts/pdp/configuration) [[2] at Scale](https://docs.permit.io/how-to/deploy/overview)
The issue should be resolved now; We’re still reviewing edge cases
v
Thanks for confirming. I am adding health probes. It is odd that one pod recovered and another one did not for us.
Copy code
Why /healthy returned 200 with OPA dead                                           
                                                                                 
  The /healthy endpoint has two code paths:                                         
                                                                                  
  async def healthy():                                                              
      healthy = await self.policy_store.is_healthy()  # checks OPA transaction state
      if healthy:                                                                   
          return 200                                                                
      if self.offline_mode_enabled and await self._is_ready():                      
          return 200  # <-- THIS IS WHY                                             
      return 503
                                                                                    
  Your sidecar has PDP_ENABLE_OFFLINE_MODE=true. When offline mode is enabled, the  
  health check bypasses OPA health and returns 200 as long as the backup config was
  loaded (_is_ready()). The comment in the source even says:                        
                                       
  ▎ "Client would only report unhealthy if server → policy-store transactions       
  failed, but not if server connection gets disconnected."
                                                                                    
  So the flow was:                     
  1. OPA lost policies → is_healthy() returns False
  2. But offline mode is enabled and backup exists → /healthy returns 200 anyway
  3. K8s thinks pod is fine
                                                                                    
  The fix
                                                                                    
  Two options:                                                                      
  
  Option 1: Set OPAL_OPA_HEALTH_CHECK_POLICY_ENABLED=True on the permit-sidecar.    
  This enables a rego policy at system.opal.healthy that checks actual policy/data
  sync state. The /healthy endpoint would then reflect real OPA health.             
                                       
  Option 2: Remove PDP_ENABLE_OFFLINE_MODE=true — but this would mean the PDP       
  reports unhealthy during any transient OPAL disconnection, which might be too
  aggressive.                                                                       
                                       
  Option 3 (best): Keep offline mode but add                                        
  OPAL_OPA_HEALTH_CHECK_POLICY_ENABLED=True, and use /ready or query
  system.opal.ready as the K8s readiness probe instead of /healthy. The ready rule  
  checks that policy bundle synced at least once AND initial data sources loaded —
  which is what you actually want for readiness.