We're seeing intermittent "Failed to send request to OPA" errors from the PDP sidecar (permitio/pdp-v2:latest) in our Kubernetes clusters. Looking for guidance on root cause and mitigation. Error from sidecar logs:

[ERROR pdp_server::api::authz::allowed] Failed to send request to OPA: 
error sending request for url (<http://localhost:8181/v1/data/permit/root>)

Error from our Go service (Permit Go SDK v1.2.7):

erroneous http response from PDP for Permit.Check()
ErrorCode: UnexpectedError, ErrorType: general_error
Message: An unexpected error happened.
response_body: {"detail":"Failed to send request to OPA"}

What we observe: - OPA subprocess inside the PDP becomes transiently unreachable at localhost:8181 - The PDP container stays running (health checks pass, K8s doesn't restart it) - Happens across multiple services and environments (both dev and prod) - Intermittent — fails for a few minutes, then self-recovers - Not correlated with traffic volume (happened on both high-traffic and low-traffic services) - Not resource-related (container uses 60m CPU / 130Mi of its 1 CPU / 512Mi limits) Our setup: - Image:

permitio/pdp-v2:latest

- Deployed as K8s sidecar container in each service pod -

PDP_ENABLE_OFFLINE_MODE=true

-

OPAL_OPA_HEALTH_CHECK_POLICY_ENABLED=true

- 1 CPU / 512Mi resource limits - Policy graph: ~260Mi memory footprint (prod environment) Impact: When OPA is unreachable, every Permit.Check() returns 502, and our middleware returns 500 to users (we fail-closed). In prod this caused a ~50 minute outage on one service. Questions: 1. Is this a known issue with the PDP's internal OPA management? 2. Are there recommended PDP configuration settings to improve OPA stability (e.g., UVICORN_NUM_WORKERS, OPA thread settings)? 3. Should we pin to a specific PDP version instead of :latest? 4. Is there a way to configure the PDP to automatically restart OPA if it becomes unresponsive?

Hi @Vivek Garg, I'm an AI assistant for Permit.io. I'll analyze available sources and get back to you shortly.

Hey, thanks for reaching out ! It seems our team is a little busy right now, a human will be with you within the next hour or so. In the meantime, can I help you ?

Based on your description, this appears to be an OPA connectivity issue within the PDP container. Let me address your questions: ​*1. Known Issue & Root Cause*​ The error pattern you're seeing—OPA becoming transiently unreachable at localhost:8181—has been reported in similar sidecar deployments (1) . The PDP uses OPAL under the hood to manage OPA, and connection failures can occur during policy updates or data synchronization . ​*2. Configuration Recommendations*​ Several PDP configurations can improve stability: ​*Resource & Worker Settings:*​ •

UVICORN_NUM_WORKERS

controls parallel workers for the PDP server (default: 1) (2) • Increase

PDP_OPA_CLIENT_QUERY_TIMEOUT

from the default 1 second if you're seeing timeouts • Set

PDP_OPA_CLIENT_FAILURE_THRESHOLD

(default: 0.1) and

PDP_OPA_CLIENT_FAILURE_THRESHOLD_INTERVAL

(default: 60 seconds) to tune failure detection ​*Health Check Tuning:*​ • Adjust

PDP_HEALTHCHECK_TIMEOUT

(default: 3 seconds) • Configure

OPAL_INLINE_OPA_LOG_FORMAT

to get better visibility into OPA issues (options:

none

,

minimal

,

http

,

full

) ​*For your specific setup with 1 CPU / 512Mi:*​

PDP_OPA_CLIENT_QUERY_TIMEOUT=2
PDP_HEALTHCHECK_TIMEOUT=5
OPAL_INLINE_OPA_LOG_FORMAT=http

​*3. Version Pinning*​ Yes, you should pin to a specific PDP version instead of

:latest

. The PDP follows Semantic Versioning, so minor version updates (e.g., 0.9.3 to 0.9.4) won't introduce breaking changes . ​*4. OPA Restart Capability*​ The PDP includes built-in health monitoring for OPA with automatic restart capabilities (added in PDP v0.9.0) : •

PDP_HORIZON_HEALTH_CHECK_TIMEOUT

(default: 1 second) - timeout for individual health checks •

PDP_HORIZON_HEALTH_CHECK_INTERVAL

(default: 5 seconds) - interval between checks •

PDP_HORIZON_HEALTH_CHECK_FAILURE_THRESHOLD

(default: 12) - consecutive failures before restart •

PDP_HORIZON_STARTUP_DELAY

(default: 5 seconds) - delay before monitoring begins •

PDP_HORIZON_RESTART_INTERVAL

(default: 1 second) - interval between restart attempts •

PDP_HORIZON_TERMINATION_TIMEOUT

(default: 30 seconds) - graceful termination timeout These settings allow the PDP to automatically recover from OPA failures. ​*Additional Recommendations:*​ For production sidecar deployments, consider (3) (4) : • Deploy the PDP in the same VPC as your application to minimize network latency (3) • Disable debug logging in production with

PDP_DEBUG=False

to enhance performance • Use local PDP deployment for tight latency requirements ​*Next Steps - Information Needed:*​ To help our engineers investigate further, please provide: 1. Which Workspace / Project / Env you are using 2. What calls you are performing (as SDK code or CURL) 3. Which PDP version you are running (exact version, not

:latest

) 4. Relevant logs from your PDP during the failure period, especially around the OPA connection errors This will help identify whether this is a configuration issue, a version-specific bug, or a resource contention problem.

AI gave a good answer and good follow-up questions.

Here's the info requested: 1. Workspace / Project / Env: - Project ID:

f45912930a77419cb908cf7ee649b46d

- Prod Environment ID:

ad9f8981852143bba60c0be0cbe448c5

- Dev Environment ID:

2bf09a0624414c208eaf033b8e8e368c

2. SDK + calls: - Go SDK:

<http://github.com/permitio/permit-golang|github.com/permitio/permit-golang> v1.2.7

- Call:

permit.Client.Check(user, "view", resource{type:"project", key:projectID, tenant:orgID})

- Equivalent curl:

curl -X POST <http://localhost:7000/allowed> \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <PDP_API_KEY>" \
  -d '{"user":{"key":"<mailto:user@example.com|user@example.com>"},"action":"view","resource":{"type":"project","key":"<projectId>","tenant":"<orgId>"}}'

3. PDP version: - Tag used:

permitio/pdp-v2:latest

(unpinned) - Deployed image SHA:

sha256:9ca15b030d9548ca68686821b0db34d3ca97af7b56d717977a5c3a3558322211

(pulled ~Apr 21, does not match current 0.9.10/0.9.11/0.9.12 tags — likely an intermediate

:latest

build) - Current

:latest

has since moved to

0.9.12

(

sha256:9f6cdcb3...

) 4. Logs during failure: Prod incident (2026-04-27, 1455 1545 UTC, docservice namespace): - 232K

"Failed to send request to OPA"

errors over 50 minutes - Sidecar logs only show healthy

GET /health -> 200

— NO error logs from the sidecar itself despite OPA being unreachable - PDP container never restarted (0 restarts,

ready=true

throughout) - K8s readiness probes intermittently returned 503 but never crossed failure threshold Error from Go SDK:

erroneous http response from PDP for Permit.Check()
ErrorCode: UnexpectedError, ErrorType: general_error
response_body: {"detail":"Failed to send request to OPA"}

Dev incident (2026-04-28, 1624 1627 UTC, tenantservice namespace): - Same error, shorter duration (~3 min, self-recovered) - Sidecar DID log the error:

[ERROR pdp_server::api::authz::allowed] Failed to send request to OPA: error sending request for url (<http://localhost:8181/v1/data/permit/root>)

- Duration per failed check: ~1000-1142ms (matches default

PDP_OPA_CLIENT_QUERY_TIMEOUT

of 1s) 5. Our concerns: We've applied the recommended tuning (

UVICORN_NUM_WORKERS=4

,

PDP_OPA_CLIENT_QUERY_TIMEOUT=3

,

PDP_HEALTHCHECK_TIMEOUT=5

) and are pinning to

0.9.12

, but we're not confident these changes will prevent a recurrence. Our doubts: -

UVICORN_NUM_WORKERS

applies to Horizon (Python), not the PDP Rust server. The actual

/allowed

requests are served by the Rust binary. So more Horizon workers shouldn't affect OPA request handling. - The 50-minute duration is unexplained. A brief sync pause would cause seconds of failures, not 50 minutes. The Horizon auto-restart should have recovered OPA within 60 seconds. Something kept OPA in a bad state that the auto-restart couldn't fix. - We could not reproduce locally. We ran 13,650 requests + 10 OPAL sync rounds (user create/role assign/delete) with

--cpus=1 --memory=512m

constraints matching our K8s limits. Zero errors. We also tried with the prod environment ID (larger policy graph, ~260Mi). Still zero errors. What would help us: - Guidance on how to reproduce this locally — are there specific sync patterns or data volumes that trigger OPA unresponsiveness? - Insight into why the Horizon auto-restart didn't recover OPA during the 50-minute prod outage - Whether the deployed SHA (

9ca15b03...

) has a known OPA stability issue that's fixed in 0.9.12 Key questions: - The sidecar logged NO errors during the prod incident (only health checks). Is there a log level setting needed to capture OPA connection failures from the Rust PDP server? - Is there a known issue where OPA becomes unresponsive but

/healthy

still returns 200?

Hey, thanks for reaching out ! It seems our team is a little busy right now, a human will be with you within the next hour or so. In the meantime, can I help you ?

OPA might simply be unresponsive due to resource consumption. I'd start with 1. upgrading to actual latest 2. reviewing your APM / cloud console for the resource utilization of the relevant containers 3. consider allocating more resources, or applying elasticity