https://linen.dev logo
Join Slack
Powered by
Hey team :wave:, running into a weird issue with t...
# permit-saas
g
Hey team đź‘‹, running into a weird issue with the PDP pod (permitio/pdp-v2:latest) in our k8s cluster. Setup: backend pod + PDP pod in the same namespace, kind of a sidecar setup. Problem: On deployment, sometimes the PDP pod goes into a restart loop. Logs say: ``Service python 3 health check failed`` and the container shuts down. After a few restarts it comes back healthy. Also happens randomly once in a while (like once a week), no clear pattern. Questions: 1. Is this normal during PDP startup? 2. Could our sidecar-ish setup be causing it? 3. Any recommended deployment tweaks for stability?
permit io
a
Hi @Ganesh Jadhav, I'm an AI assistant for Permit.io. I'll analyze available sources and get back to you shortly.
The restart loop you're experiencing with the PDP pod is likely related to the internal Horizon service health checks. The PDP has an internal Python service called Horizon that facilitates communication between different components (1) . Answering Your Questions: ​*1. Is this normal during PDP startup?*​ Some initial delay is expected. The PDP has a configurable startup delay before health checking begins - 
PDP_HORIZON_STARTUP_DELAY
defaults to 5 seconds . The Horizon service health check has a failure threshold of 12 consecutive failures by default before attempting a restart . ​*2. Could your sidecar setup be causing it?*​ Yes, there's a known issue with sidecar deployments where the PDP tries to push policy data before OPA is ready (2) . This was resolved by increasing resources on the sidecar container . The PDP runs both a Horizon service (Python) and OPA process, which need adequate resources to initialize properly. ​*3. Recommended deployment tweaks for stability:*​ • ​*Increase resource allocation*​: Based on the resolved issue, increasing CPU/memory resources helps • ​*Adjust health check timings*​ (1) : • `PDP_HORIZON_HEALTH_CHECK_TIMEOUT`: Default 1 second • `PDP_HORIZON_HEALTH_CHECK_INTERVAL`: Default 5 seconds • `PDP_HORIZON_HEALTH_CHECK_FAILURE_THRESHOLD`: Default 12 failures • `PDP_HORIZON_STARTUP_DELAY`: Default 5 seconds • ​*Configure proper probes*​ (3) : • Use 
/ready
for startup probe • Use 
/healthy
for readiness probe • Use 
/health
for liveness probe For debugging this issue, please provide: 1. Which Workspace / Project / Env you are using 2. What calls are you performing (as SDK code or CURL) 3. Which PDP version you are running (you mentioned latest, but specific version helps) 4. Relevant logs from your PDP, especially around the restart events This will help our engineers investigate the specific cause of your restart loops.
To learn more, see the sources I considered: [[1] Configurations](https://docs.permit.io/concepts/pdp/configuration) [[2] Unable to push the policy data since OPA is not ready yet in sidecar pattern](https://github.com/permitio/PDP/issues/266) [[3] Deploying PDP using raw Kubernetes YAMLs](https://docs.permit.io/how-to/deploy/cloud-hosts/kubernetes-raw)
a
Hey @Ganesh Jadhav, saw your message and will be back to you. Meanwhile, does the AI's answer make sense?
g
Hi @Alon Boshi, I don't know if that makes sense, Because those values(resources) should be default, I don’t set specific ones, they came from permit helm chart.
a
I’ll confirm with the team and provide an update shortly.
g
Okay!, sharing the resources for reference:
o
It is most likely due to low resources, can you share the usage metrics of the container during this startup ? Have you tried raising the resources requests ( especially the CPU one )?
Also it seems like the PDP fails with it’s initial startup of getting the remote config from the control plane ( our servers ). Can you verify the PDP can “interact” with outside servers ?
🙌 1
g
Hey @Omer Zuarets, I'll share you the usage metrics in a while. Haven't tried raising the resources yet, will try this!
Hi guys, here are the observations: • It is most likely due to low resources, can you share the usage metrics of the container during this startup ? => 
NAME                            CPU(cores)   MEMORY(bytes)
permitio-pdp-696f64dc64-qq8hq      86m            114Mi
• Have you tried raising the resources requests ( especially the CPU one )? => 
No, as it is default and is coming from pdp helm chart
• Also it seems like the PDP fails with it’s initial startup of getting the remote config from the control plane ( our servers ). Can you verify the PDP can “interact” with outside servers ? => 
Yes, regular ping to <http://google.com|google.com> works without any issues
a
Hey @Ganesh Jadhav, we”ll circle back to you
r
Hey @Ganesh Jadhav - can you share with us the full description of the pod (
kubectl describe …
) and the PDP logs as a file ?
o
Hey @Ganesh Jadhav , as mentioned can you please try to increase the resources and let us know if it helps ? Regarding the metrics can you extract something similar to a timeseries of the usage ( like grafana/datadog/etc)
g
Here are files, and regarding timeseries we don’t have any, as we don’t have any monitoring tool 1. How we can check direct conenction to your “servers” from pdp pod 2. How to turn debug logging. Also another observations: Randomly 5 hours ago the pod restarted
Copy code
backend         permitio-pdp-696f64dc64-qq8hq               1/1     Running   1 (5h11m ago)   5h12m
LOGS:
Copy code
[2025-09-11T03:43:38Z INFO  pdp_server::api::health::handlers] Health check failed: horizon: Failed to connect to Horizon: error sending request for url (<http://0.0.0.0:7001/healthy>), opa: Failed to connect to OPA: error sending request for url (<http://localhost:8181/health>)
[2025-09-11T03:43:40Z INFO  pdp_server::api::health::handlers] Health check failed: horizon: Failed to connect to Horizon: error sending request for url (<http://0.0.0.0:7001/healthy>), opa: Failed to connect to OPA: error sending request for url (<http://localhost:8181/health>)
[2025-09-11T03:43:48Z INFO  pdp_server::api::health::handlers] Health check failed: horizon: Failed to connect to Horizon: error sending request for url (<http://0.0.0.0:7001/healthy>), opa: Failed to connect to OPA: error sending request for url (<http://localhost:8181/health>)
[2025-09-11T03:43:50Z INFO  pdp_server::api::health::handlers] Health check failed: horizon: Failed to connect to Horizon: error sending request for url (<http://0.0.0.0:7001/healthy>), opa: Failed to connect to OPA: error sending request for url (<http://localhost:8181/health>)
[2025-09-11T03:43:58Z INFO  pdp_server::api::health::handlers] Health check failed: horizon: Failed to connect to Horizon: error sending request for url (<http://0.0.0.0:7001/healthy>), opa: Failed to connect to OPA: error sending request for url (<http://localhost:8181/health>)
[2025-09-11T03:44:00Z INFO  pdp_server::api::health::handlers] Health check failed: horizon: Failed to connect to Horizon: error sending request for url (<http://0.0.0.0:7001/healthy>), opa: Failed to connect to OPA: error sending request for url (<http://localhost:8181/health>)
[2025-09-11T03:44:00Z INFO  pdp_server::api::health::handlers] Health check failed: horizon: Failed to connect to Horizon: error sending request for url (<http://0.0.0.0:7001/healthy>), opa: Failed to connect to OPA: error sending request for url (<http://localhost:8181/health>)
pdp_pod.txtcurrent-logs.txtpdp_logs.txt
a
Hey @Ganesh Jadhav , thanks for sharing the logs! We’ll be back to you
o
Have you tried increasing the resources?
g
Since pod are not killed due to “OOM error”, and currently there plenty of room. I don't think increasing the resources would work! Still, I can start running some deployment sin in “real-time” check resource usage of pdp pod. If I notice that cpu goes above 90% and pod is restarted then will increase resources, if not then I doubt that it is resource issue.
o
I was mainly talking about CPU, pods aren’t killed due CPU issues. To be honest, it’s pretty hard to help when you decide not to do what we ask for, even if it’s just for troubleshooting purposes Increasing the requests for the CPU to 1 would easily allow us to see if there is a CPU issue here which is what I suspect is happening here
Does this also happen when you run the PDP locally on your computer via the docker without any limits ?
g
No, when I run PDP locally via docker I never faced such issue
o
This also makes my point that it’s either a resources issue or a networking issue in your cluster. Given the fact that outside network to google succeeded from your cluster I would say it’s more likely a resources issue
g
No worries, I'll try increasing the resource and let know
🙏 1
Open in Slack
PreviousNext
Powered by