Hi there!, i've been experiencing a ton of restart...
# opal
e
Hi there!, i've been experiencing a ton of restarts on my PDP on k8 using the helm chart with the following error, i have confirmed the pod security policy isn't blocking egress.
Copy code
2025-10-20T18:21:45.449442+0000 | fastapi_websocket_rpc.websocket_rpc_c...| INFO  | RPC Connection failed - [Errno -5] Name has no usable address
2025-10-20T18:21:46.391887+0000 | opal_client.policy_store.opa_client     |WARNING | OPA client health: False (policy: False, data: False)
2025-10-20T18:21:46.392485+0000 | uvicorn.protocols.http.httptools_impl   | INFO  | 127.0.0.1:45740 - "GET /healthy HTTP/1.1" 503
2025-10-20T18:21:47.785723+0000 | opal_client.policy_store.opa_client     |WARNING | OPA client health: False (policy: False, data: False)
2025-10-20T18:21:47.786254+0000 | uvicorn.protocols.http.httptools_impl   | INFO  | 127.0.0.1:45744 - "GET /healthy HTTP/1.1" 503
[2025-10-20T18:21:47Z INFO  pdp_server::api::health::handlers] Health check failed: horizon: Horizon returned status 503 Service Unavailable
2025-10-20T18:21:47.786688+0000 | opal_client.engine.logger               | INFO  | Received request.    GET /health
2025-10-20T18:21:47.787123+0000 | opal_client.engine.logger               | INFO  | Sent response.       GET /health -> 200
2025-10-20T18:21:51.391099+0000 | opal_client.policy_store.opa_client     |WARNING | OPA client health: False (policy: False, data: False)
2025-10-20T18:21:51.391510+0000 | uvicorn.protocols.http.httptools_impl   | INFO  | 127.0.0.1:45740 - "GET /healthy HTTP/1.1" 503
2025-10-20T18:21:52.716743+0000 | fastapi_websocket_rpc.websocket_rpc_c...| INFO  | Trying server - <wss://opal.permit.io/ws>
2025-10-20T18:21:52.736808+0000 | fastapi_websocket_rpc.websocket_rpc_c...| INFO  | RPC Connection failed - [Errno -5] Name has no usable address
2025-10-20T18:21:53.071268+0000 | fastapi_websocket_rpc.websocket_rpc_c...| INFO  | Trying server - <wss://opal.permit.io/ws>
2025-10-20T18:21:53.097005+0000 | fastapi_websocket_rpc.websocket_rpc_c...| INFO  | RPC Connection failed - [Errno -5] Name has no usable address
2025-10-20T18:21:54.619022+0000 | fastapi_websocket_rpc.websocket_rpc_c...| INFO  | Trying server - <wss://opal.permit.io/ws>
2025-10-20T18:21:54.640207+0000 | fastapi_websocket_rpc.websocket_rpc_c...| INFO  | RPC Connection failed - [Errno -5] Name has no usable address
2025-10-20T18:21:55.982693+0000 | opal_client.policy_store.opa_client     |WARNING | OPA client health: False (policy: False, data: False)
2025-10-20T18:21:55.983169+0000 | uvicorn.protocols.http.httptools_impl   | INFO  | 127.0.0.1:51108 - "GET /healthy HTTP/1.1" 503
[2025-10-20T18:21:55Z INFO  pdp_server::api::health::handlers] Health check failed: horizon: Horizon returned status 503 Service Unavailable
a
Hi @Elijah Bassey, I'm an AI assistant for Permit.io. I'll analyze available sources and get back to you shortly.
p
Hey, thanks for reaching out ! It seems our team is a little busy right now, a human will be with you within the next hour or so. In the meantime, can I help you ?
a
The error
[Errno -5] Name has no usable address
indicates a DNS resolution issue where your PDP cannot resolve
<http://opal.permit.io|opal.permit.io>
(1) . The PDP is trying to connect to the OPAL server for real-time policy and data updates but failing due to network/DNS issues (2) . Your logs show the OPAL client within the PDP is unable to establish a WebSocket connection to
<wss://opal.permit.io/ws>
, which is causing the health checks to fail and triggering pod restarts (1) . To help our engineers investigate this issue further, please provide: 1. Which Workspace / Project / Env you are using 2. What calls you are performing (as SDK code or CURL) 3. Which PDP version you are running 4. Any additional PDP logs, particularly from the initial startup For immediate troubleshooting, verify: • DNS resolution is working in your pod (test with
nslookup <http://opal.permit.io|opal.permit.io>
from within the pod) • Your cluster's network policies allow outbound HTTPS/WSS traffic to
<http://opal.permit.io|opal.permit.io>
• Check if you have any proxy settings that might interfere with WebSocket connections A similar issue was resolved by increasing resources allocated to the PDP container (3) , so you might also want to check if your PDP has sufficient CPU/memory resources allocated in your Helm values.
To learn more, see the sources I considered: [[1] Permit's PDP (Policy Decision Point)](https://docs.permit.io/concepts/pdp/overview) [[2] How does Permit.io work?](https://docs.permit.io/overview/how-does-it-work) [[3] Unable to push the policy data since OPA is not ready yet in sidecar pattern](https://github.com/permitio/PDP/issues/266)
t
Hey @Elijah Bassey, were is the PDP deployed?
e
AKS
The resource allocated is also sufficient for the pod (>1gb)
t
the logs show Name has no usable address, which usually means DNS or egress. Can you run a netshoot to dig opal.permit.io? Our services are up and active
d
Hey @Elijah Bassey, what is the PDP version you are using? (image tag)
e
image tag is
latest
šŸ™Œ 1
d
Can you please share the image sha hash?
e
quick question, been trying to dig into the
helm
manifest but couldn't find anything online; tried the raw
yaml
and it seems to work on a separate cluster when i specified the
PDP_ENVIRONMENT
env. what would be the equivalent on the helm?
d
Can you please share the full configuration of your PDP?
e
tried
pdp.Environment
but that didn't work either
d
There is no config named
PDP_ENVIRONMENT
on the PDP, where did you get that from? šŸ‘€
You've probably meant the pdp.Environment config on the helm values https://github.com/permitio/PDP/blob/main/charts/pdp/values.yaml
e
deployment.yaml
this works
Copy code
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: permitio-pdp
    run: permitio-pdp
  name: permitio-pdp
  namespace: <namespace>
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      run: permitio-pdp
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: permitio-pdp
        run: permitio-pdp
    spec:
      containers:
        - env:
            - name: PDP_API_KEY
              valueFrom:
                secretKeyRef:
                  key: PDP_API_KEY
                  name: permitio-pdp-secret
            - name: PDP_ENVIRONMENT
              value: "development"
            - name: PDP_DEBUG
              value: "true"
          image: permitio/pdp-v2:latest
          imagePullPolicy: Always
          name: permitio-pdp
          resources:
            limits:
              memory: 1Gi
            requests:
              cpu: 256m
              memory: 512Mi
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /health
              port: 7000
              scheme: HTTP
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 5
          readinessProbe:
            failureThreshold: 10
            httpGet:
              path: /healthy
              port: 7000
              scheme: HTTP
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 5
          startupProbe:
            failureThreshold: 12
            httpGet:
              path: /ready
              port: 7000
              scheme: HTTP
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 5
            initialDelaySeconds: 30
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
but this fails
helm install pdp pdp/pdp --set pdp.ApiKey=permit_key_<rest_of_the_keys> --set pdp.Environment=development --namespace <namespace>
d
This settings allows you to configure custom configs via Environment Variables for your PDP https://docs.permit.io/concepts/pdp/configuration/
I see the confusion now. The PDP must get an Environment-level API Key, starting the PDP connected to that environment on Permit. The
PDP_ENVIRONMENT
config is not used and can be removed šŸ™‚
Seem like your Helm setup is incorrect, using a custom
deployment.yaml
instead of our official one. Let me refer to @Eli Moshkovich to help you setup your Helm as the recommended setup 🌟
e
Hi @Elijah Bassey Thank you for reaching out. Please look on our values.yaml: https://github.com/permitio/PDP/blob/main/charts/pdp/values.yaml And make sure your helm command are aligned with the configuration here
e
Thanks a bunch, i just found the issue. My WAF on the cluster this was deployed hasn't cleared permit.io as part of the allowed domains
e
Great, glad it's solved and works for you now @Elijah Bassey šŸ™‚
šŸ™Œ 1
e
Oh the
deployment.yaml
i shared was from the raw (yaml) i tried out
e
Yes, I see. No worries šŸ™‚