Hi all We are running into an issue with the communication b Permit #opal

Hi all, We are running into an issue with the comm...

Saliya Samarawickrama

03/12/2026, 9:36 AM

Hi all, We are running into an issue with the communication between the OPAL server and the OPAL client. We are using a Kubernetes cluster in GCP and Helm to deploy the OPAL server, client, and PostgreSQL. Below are the logs from the OPAL server and client. The issue disappears when I restart or redeploy the OPAL server. This does not happen all the time and appears to be a random event. When this happens, the OPAL client cannot start. Any thoughts? Let me know if you need more information. Thank you in advance. 🙂 OPAL server log:

Copy code

2026-03-12T09:20:44.817436+0000 | fastapi_websocket_rpc.websocket_rpc_e...| INFO  | Client disconnected - 43490 :: d78ae83fc0b4447a98a544a81f7465e0
2026-03-12T09:20:44.817686+0000 | websockets.legacy.server                | INFO  | connection closed
2026-03-12T09:20:44.889455+0000 | fastapi_websocket_rpc.websocket_rpc_e...| INFO  | Client connected
2026-03-12T09:20:44.889886+0000 | websockets.legacy.server                | INFO  | connection open
2026-03-12T09:20:44.894689+0000 | fastapi_websocket_rpc.websocket_rpc_e...| INFO  | Client disconnected - 39376 :: dee642b8daa4491a8ba991f19a80b981
2026-03-12T09:20:44.894872+0000 | websockets.legacy.server                | INFO  | connection closed
2026-03-12T09:20:46.382346+0000 | fastapi_websocket_rpc.websocket_rpc_e...| INFO  | Client connected
2026-03-12T09:20:46.382868+0000 | websockets.legacy.server                | INFO  | connection open
2026-03-12T09:20:46.388795+0000 | fastapi_websocket_rpc.websocket_rpc_e...| INFO  | Client disconnected - 37782 :: 44457ab56137481e9fb742fd39fe10b0
2026-03-12T09:20:46.389028+0000 | websockets.legacy.server                | INFO  | connection closed
2026-03-12T09:20:47.124470+0000 | fastapi_websocket_rpc.websocket_rpc_e...| INFO  | Client connected

OPAL client log:

Copy code

File "/usr/local/lib/python3.10/site-packages/fastapi_websocket_rpc/rpc_channel.py", line 433, in async_call
    await self.send(msg)
          │    │    └ RpcMessage(request=RpcRequest(method='_ping_', arguments={}, call_id='5369bcdcc34a4b4a9c32e3837ddec06a'), response=None)
          │    └ <function RpcChannel.send at 0x7c45cc2e69e0>
          └ <fastapi_websocket_rpc.rpc_channel.RpcChannel object at 0x7c45c541a890>
  File "/usr/local/lib/python3.10/site-packages/fastapi_websocket_rpc/rpc_channel.py", line 211, in send
    await self.socket.send(data)
          │    │      │    └ RpcMessage(request=RpcRequest(method='_ping_', arguments={}, call_id='5369bcdcc34a4b4a9c32e3837ddec06a'), response=None)
          │    │      └ <function JsonSerializingWebSocket.send at 0x7c45cc2e7c70>
          │    └ <fastapi_websocket_rpc.simplewebsocket.JsonSerializingWebSocket object at 0x7c45c54183d0>
          └ <fastapi_websocket_rpc.rpc_channel.RpcChannel object at 0x7c45c541a890>
  File "/usr/local/lib/python3.10/site-packages/fastapi_websocket_rpc/simplewebsocket.py", line 44, in send
    await self._websocket.send(self._serialize(msg))
          │    │          │    │    │          └ RpcMessage(request=RpcRequest(method='_ping_', arguments={}, call_id='5369bcdcc34a4b4a9c32e3837ddec06a'), response=None)
          │    │          │    │    └ <function JsonSerializingWebSocket._serialize at 0x7c45cc2e7b50>
          │    │          │    └ <fastapi_websocket_rpc.simplewebsocket.JsonSerializingWebSocket object at 0x7c45c54183d0>
          │    │          └ <function WebSocketsClientHandler.send at 0x7c45cc2fc280>
          │    └ <fastapi_websocket_rpc.websocket_rpc_client.WebSocketsClientHandler object at 0x7c45c753c280>
          └ <fastapi_websocket_rpc.simplewebsocket.JsonSerializingWebSocket object at 0x7c45c54183d0>
  File "/usr/local/lib/python3.10/site-packages/fastapi_websocket_rpc/websocket_rpc_client.py", line 147, in send
    await self._websocket.send(msg)
          │    │          │    └ '{"request": {"method": "_ping_", "arguments": {}, "call_id": "5369bcdcc34a4b4a9c32e3837ddec06a"}, "response": null}'
          │    │          └ <function Connection.send at 0x7c45c754cd30>
          │    └ <websockets.asyncio.client.ClientConnection object at 0x7c45c547c790>
          └ <fastapi_websocket_rpc.websocket_rpc_client.WebSocketsClientHandler object at 0x7c45c753c280>
  File "/usr/local/lib/python3.10/site-packages/websockets/asyncio/connection.py", line 476, in send
    async with self.send_context():
               │    └ <function Connection.send_context at 0x7c45c754d360>
               └ <websockets.asyncio.client.ClientConnection object at 0x7c45c547c790>
  File "/usr/local/lib/python3.10/contextlib.py", line 199, in __aenter__
    return await anext(self.gen)
                       │    └ <async_generator object Connection.send_context at 0x7c45c75f5fc0>
                       └ <contextlib._AsyncGeneratorContextManager object at 0x7c45c541aaa0>
  File "/usr/local/lib/python3.10/site-packages/websockets/asyncio/connection.py", line 957, in send_context
    raise self.protocol.close_exc from original_exc
          │    │        │              └ None
          │    │        └ <property object at 0x7c45c7e9f830>
          │    └ <websockets.client.ClientProtocol object at 0x7c45c547c6d0>
          └ <websockets.asyncio.client.ClientConnection object at 0x7c45c547c790>

websockets.exceptions.ConnectionClosedOK: received 1000 (OK); then sent 1000 (OK)
2026-03-12T09:21:48.511830+0000 | opal_client.policy_store.opa_client     |WARNING | OPA client health: False (policy: False, data: False)
2026-03-12T09:21:48.512277+0000 | uvicorn.protocols.http.httptools_impl   | INFO  | 10.246.16.157:55704 - "GET /healthcheck HTTP/1.1" 503
2026-03-12T09:21:52.514744+0000 | opal_client.policy_store.opa_client     |WARNING | OPA client health: False (policy: False, data: False)
2026-03-12T09:21:52.515261+0000 | uvicorn.protocols.http.httptools_impl   | INFO  | 10.246.16.157:57878 - "GET /healthcheck HTTP/1.1" 503
2026-03-12T09:22:04.668932+0000 | opal_client.policy_store.opa_client     |WARNING | OPA client health: False (policy: False, data: False)
2026-03-12T09:22:04.669366+0000 | uvicorn.protocols.http.httptools_impl   | INFO  | 10.246.16.157:54528 - "GET /healthcheck HTTP/1.1" 503
2026-03-12T09:22:04.672225+0000 | opal_client.policy_store.opa_client     |WARNING | OPA client health: False (policy: False, data: False)
2026-03-12T09:22:04.672794+0000 | uvicorn.protocols.http.httptools_impl   | INFO  | 10.246.16.157:54536 - "GET /healthcheck HTTP/1.1" 503
2026-03-12T09:22:19.672953+0000 | opal_client.policy_store.opa_client     |WARNING | OPA client health: False (policy: False, data: False)
2026-03-12T09:22:19.673463+0000 | uvicorn.protocols.http.httptools_impl   | INFO  | 10.246.16.157:44232 - "GET /healthcheck HTTP/1.1" 503
2026-03-12T09:22:34.672830+0000 | opal_client.policy_store.opa_client     |WARNING | OPA client health: False (policy: False, data: False)
2026-03-12T09:22:34.673382+0000 | uvicorn.protocols.http.httptools_impl   | INFO  | 10.246.16.157:35314 - "GET /healthcheck HTTP/1.1" 503

Ask Inkeep

03/12/2026, 9:36 AM

Hi @Saliya Samarawickrama, I'm an AI assistant for Permit.io. I'll analyze available sources and get back to you shortly.

Ask Inkeep

03/12/2026, 9:36 AM

Based on your logs, the OPAL client is experiencing WebSocket connection closures (

ConnectionClosedOK: received 1000 (OK)

) during ping operations, followed by health check failures. The server logs show rapid connect/disconnect cycles. This intermittent connection issue between OPAL server and client can have several causes: *Network/Infrastructure Considerations:* • WebSocket connection stability in Kubernetes can be affected by load balancers, ingress controllers, or network policies that don't properly handle long-lived connections • GCP-specific networking configurations or firewall rules may be intermittently dropping connections • Resource constraints (CPU/memory) on either server or client pods could cause connection handling issues *Configuration to Review:* Check your broadcast channel configuration. OPAL uses Redis or PostgreSQL for pub/sub communication. (1) Ensure your

OPAL_BROADCAST_URI

is properly configured and the backing service (Redis/PostgreSQL) is stable. Review timeout settings: •

PDP_CONTROL_PLANE_TIMEOUT

(default: 75 seconds) controls timeout for control plane requests • Connection keepalive settings may need adjustment for your environment *Troubleshooting Steps:* To help investigate further, please provide: 1. Which Workspace / Project / Env you are using 2. What calls are the OPAL client and server performing (configuration snippets or CURL examples) 3. Which PDP version you are running 4. Complete logs from both OPAL server and client during a failure event, including any earlier connection establishment logs *Additional Information Needed:* • Your Helm chart configuration for OPAL server and client • Redis/PostgreSQL configuration and health status • Kubernetes service and ingress configurations • Any recent infrastructure or configuration changes The fact that restarting the OPAL server resolves the issue temporarily suggests a state or resource accumulation problem rather than a fundamental configuration error.

Ask Inkeep

03/12/2026, 9:36 AM

To learn more, see the sources I considered: [[1] Configurations](https://docs.permit.io/concepts/pdp/configuration)

Or Weis

03/12/2026, 9:53 AM

AI gave a good answer. As the AI indicates this is likely a GCP / network configuration or resource allocation issue.

👍 1

Saliya Samarawickrama

03/12/2026, 10:09 AM

Hi Or, I'm checking on this. thanks.

David Rennalls

03/20/2026, 10:27 PM

It sounds like the same issue that I raised a bug for recently, see https://github.com/permitio/opal/issues/883

permit support automation2

03/20/2026, 10:27 PM

Hey, thanks for reaching out ! It seems our team is a little busy right now, a human will be with you within the next hour or so. In the meantime, can I help you ?

2 Views

Open in Slack

Previous Next