https://linen.dev logo
Join Slack
Powered by
hi. I'm getting an issue (between my Permit ECS ta...
# permit-saas
m
hi. I'm getting an issue (between my Permit ECS task (permitio/pdp-v2:0.9.2) and the Permit cloud). I'm attaching the logs.
a
Hi @Marius Baci, I'm an AI assistant for Permit.io. I'll analyze available sources and get back to you shortly.
* Hi! I'd be happy to help troubleshoot the connectivity issue between your ECS PDP task and the Permit cloud. Please share the logs you mentioned so I can take a closer look. In the meantime, here are a few common causes for PDP-to-cloud connectivity issues: ​*Common causes:*​ • ​*DNS resolution issues*​ – The PDP needs to resolve 
<http://api.permit.io|api.permit.io>
. In containerized environments (especially ECS) , DNS configuration can sometimes cause failures. (1) • ​*Network/firewall rules*​ – Ensure your ECS task has outbound HTTPS access (port 443) to 
<http://api.permit.io|api.permit.io>
. • ​*Invalid or expired API key*​ – Double-check your 
PDP_API_KEY
environment variable is correctly set. (2) ​*For ECS deployments, verify your task definition includes:*​ • Container port 
7000
open (required for SDK connections) • The 
PDP_API_KEY
environment variable set correctly • Image URI: 
permitio/pdp-v2:latest
(or your specific version like 
0.9.2
) * To help our engineers investigate further, please provide: 1. Which ​*Workspace / Project / Environment*​ you're using 2. What ​*calls*​ you're performing (SDK code or CURL) 3. The ​*PDP version*​ you're running (you mentioned 
0.9.2
) 4. The ​*relevant logs*​ from your PDP (please paste them here)
To learn more, see the sources I considered: [[1] Docker image errors out - cannot get config from control plane](https://github.com/permitio/PDP/issues/251) [[2] Step-by-step guide - Deploying PDP using AWS-ECS-Fargate](https://docs.permit.io/how-to/deploy/cloud-hosts/aws-ecs-fargate)
m
Copy code
WARNING | Opa connection error: ClientConnectorError(ConnectionKey(host='localhost', port=8181, is_ssl=False, ssl=True, proxy=None, proxy_auth=None, proxy_headers_hash=None), ConnectionRefusedError(111, "Connect call failed ('127.0.0.1', 8181)"))
[34mopal_client.data.updater [0m|[31m[1mERROR | Failed to save data update to policy-store: RetryError[<Future at 0xffff949e5c00 state=finished raised ClientConnectorError>]
[1mTraceback (most recent call last):[0m
File "/usr/local/lib/python3.10/site-packages/aiohttp/connector.py", line 1248, in _wrap_create_connection
sock = await aiohappyeyeballs.start_connection(
File "/usr/local/lib/python3.10/site-packages/aiohappyeyeballs/impl.py", line 122, in start_connection
raise first_exception
File "/usr/local/lib/python3.10/site-packages/aiohappyeyeballs/impl.py", line 73, in start_connection
sock = await _connect_sock(
File "/usr/local/lib/python3.10/site-packages/aiohappyeyeballs/impl.py", line 208, in _connect_sock
await loop.sock_connect(sock, address)
File "uvloop/loop.pyx", line 2633, in sock_connect
await fut
File "uvloop/loop.pyx", line 1108, in uvloop.loop.Loop._sock_connect_cb
raise OSError(err, 'Connect call failed %s' % (address,))
[31m[1mConnectionRefusedError[0m:[1m [Errno 111] Connect call failed ('127.0.0.1', 8181)[0m
[1mThe above exception was the direct cause of the following exception:[0m
[33m[1mTraceback (most recent call last):[0m
File "/usr/local/lib/python3.10/site-packages/tenacity/asyncio/__init__.py", line 114, in __call__
result = await fn(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/opal_client/policy_store/opa_client.py", line 766, in set_policy_data
async with session.put(
File "/usr/local/lib/python3.10/site-packages/aiohttp/client.py", line 1482, in __aenter__
self._resp: _RetType = await self._coro
File "/usr/local/lib/python3.10/site-packages/aiohttp/client.py", line 770, in _request
resp = await handler(req)
File "/usr/local/lib/python3.10/site-packages/aiohttp/client.py", line 725, in _connect_and_send_request
conn = await self._connector.connect(
File "/usr/local/lib/python3.10/site-packages/aiohttp/connector.py", line 622, in connect
proto = await self._create_connection(req, traces, timeout)
File "/usr/local/lib/python3.10/site-packages/aiohttp/connector.py", line 1189, in _create_connection
_, proto = await self._create_direct_connection(req, traces, timeout)
File "/usr/local/lib/python3.10/site-packages/aiohttp/connector.py", line 1561, in _create_direct_connection
raise last_exc
File "/usr/local/lib/python3.10/site-packages/aiohttp/connector.py", line 1530, in _create_direct_connection
transp, proto = await self._wrap_create_connection(
File "/usr/local/lib/python3.10/site-packages/aiohttp/connector.py", line 1271, in _wrap_create_connection
raise client_error(req.connection_key, exc) from exc
[31m[1maiohttp.client_exceptions.ClientConnectorError[0m:[1m Cannot connect to host localhost:8181 ssl:default [Connect call failed ('127.0.0.1', 8181)][0m
[1mThe above exception was the direct cause of the following exception:[0m
[33m[1mTraceback (most recent call last):[0m
File "/usr/local/lib/python3.10/site-packages/opal_client/data/updater.py", line 547, in _fetch_and_save_data
await self._store_fetched_data(entry, result, store_transaction)
File "/usr/local/lib/python3.10/site-packages/opal_client/data/updater.py", line 645, in _store_fetched_data
await self._set_policy_data(
File "/usr/local/lib/python3.10/site-packages/opal_client/data/updater.py", line 706, in _set_policy_data
await tx.set_policy_data(data, path=path)
File "/usr/local/lib/python3.10/site-packages/tenacity/asyncio/__init__.py", line 189, in async_wrapped
return await copy(fn, *args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/tenacity/asyncio/__init__.py", line 111, in __call__
do = await self.iter(retry_state=retry_state)
File "/usr/local/lib/python3.10/site-packages/tenacity/asyncio/__init__.py", line 153, in iter
result = await action(retry_state)
File "/usr/local/lib/python3.10/site-packages/tenacity/_utils.py", line 99, in inner
return call(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/tenacity/__init__.py", line 419, in exc_check
raise retry_exc from fut.exception()
[31m[1mtenacity.RetryError[0m:[1m RetryError[<Future at 0xffff949e5c00 state=finished raised ClientConnectorError>][0m
[32m2025-12-10T12:43:32.205560+0000[0m | [34mopal_client...base_policy_store_client [0m|[31m[1mERROR | OPA transaction failed, transaction id=c2792dc539474d51860e8d76b6dc3d67, actions=['set_policy_data'], error=Failed to save data to policy store: RetryError[<Future at 0xffff949e5c00 state=finished raised ClientConnectorError>][0m
I'm only getting this error sometimes. Initially when I deploy the container in aws it works fine, but after some time when I'm requesting data from Permit, it doesn't work anymore, and I keep getting the above error.
a
Hey @Marius Baci, Thanks for reaching out! @Ask Inkeep
a
Thanks for tagging me @Alon Boshi. I'll look through relevant sources and get back to you shortly.
* Thanks for sharing the logs! The error shows the OPAL client cannot connect to OPA on 
localhost:8181
— this is an internal connection issue within the PDP container where the inline OPA engine becomes unreachable after running for some time. ​*What's happening:*​ The PDP runs OPA inline (inside the same container) , and the OPAL client needs to push policy/data updates to OPA at 
localhost:8181
. Your logs show 
ConnectionRefusedError [Errno 111]
which means OPA stopped responding or crashed. (1) ​*Possible causes for intermittent failures:*​ 1. ​*OPA process crashing*​ – The inline OPA binary may be terminating unexpectedly due to memory pressure or other resource constraints in your ECS task. 2. ​*Resource limits*​ – If your ECS task has tight memory/CPU limits, OPA might get OOM-killed after running for a while. 3. ​*Architecture mismatch*​ – If you're running on ARM64 (Graviton) , there's a known issue with the OPA binary in certain PDP versions. (2) ​*Troubleshooting steps:*​ 1. ​*Check ECS task resource allocation*​ – Ensure sufficient memory (recommend at least 512MB-1GB for the PDP container) . 2. ​*Enable more verbose logging*​ to see if OPA is crashing:
Copy code
OPAL_INLINE_OPA_LOG_FORMAT=http
    OPAL_LOG_LEVEL=DEBUG
(3) , (4) 3. ​*Check if you're on ARM64*​ – If using Graviton instances, try pinning to an AMD64 image or check if a newer PDP version fixes the ARM64 OPA binary issue. (2) 4. ​*Monitor container health*​ – Look at ECS CloudWatch metrics for memory spikes or container restarts. * To help our engineers investigate further, please provide: 1. Which ​*Workspace / Project / Environment*​ you're using 2. What ​*calls*​ you're performing (SDK code or CURL) 3. Confirm the ​*PDP version*​ (
0.9.2
) and ​*ECS instance architecture*​ (x86 or ARM64/Graviton) 4. Any additional logs showing what happens before the OPA connection errors start
To learn more, see the sources I considered: [[1] Unable to push the policy data since OPA is not ready yet in sidecar pattern](https://github.com/permitio/PDP/issues/266) [[2] [Bug] permitio/pdp-v2:latest fails on ARM64 (linux/arm64) due to OPA binary linker mismatch](https://github.com/permitio/PDP/issues/289) [[3] Troubleshooting](https://docs.opal.ac/getting-started/running-opal/troubleshooting) [[4] OPAL Configuration Guide | OPAL](https://docs.opal.ac/getting-started/configuration)
Thanks for the feedback, glad I was helpful.
m
hi @Alon Boshi. we are using ARM64 (as all our AWS infrastructure is on ARM64). so, I upgraded the Docker image we are using to 
permitio/pdp-v2:0.9.9
as this version seemed to have a fix for that (https://github.com/permitio/PDP/commit/b42fae2cf18a13ae1263b22daba703082967ce9c from https://github.com/permitio/PDP/compare/0.9.8...0.9.9). but unfortunately the issue still appears.
you also mentioned some 'resource constrains', but this shouldn't be the case (I guess), as we are using this setup in AWS (below), and the official Docker image for Permit Operating system/Architecture Linux/ARM64 CPU | Memory 1 vCPU | 2 GiB Launch type Fargate
can you please let me know what else can I try ?
thank you 🙂
o
Please include your PDP logs that correlate to the error. It’s likely a health-check configuration
e
Hi @Marius Baci, In addition to what @Or Weis mentioned, could you please also share the ECS task definition JSON for this deployment? I’d like to review it as well, especially around the health check and container configuration. Thanks!
@Marius Baci Also, based on the logs you’ve shared, this issue is not related to connectivity between the PDP and the Permit cloud. Instead, the logs indicate that the PDP’s internal OPAL client is temporarily unable to communicate with the local OPA process inside the same task:
Copy code
Cannot connect to host localhost:8181
ConnectionRefusedError ('127.0.0.1', 8181)
This means that, at that point in time, the OPA engine was not listening on port 8181 - for example, due to a restart or crash. Can you confirm whether the ECS tasks are still running when this happens, and whether you see any restarts or unhealthy events around that time?
m
hi. thank you all for the replies. sorry I didn't got the chance to get back to you. the issue is still present. whenever I start a new the ECS, it works fine for a while, then it starts getting these errors. I attached here some logs and the ECS task definition.
log-events-viewer-result.csvplatform-test-brickguard-pdp-task-revision23.json
@Or Weis & @Eli Moshkovich ☝️. thank you 🙂
o
Hi @Marius Baci , thanks for adding the logs. we'll review and circle back to you
👍 1
e
Hi @Marius Baci, thanks again for sharing the details. I noticed that the ECS task is running a non-official PDP image: 
<http://746127402929.dkr.ecr.eu-west-1.amazonaws.com/platform-test-brickguard-pdp-repository:platform-test-brickguard-pdp-0.9.9|746127402929.dkr.ecr.eu-west-1.amazonaws.com/platform-test-brickguard-pdp-repository:platform-test-brickguard-pdp-0.9.9>
Just to clarify - we can only provide support for the official PDP images. Could you please try running the latest official version instead? 
permitio/pdp-v2:latest
This will help us confirm whether the issue is coming from the custom build. One more thing I noticed: In your task definition the container CPU reservation is set to 0, which can sometimes lead to CPU starvation on ECS Fargate. That can cause OPA to become temporarily unresponsive. Let us know once you’ve tested with the official image, and we can continue from there.
m
hi @Eli Moshkovich. sorry for the confusion created. it is the official image (we just rename it in the pipeline before pushing it to ECR in AWS). here it's the code:
Copy code
- name: Pull, tag, and push Docker BrickGuard-PDP image
        env:
          ECR_REPOSITORY_URL: ${{ steps.get-ecr-url.outputs.ecr-url }}
          WORKSPACE: ${{ inputs.workspace }}
          SOURCE_IMAGE: permitio/pdp-v2:0.9.9
        run: |
          set -eo pipefail
          VERSION=$(echo $SOURCE_IMAGE | cut -d':' -f2)
          IMAGE_TAG=${WORKSPACE}-${{ vars.ACCOUNT }}-brickguard-pdp-${VERSION}
          ECR_IMAGE=${ECR_REPOSITORY_URL}:${IMAGE_TAG}
          ECR_REPO_NAME=$(echo $ECR_REPOSITORY_URL | cut -d'/' -f2)
          IMAGE_EXISTS=$(aws ecr describe-images \
            --repository-name $ECR_REPO_NAME \
            --image-ids imageTag=${IMAGE_TAG}\
            --region ${{ vars.AWS_REGION }} \
            --query 'imageDetails' --output text 2>/dev/null || true)
          if [[ -n "$IMAGE_EXISTS" ]]; then
            echo "Image: $ECR_IMAGE already exists. Skipping push."
          else
            echo "Pulling image: $SOURCE_IMAGE"
            docker pull --platform linux/arm64 $SOURCE_IMAGE
            echo "Tagging image: $SOURCE_IMAGE as $ECR_IMAGE"
            docker tag $SOURCE_IMAGE $ECR_IMAGE
            echo "Pushing image: $ECR_IMAGE to ECR"
            docker push $ECR_IMAGE
          fi
also, where do you see the 'container CPU reservation'. in the container definitions ?
e
Hi @Marius Baci, yes - it’s in the 
containerDefinitions
section. In your current task definition you have:
Copy code
"containerDefinitions": [
  {
    "cpu": 0,
    ...
  }
]
When 
cpu: 0
, the container has no reserved CPU allocation, which can cause resource starvation under load. This could explain why OPA becomes unresponsive intermittently. Try updating the container definition to explicitly reserve resources:
Copy code
"containerDefinitions": [
  {
    "cpu": 512,
    "memory": 1024,
    ...
  }
]
Let us know once you try this, and we’ll continue from there.
m
as I only have the Permit container for this ECS Task
I thought that if the Task has already 1024, and it's a single container (Permit) inside, that won't be a problem.
but I'll try to update and will let you know after that.
thank you.
e
You're right - with a single container, it should inherit all the task-level resources. Worth trying anyway. While you're testing that, could you also: 1. Add 
PDP_DEBUG=True
to the environment variables for more detailed logs 2. Check CloudWatch metrics for Memory Utilization on this task - if OPA is crashing due to OOM, you'd see memory spike to ~100% right before the errors Let us know how it goes.
👍 1
m
sure, I'll let you know on Monday. thank you. have a nice weekend 🙂
🙌 1
2 Views
Open in Slack
PreviousNext
Powered by