https://linen.dev logo
Join Slack
Powered by
hi all. the PDP which is running in ECS behind an ...
# permit-saas
w
hi all. the PDP which is running in ECS behind an ALB typically responds very quickly. but we are seeing that there are intermittent spikes in latency and traced them to 5xx errors on the PDP with these corresponding logs:
Copy code
Failed to send request to OPA: Failed to send request to OPA: error sending request for url (<http://localhost:8181/v1/data/permit/root>)
also see a couple of these at the same time we see the above
Copy code
Health check failed: opa: Failed to connect to OPA … <http://localhost:8181/health>
these tasks are well provisioned, not seeing it anywhere close to being starved for resources. any idea what the cause might be? we see the spikes (which correlate to spikes in traffic from about 5 req/s to about 100 req/s) and then the latency returns to baseline after the burst in load passes. are there limits to what OPA can handle or best practice for how to handle these bursts? i found this similar thread above but not sure if there was any resolution
a
Hi @wesley.wu, I'm an AI assistant for Permit.io. I'll analyze available sources and get back to you shortly.
I can't pin down the root cause of those OPA connection errors from the sources I have. They don't describe why the PDP's OPA process would refuse connections at 
localhost:8181
under a burst from about 5 to about 100 req/s, and they don't state a throughput limit for OPA. What the sources do support: • The Edge PDP bundles OPA, the OPAL client, and an API server in one container, so a failure to reach OPA on 
localhost
is internal to the PDP container, not an ALB issue.(1) • OPA evaluation has a query timeout set by 
PDP_OPA_CLIENT_QUERY_TIMEOUT
, in seconds, with a default of 
1
. The bulk check documentation describes raising it for bulk checks that take longer to evaluate, and says to give the PDP more CPU if timeouts continue.(2)
Copy code
PDP_OPA_CLIENT_QUERY_TIMEOUT=10
(2) That setting is documented for timeouts on slow evaluations. I can't confirm it applies to "error sending request" or health check connection failures, so treat it as something to test, not a confirmed fix. • Edge PDP caching is off by default. When enabled, it can serve repeated checks without re-evaluating policy, and it supports an in-memory store for a single PDP or Redis for sharing across instances. Cached decisions can be stale until the TTL expires. See Cache configuration.(1) • For high availability and scale, the documented pattern is several Edge PDPs behind a load balancer, which spreads burst load across instances.(1) • If you send multiple checks per request handler, 
permit.bulkCheck()
sends them to the PDP in one request instead of several.(2) The GitHub issue you found (ARM64) is about the OPA binary failing to execute on 
linux/arm64
because of a linker mismatch. It was closed, with a fix included in a subsequent PDP release.(3) That produces a different failure (OPA never starting), so it's unlikely to explain intermittent errors that recover on their own. If your tasks run on ARM64 Fargate, confirm you're on a PDP version that includes that fix. To help our engineers investigate, please share: 1. Which Workspace / Project / Env you're using 2. The calls you're performing (as SDK code or cURL) 3. The PDP version (image tag) you're running 4. The relevant PDP logs around a spike, including any OPA output and the timestamps of the 
localhost:8181
errors If you want, I can also give a general answer based on broader knowledge, but it wouldn't be backed by these sources.
To learn more, see the sources I considered: [[1] PDP overview](https://docs.permit.io/concepts/pdp/overview) [[2] Check many permissions with bulk check](https://docs.permit.io/how-to/enforce-permissions/bulk-check) [[3] [Bug] permitio/pdp-v2:latest fails on ARM64 (linux/arm64) due to OPA binary linker mismatch](https://github.com/permitio/PDP/issues/289)
w
please direct me to human help
e
Hi @wesley.wu, thanks for the detail. This looks like OPA getting briefly saturated during the burst, not crashing. The PDP gives each query to its local OPA a short timeout (
PDP_OPA_CLIENT_QUERY_TIMEOUT
, 1s by default). When traffic jumps from ~5 to ~100 req/s, requests queue up in OPA. Any that wait longer than the timeout come back as 
Failed to send request to OPA
and turn into 5xx, and OPA's health endpoint is slow for the same reason. Once the burst passes, the queue drains and latency goes back to normal, which matches what you're seeing. Things to do: 1. Raise the timeout so bursts give slower responses instead of errors: 
PDP_OPA_CLIENT_QUERY_TIMEOUT=3
(or 5). 2. Provision for the peak, not the average. ECS autoscaling reacts too slowly for short bursts, so run enough tasks (at least 2) to handle ~100 req/s on their own. Average CPU over 1 minute can hide a short spike where OPA is pinned. Look at per-container CPU and throttling over the burst window. 3. Check your ALB/ECS health check settings. The PDP's health check fails while OPA is busy. If the health check thresholds are tight, the target group can pull tasks out in the middle of a burst, which pushes more load onto the rest. Loosen the unhealthy threshold and interval so a few seconds of slowness doesn't remove a task. 4. Check what's being called. A bulk check or a user-permissions call does many evaluations in one request, so 100 req/s of those costs far more than 100 single checks. To confirm, could you send: • the PDP version and the task's CPU/memory • the full error line, including any "caused by" part (a timeout and a connection refused point to different causes) • whether you see OPA restart logs during the spikes • which endpoints/SDK methods make up the burst
w
• 0.9.17, two tasks at .5 vCPU | 2 GiB each, max four tasks based on average cpu/memory • no 'caused by' logs • no OPA restart logs • we are not making bulk checks
i don't think we want to extend the timeout because we need this thing to be fast. do you think more CPU would resolve this problem?
e
Thanks @wesley.wu, that explains it. 0.5 vCPU is below what we recommend for the PDP. One PDP container runs several processes (the PDP server, OPA, and its sync components), and they all share that half core. Fargate enforces the limit strictly, so when traffic jumps, OPA gets throttled and requests queue past the 1s timeout. Average CPU looks healthy because the bursts are much shorter than the 1-minute average. That's also why autoscaling on average CPU doesn't kick in. Keeping the 1s timeout makes sense once the PDP has enough headroom. I'd suggest: 1. At least 1 vCPU per task. Our ECS guide uses that as the starting point. Since you go from 5 to 100 req/s, I'd go straight to 2 vCPU / 4 GiB so you have room during the burst. 2. Size the minimum number of tasks for your peak traffic, since autoscaling takes minutes and your bursts take seconds. Keep autoscaling for longer increases in traffic. 3. Keep your ALB health check settings forgiving, so a task isn't marked unhealthy during a burst. That would push its traffic onto the other task. If you still see spikes at 1–2 vCPU, let us know which endpoint/SDK method the burst is made of and we'll dig in further.
w
ok will give that a try, thank you
✅ 1
Open in Slack
PreviousNext
Powered by