https://linen.dev logo
Join Slack
Powered by
Hi, flagging a recurring issue with the bulk users...
# permit-saas
m
Hi, flagging a recurring issue with the bulk users endpoint. 
POST .../bulk/users
occasionally returns 
HTTP 502 Bad Gateway: "Request timed out while connecting to horizon server"
It's not constant but keeps coming back intermittently. We're running a local PDP pod in Docker. Anyone aware of what might be causing this in that setup? Thanks!
a
Hi @Michał Sz, I'm an AI assistant for Permit.io. I'll analyze available sources and get back to you shortly.
I couldn't find specific information about the 
POST .../bulk/users
endpoint returning 
HTTP 502 Bad Gateway
errors with "Request timed out while connecting to horizon server" in the available sources. However, the Horizon service is the PDP's internal Python service bridge that facilitates communication between different components within the PDP (1) . There are several timeout configurations that might be relevant: • ​*PDP_HORIZON_CLIENT_TIMEOUT*​: Default is 
60
seconds for Horizon client requests • ​*PDP_CONTROL_PLANE_TIMEOUT*​: Default is 
75
seconds for control plane requests, which includes proxied requests A similar connectivity issue was reported in a GitHub issue where resource constraints on the sidecar caused OPA connection problems (2) . That issue was resolved by increasing the resources allocated to the sidecar container . To help investigate this further, could you provide: 1. Which Workspace / Project / Env you are using 2. What calls are you performing (as SDK code or CURL) 3. Which PDP version you are running 4. Relevant logs from your PDP showing the 502 errors and any surrounding context You might also want to check if your Docker container has sufficient resources allocated, and consider adjusting the 
PDP_HORIZON_CLIENT_TIMEOUT
or 
PDP_CONTROL_PLANE_TIMEOUT
environment variables if the issue persists (1) .
To learn more, see the sources I considered: [[1] Configurations](https://docs.permit.io/concepts/pdp/configuration) [[2] Unable to push the policy data since OPA is not ready yet in sidecar pattern](https://github.com/permitio/PDP/issues/266)
m
1. and 2. It's /v2/facts/titan/staging/bulk/users 3. 0.9.9 4. Logs:
Copy code
[2026-04-13T14:12:02Z ERROR pdp_server::api::horizon_fallback] Failed to send request: error sending request for url (<http://0.0.0.0:7001/v2/facts/titan/staging/bulk/users>) (None)
    URL: <http://0.0.0.0:7001/v2/facts/titan/staging/bulk/users>
    Error details: reqwest::Error { kind: Request, url: "<http://0.0.0.0:7001/v2/facts/titan/staging/bulk/users>", source: TimedOut }
    Source error: Some(TimedOut)
2026-04-13T14:12:02.829978+0000 | uvicorn.protocols.http.httptools_impl   | INFO  | 127.0.0.1:37452 - "GET /healthy HTTP/1.1" 200
2026-04-13T14:12:03.127860+0000 | httpx._client                           | INFO  | HTTP Request: POST <https://api.permit.io/v2/facts/[sanitized]/[sanitized]/bulk/users> "HTTP/1.1 504 Gateway Time-out"
o
@Ask Inkeep
a
Thanks for tagging me @Or Weis. I'll look through relevant sources and get back to you shortly.
Based on your logs, the issue shows a timeout when the PDP's Rust server (port 7000) tries to reach the Horizon Python service (port 7001) for the 
/v2/facts/titan/staging/bulk/users
endpoint (1) . The error indicates 
reqwest::Error { kind: Request, url: "<http://0.0.0.0:7001/v2/facts/titan/staging/bulk/users>", source: TimedOut }
. This is an internal PDP communication timeout between components. The subsequent 
504 Gateway Time-out
from 
<http://api.permit.io|api.permit.io>
suggests the request is also timing out at the control plane level. A similar sidecar connectivity issue was resolved by increasing container resources . For your Docker setup, check: 1. ​*Container resource allocation*​ - Ensure your PDP pod has sufficient CPU and memory 2. ​*Timeout configurations*​ - Consider increasing these environment variables: • 
PDP_HORIZON_CLIENT_TIMEOUT
(default: 60 seconds) • 
PDP_CONTROL_PLANE_TIMEOUT
(default: 75 seconds) 3. ​*Network configuration*​ - Since you're using 
0.0.0.0:7001
, verify internal container networking is functioning properly The intermittent nature suggests resource contention or temporary network issues within your Docker environment. Try allocating more resources to your PDP container first, as this resolved a s
To learn more, see the sources I considered: [[1] Unable to push the policy data since OPA is not ready yet in sidecar pattern](https://github.com/permitio/PDP/issues/266)
t
Hey @Michał Sz did the AI suggestion work?
m
Hi, I still have that problem. I've increased both timeout variables to 120 seconds and now I'm getting following error:
Copy code
<html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
</body>
</html>
and all the time utilization is on low levels:
o
@Ask Inkeep WDYT ?
a
Thanks for tagging me @Or Weis. I'll look through relevant sources and get back to you shortly.
The 504 Gateway Time-out from 
<http://api.permit.io|api.permit.io>
after increasing your PDP timeouts to 120 seconds suggests the bottleneck is now at the control plane level, not within your local PDP (1) . Since your resource utilization is low and you've already increased 
PDP_HORIZON_CLIENT_TIMEOUT
and 
PDP_CONTROL_PLANE_TIMEOUT
to 120 seconds , the issue appears to be: 1. ​*Control plane timeout*​ - The default 
PDP_CONTROL_PLANE_TIMEOUT
is 75 seconds and covers "proxied requests for the Consistent Updates APIs" . You've set it to 120 seconds, but the control plane itself at 
<http://api.permit.io|api.permit.io>
may have its own timeout limit that's being hit. 2. ​*Bulk payload size*​ - The 
/v2/facts/.../bulk/users
endpoint may be timing out due to the size of your bulk operation. The available sources discuss bulk permission checks (2) but don't specifically document limits for bulk user operations. ​*Next steps:*​ Since this is a recurring issue with the bulk users endpoint specifically and you're running PDP v0.9.9, I'd recommend: • Reach out to Permit.io support at help@permit.io with your findings • Consider breaking your bulk users payload into smaller batches • Check if there's a proxy or load balancer between your PDP and 
<http://api.permit.io|api.permit.io>
that might have its own timeout To help engineers investigate further, please share: 1. The approximate size/count of users in your bulk requests 2. Your full PDP configuration (sanitized) 3. Whether this happens with all bulk user operations or only above a certain size threshold
To learn more, see the sources I considered: [[1] Configurations](https://docs.permit.io/concepts/pdp/configuration) [[2] Bulk Check](https://docs.permit.io/how-to/enforce-permissions/bulk-check)
o
• Consider breaking your bulk users payload into smaller batches
• Check if there’s a proxy or load balancer between your PDP and 
<http://api.permit.io|api.permit.io>
that might have its own timeout
What’s your current bulk batch size?
m
ok, I will try, right now it's 500 items per chunk
d
Hi @Michał Sz, try reducing your chunk size to 50-100 users. This should complete well within the timeout windows and reduce the load on both the Horizon service and the control plane.
👍 1
m
ok, reduced to 100 and it worked, thanks
🚀 1
3 Views
Open in Slack
PreviousNext
Powered by