Hey <Permit.io> team — we experienced a temporary ...
# permit-saas
k
Hey Permit.io team — we experienced a temporary outage today where users lost access. Timeline: • Around 17:06 UTC — PDP service started returning 502 errors. • Logs showed Service 'python3' health check failed with Connection reset by peer errors. • Restarting the PDP pod restored service immediately. Request: Could you help us understand what could have caused the PDP to fail health checks and return 502 errors, and whether there’s anything we should adjust on our side to prevent this in future?
p
Hey, thanks for reaching out ! It seems our team is a little busy right now, a human will be with you within the next hour or so. In the meantime, can I help you ?
a
Hi @Kator James, I'm an AI assistant for Permit.io. I'll analyze available sources and get back to you shortly.
Based on the 502 errors and health check failures you experienced, this appears to be related to the PDP's internal service monitoring. The PDP uses a watchdog system to monitor its internal Python service (Horizon) (1) . The "Connection reset by peer" errors suggest the Horizon service became unresponsive. The PDP has built-in health checking that monitors this service and can automatically restart it when failures are detected (2) . Here are the relevant configuration options that control this behavior: •
PDP_HORIZON_HEALTH_CHECK_INTERVAL
(default: 5 seconds) - How often health checks run •
PDP_HORIZON_HEALTH_CHECK_FAILURE_THRESHOLD
(default: 12) - Number of consecutive failures before restart •
PDP_HORIZON_HEALTH_CHECK_TIMEOUT
(default: 1 second) - Timeout for each health check •
PDP_HORIZON_STARTUP_DELAY
(default: 5 seconds) - Initial delay before health checking begins With defaults, the PDP would wait 60 seconds (12 failures × 5 second intervals) before automatically restarting the Horizon service . To help our engineers investigate further, could you please provide: 1. Which Workspace / Project / Env you are using 2. What calls you were performing (as SDK code or CURL examples) 3. Which PDP version you are running 4. The relevant PDP logs from around 17:06 UTC, including any startup logs after the restart To prevent future occurrences, you might consider: • Adjusting
PDP_HORIZON_HEALTH_CHECK_FAILURE_THRESHOLD
to a lower value for faster automatic recovery • Monitoring the
PDP_HEALTHCHECK_TIMEOUT
setting (default: 3 seconds) which controls overall PDP h
To learn more, see the sources I considered: [[1] Watchdog](https://github.com/permitio/PDP/blob/main/watchdog/README.md) [[2] Configurations](https://docs.permit.io/concepts/pdp/configuration)
o
Hi Kator, thanks for reaching out - we’d be happy to help investigate here Can you share the items the AI agent as asked for - starting with the PDP version
o
Can you also share your deployment setup • environment variables • num of replicas • cpu, mem request & limits • liveness,readiness,startup probes
t
@Kator James when you say 502, are they from specific PDP API calls? Can you provide the detailed logs from that time?
k
I'll run these questions by our platform team and provide the answers shortly
r
FYI, we had the same issues, and we downgraded to version 0.9.2 and all those issues went away. I think there might be something with 0.9.3
🤔 2
d
Hey @Ryan Barriger, thank you for sharing that. Can you please provide an example request that got a 502 response from the PDP? (API path and body, curl would be awesome)
Have you noticed it in the PDP logs? Can you please share those as well?
r
sorry, that was like 2 weeks ago. didnt keep anything, just in problem-solving mode and swapped to 0.9.2. I can tell you how I came to the conclusion: I was releasing new software with the PDP in a sidecar configuration and it was failing health checks, but my prior releases were fine. I noticed 0.9.3 had been released recently, so I downgraded, saw no more healthcheck errors, and moved on with things and plan on circling back to the latest version for verification in a couple of months