We just hit an issue where out of nowhere our PDP ...
# permit-saas
s
We just hit an issue where out of nowhere our PDP stopped returning all the permissions. We had to restart the PDP which got us back into a good place. Seems like the cache got totally messed up. We are using permitio/pdp-v2:0.8.1. Any ideas what could have caused this? Never have had issues with this before.
a
Hi @Stephen Morrison, I'm an AI assistant for Permit.io. I'll analyze available sources and get back to you shortly.
p
Hey, thanks for reaching out ! It seems our team is a little busy right now, a human will be with you within the next hour or so. In the meantime, can I help you ?
a
There are several potential causes for cache-related issues with the PDP. The PDP includes a configurable caching mechanism that can use either in-memory store for single instance deployments or Redis for distributed caching across multiple PDP instances (1) . The cache applies to individual queries within bulk requests and is shared between
/allowed
and
/allowed/bulk
endpoints . While caching improves performance, it may lead to temporarily inaccurate results if permissions change during the cache TTL period . Some configuration factors that could affect cache behavior include: •
PDP_CACHE_STORE
(default:
none
) - The cache storage type (2) •
PDP_CACHE_TTL
(default:
3600
) - Time-to-live in seconds for cached items •
PDP_CACHE_MEMORY_CAPACITY
(default:
128
) - Memory capacity in MiB for in-memory cache To help our engineers investigate this further, could you provide: 1. Which Workspace / Project / Env you are using 2. What calls you were performing (as SDK code or CURL) 3. Which PDP version you are running (you mentioned 0.8.1) 4. Relevant logs from your PDP around the time the issue occurred This information will help identify what might have caused the cache to become corrupted and prevent similar issues in the future.
To learn more, see the sources I considered: [[1] Permit's PDP (Policy Decision Point)](https://docs.permit.io/concepts/pdp/overview) [[2] Configurations](https://docs.permit.io/concepts/pdp/configuration)
s
Using alliehealth production. Calling
getUserPermissions
on the JS SDK
o
Hi Stephen, we'd need more details here. Are you saying you are using the built-in cache feature? If so which type of cache are you using? Can you provide logs, and the details the AI highlighted (workspace, environment, ...)
s
Yeah so it's alliehealth on the production env
Just realized we aren't specifying caching at all so maybe we don't have a cache at all?
we're on 0.8.1 - looks like those cache variables didn't exist for that
o
The cache is not turned on by default. Are you maybe setting up your own cache layer? Next we should look into logs in the PDP, perhaps those can indicate an error.
s
we aren't
I'll send you the output from the direct SDK call before and after the restart
unfortunately we don't have the logs from the time
sent the SDK output via DM
o
Thanks I'll share it with the team internally. If you can reproduce the logs, or reproduce the bug, that would be of course very valuable for the investigation. Regardless I'd recommend upgrading to the recent 0.9.4 version, tons of improvements
s
ok got it. Appreciate it
Been holding off on upgrading because of all the changes. Is it pretty stable at this point?
o
Yes, very and critically acclaimed 😅
s
nice!
r
Hi @Or Weis! We've tried to spin up a PDP using V0.9.3 and it failed. I'm going to share the logs with you in private
o
Please do.