https://linen.dev logo
Join Slack
Powered by
Hi <@U01RH8JQKHD>, I understand the recommendation...
# permit-saas
g
Hi @Or Weis, I understand the recommendation to increase the memory limit, but I’d like to better understand the root cause before making that change. From your documentation, the approximate memory footprint is calculated as number of policy objects × ~6 KB. Based on our current setup, we have a relatively small configuration: one (default) tenant, 23 users, and around 66 instances, which should not require significant memory. Even assuming a baseline memory of ~100 MB for the PDP itself, it’s not clear what would cause a spike beyond our current 512 MB limit. • Most of the pod restarts are due to OOM kills, and we’ve observed that this often happens shortly after a backend deployment. This makes me suspect that there may be some transient memory spikes during. Could you help clarify: 1. What are the main contributors to memory usage in the PDP beyond the policy object size ? 2. Are there known memory spikes during startup or during synchronization with the control plane? 3. Does the PDP temporarily load full policy/state snapshots in memory during refresh? I’m happy to increase the resources if required, but having a clear technical explanation will help us justify the change.
o
@Ask Inkeep
a
Thanks for tagging me @Or Weis. I'll look through relevant sources and get back to you shortly.
Based on the available documentation, here's what I can share about PDP memory usage: ​*Main Memory Contributors:*​ The PDP achieves fast policy evaluation by storing the environment's policy data entirely in local process memory and using redundant data structures and caches. (1) The formula 
# of policy objects × 6kb
(where policy objects include users, resource instances, and tenants) is a rough estimate. ​*Regarding Startup/Sync Behavior:*​ The documentation confirms that when the PDP starts, it fetches configuration from the control plane and syncs policies. Looking at your logs, you can see this sequence: • Fetching PDP config from control plane • Receiving remote config and applying overrides • Launching the engine runner and policy/data updaters • Connecting to the OPAL server and fetching policy bundles ​*Your Observation About Backend Deployments:*​ Your observation that OOM kills occur shortly after backend deployments is valuable. The PDP does perform initial data loading and policy synchronization during startup, which could cause temporary memory spikes. ​*Offline Mode Consideration:*​ If you want extra resilience, the PDP can be configured with offline mode using 
PDP_ENABLE_OFFLINE_MODE
, which uses local backup files. This requires mounting a volume to 
/app/backup
. * I don't have explicit documentation detailing the exact memory spike patterns during synchronization or whether full snapshots are temporarily loaded. This is a technical question that would benefit from input from the Permit.io engineering team. To help our engineers investigate further, could you please confirm: 1. Which Workspace / Project / Env: You mentioned 
owners-table/production/production
- is this correct? 2. What SDK calls are you performing (code snippets would help) ? 3. Which PDP version are you running? (You can check with 
docker inspect permitio/pdp-v2:latest
or check the image digest) 4. Can you share the full PDP logs from just before the OOM kill occurs?
To learn more, see the sources I considered: [[1] Deploy Permit to Production](https://docs.permit.io/how-to/deploy/deploy-to-production)
o
The AI gave a good basic answer here. Regarding on policy data expansion: The actual inflation in memory is dependent on the structure of the data (you can read more here) which would vary between policy types you'd choose. There are multiple factors that affect memory usage. From the data and policy late, custom Rego policy, the pod setup and other services consuming resources, changes happening concurrently to the PDP execution or startup. I have no real way of knowing what in your setup affects resource utilization without doing a deeper dive. Usually in vanilla setups for the amount your objects your describing you'd see less memory utilization I can offer you three suggestions - • start a new vanilla setup add stuff gradually and see when memory spikes • use APM tools (like Datadog) to track memory utilization and specific process involved. • You can consider upgrading to our Enterprise tier and procuring professional services to take on this task for you.
8 Views
Open in Slack
PreviousNext
Powered by