https://linen.dev logo
Join Slack
Powered by
Hi <@U06EXDBTBQV>, <@U06SX8Z2CTU>, and the <Permit...
# permit-saas
m
Hi @Natalia Edelson, @Thomas, and the Permit.io team, During our recent meeting with both of you, we agreed that we would proceed with performance testing of our solution built on top of Permit.io. We had Rate_limit exception so we decided upgraded to Pro plan. However, despite the upgrade, we are still unable to generate mock data for the tests. We continue to receive rate limit (429) errors, even though our code does not use any parallelization or concurrency — the seeding is done through simple, sequential operations. My Api logs:
Copy code
2025-11-28 08:48:12.202Z fga-api fail: Sea.Fga.Api.Services.SeedDataGenerationService[0]
2025-11-28 08:48:12.202Z fga-api       Failed during test data generation
2025-11-28 08:48:12.202Z fga-api       HTTP Response:
2025-11-28 08:48:12.202Z fga-api       
2025-11-28 08:48:12.202Z fga-api       {
2025-11-28 08:48:12.202Z fga-api         "id": "8d98906253db42908260fb8696cb9781",
2025-11-28 08:48:12.202Z fga-api         "title": "You have exceeded the rate limit",
2025-11-28 08:48:12.202Z fga-api         "support_link": "<https://permit-io.slack.com/ssb/redirect>",
2025-11-28 08:48:12.202Z fga-api         "error_code": "RATE_LIMITED",
2025-11-28 08:48:12.202Z fga-api         "message": "We are sorry for the inconvenience, You have exceeded the rate limit for this resource. 
                       Please try again later.
                       If the issue keeps happening, contact our support on Slack for further guidance.",
2025-11-28 08:48:12.202Z fga-api         "additional_info": null
2025-11-28 08:48:12.202Z fga-api       }
2025-11-28 08:48:12.202Z fga-api 
2025-11-28 08:48:12.202Z fga-api       PermitSDK.OpenAPI.Models.PermitApiException: The HTTP status code of the response was not expected (429).
2025-11-28 08:48:12.202Z fga-api 
2025-11-28 08:48:12.202Z fga-api       Status: 429
2025-11-28 08:48:12.202Z fga-api       Response:
2025-11-28 08:48:12.202Z fga-api       {
2025-11-28 08:48:12.202Z fga-api         "id": "8d98906253db42908260fb8696cb9781",
2025-11-28 08:48:12.202Z fga-api         "title": "You have exceeded the rate limit",
2025-11-28 08:48:12.202Z fga-api         "support_link": "<https://permit-io.slack.com/ssb/redirect>",
2025-11-28 08:48:12.202Z fga-api         "error_code": "RATE_LIMITED",
2025-11-28 08:48:12.202Z fga-api         "message": "We are sorry for the inconvenience, You have exceeded the rate limit for this resource. 
                       Please try again later.
                       If the issue keeps happening, contact our support on Slack for further guidance.",
2025-11-28 08:48:12.202Z fga-api         "additional_info": null
2025-11-28 08:48:12.202Z fga-api       }
2025-11-28 08:48:12.202Z fga-api 
2025-11-28 08:48:12.202Z fga-api          at PermitSDK.OpenAPI.PermitClient.Create_relationship_tupleAsync(
                                                String proj_id,
                                                String env_id,
                                                RelationshipTupleCreate body,
                                                CancellationToken cancellationToken)
2025-11-28 08:48:12.202Z fga-api          at PermitSDK.Api.CreateRelationshipTuple(RelationshipTupleCreate relationshipTuple)
2025-11-28 08:48:12.202Z fga-api          at Sea.Fga.Permit.Services.PdpService.<>c__DisplayClass20_0.<<AddRelationship>b__0>d.MoveNext()
                                                in /app/src/BE/Sea.Fga.Permit/Services/PdpService.cs:line 402
Pdp logs:
Copy code
2025-11-28 09:21:50.791Z pdp 2025-11-28T09:21:50.790972+0000 | opal_client.data.fetcher                |ERROR  | Timeout while fetching url: <https://opal-relay.api.permit.io/v2/pdp/callback?retry=0>
2025-11-28 09:21:50.791Z pdp Traceback (most recent call last):
2025-11-28 09:21:50.791Z pdp   File "/usr/local/lib/python3.10/asyncio/locks.py", line 214, in wait
2025-11-28 09:21:50.791Z pdp     await fut
2025-11-28 09:21:50.791Z pdp asyncio.exceptions.CancelledError
2025-11-28 09:21:50.791Z pdp 
2025-11-28 09:21:50.791Z pdp During handling of the above exception, another exception occurred:
2025-11-28 09:21:50.791Z pdp 
2025-11-28 09:21:50.791Z pdp Traceback (most recent call last):
2025-11-28 09:21:50.791Z pdp   File "/usr/local/lib/python3.10/asyncio/tasks.py", line 456, in wait_for
2025-11-28 09:21:50.791Z pdp     return fut.result()
2025-11-28 09:21:50.791Z pdp asyncio.exceptions.CancelledError
2025-11-28 09:21:50.791Z pdp 
2025-11-28 09:21:50.791Z pdp The above exception was the direct cause of the following exception:
2025-11-28 09:21:50.791Z pdp 
2025-11-28 09:21:50.791Z pdp Traceback (most recent call last):
2025-11-28 09:21:50.791Z pdp   File "/usr/local/lib/python3.10/site-packages/opal_client/data/fetcher.py", line 75, in handle_url
2025-11-28 09:21:50.791Z pdp     response = await self._engine.handle_url(url, config=config)
2025-11-28 09:21:50.791Z pdp   File "/usr/local/lib/python3.10/site-packages/opal_common/fetcher/engine/fetching_engine.py", line 116, in handle_url
2025-11-28 09:21:50.791Z pdp     await asyncio.wait_for(wait_event.wait(), timeout)
2025-11-28 09:21:50.791Z pdp   File "/usr/local/lib/python3.10/asyncio/tasks.py", line 458, in wait_for
2025-11-28 09:21:50.791Z pdp     raise exceptions.TimeoutError() from exc
2025-11-28 09:21:50.791Z pdp asyncio.exceptions.TimeoutError
2025-11-28 09:21:50.791Z pdp 2025-11-28T09:21:50.791548+0000 | opal_client.callbacks.reporter          |ERROR  | Failed to send report to <https://opal-relay.api.permit.io/v2/pdp/callback?retry=0>, info=TimeoutError()
1. What are the exact rate limits for the Pro plan and for the Enterprise plan? 2. Are the rate limits applied per endpoint, or are they global (e.g., shared across all API calls or per IP address)? 3. Is there an overload or a bulk-insert option for relationship tuples? We already use all available bulk operations in the API (batch size 1000), but relationship can't find bulk for relationship. 4. Is it possible to temporarily increase the rate limit for our project/environment? Our mock data generation is fully synchronous (no concurrency), yet we still hit rate limits, making performance testing impossible at the moment.
o
Hi @Michał Wójcik and thanks for reaching out 1. There are no differences in rate limits between the plans for the API - there are differences in quotas (overall objects) , and in QoS (priority) a. Rate limits are described here- https://docs.permit.io/api/api-with-cli/#rate-limiting 2. Rate limits are applied per IP address 3. https://api.permit.io/scalar#tag/relationship-tuples/post/v2factsproj_idenv_idrelationship_tuplesbulk 4. Not at the moment > Our mock data generation is fully synchronous (no concurrency), yet we still hit rate limits, making performance testing impossible at the moment. You can still go over the limit even if you don’t do things in parallel. Performance testing shouldn’t be focused on loading data on-mass but critical paths like enforcement, synchronization, reverse-indices (e.g. get-user-permissions), Permit isn’t a DB service (at least not primarily) Usually loading data on volume happens infrequently (e.g. initial setup. new end-customer on boarding) -we provide other means (such as sharing data as files with us to be synced onto a system for mass data load. Both @Natalia Edelson and @Thomas are OOO for Thanksgiving, I suggest you schedule a call with them for next week to get into the details you need
m
Hi, thanks a lot for the quick and detailed reply — the rate-limit documentation link clarifies many things for us. I was also looking for bulk relationship tuple creation under the “BulkOperations” section before, so thank you for pointing me to the correct endpoint. I’ll adjust our code to use it, and this alone should reduce our request volume by roughly 5×, which may already be enough to stay safely under the limits. And yes, we’re not trying to performance-test writes. This is more of an initialization phase — we want to load roughly the same volume of data that we expect in production, and we simply need a practical way to get it into the system. The actual performance testing we care about focuses on Check, BulkCheck, AuthorizeUsers, and user-permissions endpoints. I assumed Thomas and Natalia might be OOO for Thanksgiving, that’s exactly why I reached out here — hoping to gather some extra information in the meantime. Thanks again for the guidance!
Hi team, I'm back with my findings. The check function is fast enough, but when it comes to bulk checks, it doesn't look promising. Our current architecture consists of our API + PDP as a sidecar in a Kubernetes cluster. HPA disabled. Replica count 1. PDP settings: PDP_DEBUG = false PDP_OPA_CLIENT_QUERY_TIMEOUT=30 PDP_CACHE_STORE= in-memory current resources assigned for pdp : cpu limit: 4.5 vcore cpu request: 500m memory limit: 6144 Mi memory request: 4096 Mi For a data set of ~60,000 objects. According to the documentation, 60,000 x 6 kB ~ 360 MB. Currently, our PDP without any load after restart takes up 1 GB. This is the first question: why does the PDP without load consume 1 GB instead of 360 MB?
Copy code
const STAGES = [ 
{ target: 50, duration: '1m' },
 { target: 100, duration: '1m' },
 { target: 150, duration: '1m' },
 { target: 200, duration: '1m' },
 { target: 250, duration: '1m' } ];
 export const options = { 
discardResponseBodies: true, 
scenarios: { 
constant_load: 
{ executor: 'ramping-arrival-rate', 
startRate: 50, 
timeUnit: '1s', 
preAllocatedVUs: 100,
 maxVUs: 3000, 
stages: STAGES, } },
 thresholds: Object.assign({}, THRESHOLDS, TAG_THRESHOLDS), summaryTrendStats: ['min', 'avg', 'med', 'max','p(50)','p(75)', 'p(90)', 'p(95)', 'p(99)']
When performing a load test for the bulk endpoint in k6s with the following settings (100 checks in a single request), memory consumption peaks at 18 GB(first grafana screenshot), metrics from 2 attempts on above test settings.. During the test, it restarts several times. Therefore, the results are disappointing (second screenshot). Before restart I'm having logs inside pdp like that:
Copy code
2025-12-08 12:43:45.431info2025-12-08T12:43:45.431008+0000 | opal_client.engine.logger               | INFO  | Received request.    POST /v1/data/permit/bulk
2025-12-08 12:43:45.419info2025-12-08T12:43:45.418782+0000 | opal_client.engine.logger               | INFO  | Received request.    POST /v1/data/permit/bulk
2025-12-08 12:43:45.404info2025-12-08T12:43:45.403821+0000 | opal_client.engine.logger               | INFO  | Received request.    POST /v1/data/permit/bulk
2025-12-08 12:43:45.402error[2025-12-08T12:43:45Z ERROR pdp_server::api::authz::allowed_bulk] Failed to send request to OPA: Failed to send request to OPA: error sending request for url (<http://localhost:8181/v1/data/permit/bulk>)
2025-12-08 12:43:45.392error[2025-12-08T12:43:45Z ERROR pdp_server::api::authz::allowed_bulk] Failed to send request to OPA: Failed to send request to OPA: error sending request for url (<http://localhost:8181/v1/data/permit/bulk>)
2025-12-08 12:43:45.391info2025-12-08T12:43:45.390236+0000 | opal_client.engine.logger               | INFO  | Received request.    POST /v1/data/permit/bulk
2025-12-08 12:43:45.390info2025-12-08T12:43:45.389390+0000 | opal_client.engine.logger               | INFO  | Sent response.       POST /v1/data/permit/bulk -> 500
2025-12-08 12:43:45.379info2025-12-08T12:43:45.379622+0000 | opal_client.engine.logger               | INFO  | Received request.    POST /v1/data/permit/bulk
2025-12-08 12:43:45.378error[2025-12-08T12:43:45Z ERROR pdp_server::api::authz::allowed_bulk] Failed to send request to OPA: Failed to send request to OPA: error sending request for url (<http://localhost:8181/v1/data/permit/bulk>)
2025-12-08 12:43:45.371info2025-12-08T12:43:45.370995+0000 | opal_client.engine.logger               | INFO  | Received request.    POST /v1/data/permit/bulk
2025-12-08 12:43:45.365error[2025-12-08T12:43:45Z ERROR pdp_server::api::authz::allowed_bulk] Failed to send request to OPA: Failed to send request to OPA: error sending request for url (<http://localhost:8181/v1/data/permit/bulk>)
2025-12-08 12:43:45.362info2025-12-08T12:43:45.362281+0000 | opal_client.engine.logger               | INFO  | Received request.    POST /v1/data/permit/bulk
2025-12-08 12:43:45.361error[2025-12-08T12:43:45Z ERROR pdp_server::api::authz::allowed_bulk] Failed to send request to OPA: Failed to send request to OPA: error sending request for url (<http://localhost:8181/v1/data/permit/bulk>)
2025-12-08 12:43:45.353info2025-12-08T12:43:45.352767+0000 | opal_client.engine.logger               | INFO  | Received request.    POST /v1/data/permit/bulk
2025-12-08 12:43:45.346error[2025-12-08T12:43:45Z ERROR pdp_server::api::authz::allowed_bulk] Failed to send request to OPA: Failed to send request to OPA: error sending request for url (<http://localhost:8181/v1/data/permit/bulk>)
2025-12-08 12:43:45.344info2025-12-08T12:43:45.344273+0000 | opal_client.engine.logger               | INFO  | Sent response.       POST /v1/data/permit/bulk -> 500
2025-12-08 12:43:45.344error2025-12-08T12:43:45.343908+0000 | opal_client.engine.logger               |ERROR  | Log event masking failed: eval_cancel_error: context canceled. {"plugin": "decision_logs", "time": "2025-12-08T12:43:38Z"}
2025-12-08 12:43:45.336info2025-12-08T12:43:45.335303+0000 | opal_client.engine.logger               | INFO  | Received request.    POST /v1/data/permit/bulk
2025-12-08 12:43:45.329error[2025-12-08T12:43:45Z ERROR pdp_server::api::authz::allowed_bulk] Failed to send request to OPA: Failed to send request to OPA: error sending request for url (<http://localhost:8181/v1/data/permit/bulk>)
2025-12-08 12:43:45.321info2025-12-08T12:43:45.320948+0000 | opal_client.engine.logger               | INFO  | Received request.    POST /v1/data/permit/bulk
2025-12-08 12:43:45.321error[2025-12-08T12:43:45Z ERROR pdp_server::api::authz::allowed_bulk] Failed to send request to OPA: Failed to send request to OPA: error sending request for url (<http://localhost:8181/v1/data/permit/bulk>)
2025-12-08 12:43:45.320error[2025-12-08T12:43:45Z ERROR pdp_server::api::authz::allowed_bulk] Failed to send request to OPA: Failed to send request to OPA: error sending request for url (<http://localhost:8181/v1/data/permit/bulk>)
2025-12-08 12:43:45.317info2025-12-08T12:43:45.316112+0000 | opal_client.engine.logger               | INFO  | Received request.    POST /v1/data/permit/bulk
2025-12-08 12:43:45.311error[2025-12-08T12:43:45Z ERROR pdp_server::api::authz::allowed_bulk] Failed to send request to OPA: Failed to send request to OPA: error sending request for url (<http://localhost:8181/v1/data/permit/bulk>)
2025-12-08 12:43:45.300info2025-12-08T12:43:45.299852+0000 | opal_client.engine.logger               | INFO  | Received request.    POST /v1/data/permit/bulk
2025-12-08 12:43:45.299error[2025-12-08T12:43:45Z ERROR pdp_server::api::authz::allowed_bulk] Failed to send request to OPA: Failed to send request to OPA: error sending request for url (<http://localhost:8181/v1/data/permit/bulk>)
2025-12-08 12:43:45.291info2025-12-08T12:43:45.291294+0000 | opal_client.engine.logger               | INFO  | Received request.    POST /v1/data/permit/bulk
2025-12-08 12:43:45.276info2025-12-08T12:43:45.274419+0000 | opal_client.engine.logger               | INFO  | Received request.    POST /v1/data/permit/bulk
2025-12-08 12:43:45.276error[2025-12-08T12:43:45Z ERROR pdp_server::api::authz::allowed_bulk] Failed to send request to OPA: Failed to send request to OPA: error sending request for url (<http://localhost:8181/v1/data/permit/bulk>)
2025-12-08 12:43:45.254info2025-12-08T12:43:45.254273+0000 | opal_client.engine.logger               | INFO  | Received request.    POST /v1/data/permit/bulk
2025-12-08 12:43:45.247error[2025-12-08T12:43:45Z ERROR pdp_server::api::authz::allowed_bulk] Failed to send request to OPA: Failed to send request to OPA: error sending request for url (<http://localhost:8181/v1/data/permit/bulk>)
2025-12-08 12:43:45.236error[2025-12-08T12:43:45Z ERROR pdp_server::api::authz::allowed_bulk] Failed to send request to OPA: Failed to send request to OPA: error sending request for url (<http://localhost:8181/v1/data/permit/bulk>)
2025-12-08 12:43:45.235error[2025-12-08T12:43:45Z ERROR pdp_server::api::authz::allowed_bulk] Failed to send request to OPA: Failed to send request to OPA: error sending request for url (<http://localhost:8181/v1/data/permit/bulk>)
2025-12-08 12:43:45.232info2025-12-08T12:43:45.231673+0000 | opal_client.engine.logger               | INFO  | Received request.    POST /v1/data/permit/bulk
2025-12-08 12:43:45.211info2025-12-08T12:43:45.210955+0000 | opal_client.engine.logger               | INFO  | Received request.    POST /v1/data/permit/bulk
2025-12-08 12:43:45.199info2025-12-08T12:43:45.199029+0000 | opal_client.engine.logger               | INFO  | Received request.    POST /v1/data/permit/bulk
2025-12-08 12:43:45.195error[2025-12-08T12:43:45Z ERROR pdp_server::api::authz::allowed_bulk] Failed to send request to OPA: Failed to send request to OPA: error sending request for url (<http://localhost:8181/v1/data/permit/bulk>)
2025-12-08 12:43:45.177error[2025-12-08T12:43:45Z ERROR pdp_server::api::authz::allowed_bulk] Failed to send request to OPA: Failed to send request to OPA: error sending request for url (<http://localhost:8181/v1/data/permit/bulk>)
2025-12-08 12:43:45.174error[2025-12-08T12:43:45Z ERROR pdp_server::api::authz::allowed_bulk] Failed to send request to OPA: Failed to send request to OPA: error sending request for url (<http://localhost:8181/v1/data/permit/bulk>)
2025-12-08 12:43:45.173error[2025-12-08T12:43:45Z ERROR pdp_server::api::authz::allowed_bulk] Failed to send request to OPA: Failed to send request to OPA: error sending request for url (<http://localhost:8181/v1/data/permit/bulk>)
2025-12-08 12:43:45.168error[2025-12-08T12:43:45Z ERROR pdp_server::api::authz::allowed_bulk] Failed to send request to OPA: Failed to send request to OPA: error sending request for url (<http://localhost:8181/v1/data/permit/bulk>)
2025-12-08 12:43:45.167info2025-12-08T12:43:45.167181+0000 | opal_client.engine.logger               | INFO  | Received request.    POST /v1/data/permit/bulk
2025-12-08 12:43:45.156error[2025-12-08T12:43:45Z ERROR pdp_server::api::authz::allowed_bulk] Failed to send request to OPA: Failed to send request to OPA: error sending request for url (<http://localhost:8181/v1/data/permit/bulk>)
2025-12-08 12:43:45.146info2025-12-08T12:43:45.145708+0000 | opal_client.engine.logger               | INFO  | Received request.    POST /v1/data/permit/bulk
2025-12-08 12:43:45.145error[2025-12-08T12:43:45Z ERROR pdp_server::api::authz::allowed_bulk] Failed to send request to OPA: Failed to send request to OPA: error sending request for url (<http://localhost:8181/v1/data/permit/bulk>)
2025-12-08 12:43:45.137error[2025-12-08T12:43:45Z ERROR pdp_server::api::authz::allowed_bulk] Failed to send request to OPA: Failed to send request to OPA: error sending request for url (<http://localhost:8181/v1/data/permit/bulk>)
2025-12-08 12:43:45.135error[2025-12-08T12:43:45Z ERROR pdp_server::api::authz::allowed_bulk] Failed to send request to OPA: Failed to send request to OPA: error sending request for url (<http://localhost:8181/v1/data/permit/bulk>)
2025-12-08 12:43:45.134error[2025-12-08T12:43:45Z ERROR pdp_server::api::authz::allowed_bulk] Failed to send request to OPA: Failed to send request to OPA: error sending request for url (<http://localhost:8181/v1/data/permit/bulk>)
2025-12-08 12:43:45.117info2025-12-08T12:43:45.117324+0000 | opal_client.engine.logger               | INFO  | Received request.    POST /v1/data/permit/bulk
2025-12-08 12:43:45.103error[2025-12-08T12:43:45Z ERROR pdp_server::api::authz::allowed_bulk] Failed to send request to OPA: Failed to send request to OPA: error sending request for url (<http://localhost:8181/v1/data/permit/bulk>)
2025-12-08 12:43:45.100info[2025-12-08T12:43:45Z INFO  pdp_server::api::health::handlers] Health check failed: opa: OPA health check timed out after 3 seconds
2025-12-08 12:43:45.097error[2025-12-08T12:43:45Z ERROR pdp_server::api::authz::allowed_bulk] Failed to send request to OPA: Failed to send request to OPA: error sending request for url (<http://localhost:8181/v1/data/permit/bulk>)
I found the issue on the github https://github.com/open-policy-agent/opa/issues/6753#issue-2306024817 is it something similar?
o
Hi @Michał Wójcik, thanks for sharing. > This is the first question: why does the PDP without load consume 1 GB instead of 360 MB? > OPA expand objects in memory with the policy and is known to add bloat on top of the baseline data https://www.openpolicyagent.org/docs/policy-performance#resource-utilization Any restarts you’re experiencing are likely a factor of your K8s setup terminating the pods due to resource limits or delays on the health checks due to resource limits. This is to be expected with no scale-out, high workload, and such limited resources. You can consider: 1. Allocating more resources(memory / CPU) to your PDPs 2. Experiment with combinations of different bulk batch sizes, and number of PDPs you load balance the requests between I’ll also ask @Omer Zuarets to review and see if he has more suggestions
o
Hey @Michał Wójcik, Regarding bulk checks, as @Or Weis mentioned, there is a very thin line between concurrent requests and bigger bulk size, there is no magic number to give, it varies between policy to policy. Regarding the memory consumption, I assume it’s a combination of the cache getting larger due to unique check requests that grows because of the tests and concurrent request handling
m
Hi @Thomas, @Omer, @Alon and @Or Weis First, I want to highlight that we’re very happy with Permit.io — the functionality fully matches our authorization requirements, and the developer experience has been great so far. Right now we’re running performance tests simply to understand: • what latency we can expect from our specific authorization model, and • how to size our hardware in terms of memory/CPU for production. Yesterday, for this specific test case, we deployed 3 replicas of the PDP and allocated ~7 GB RAM per pod (total ~21 GB). Even in this setup, running the exact same narrow test scenario, the pods eventually start restarting. The PDP logs always show the same error:
Copy code
[2025-12-08T15:23:44Z ERROR pdp_server::api::authz::allowed_bulk]
Failed to send request to OPA: Failed to send request to OPA:
error sending request for url (<http://localhost:8181/v1/data/permit/bulk>)
At this stage, we’re not sure whether the issue is caused by: • our configuration, • our Kubernetes infrastructure setup, or • inherent limits in the Bulk Check functionality that we’re simply hitting and cannot bypass. Given this, we need more concrete guidance from your engineering team. Specifically: 1. Does the PDP cache have any eviction/TTL mechanism, or can it grow without limit with unique fact patterns? 2. Is there a recommended configuration to limit cache size or disable caching for performance tests? My current cache setting: - name: PDP_CACHE_STORE value: "in-memory" - name: PDP_CACHE_TTL value: "300" - name: PDP_CACHE_MEMORY_CAPACITY value: "256" PDP_CACHE_STORE :"none" decrease idle pdp memory state to 900 MB, but under load still falling
o
Hi I've asked the team to take a deeper look. Meanwhile - Have you tried changing the size of the bulk batches (both number of entries, and sizes of objects passed as part of the entry), as was suggested? the more data you load the more OPA inflates it in memory, and data loading is the slowest process for OPA, slowing everything down in addition. Can you provide an example of your average bulk call? might help us understand where this is constraining.
g
good day, guys @Thomas I am sorry to bother again but next week we will need to get our executives to sign off on the vendor, the load test results have to be shared; and we are very interested to polish them up a bit, there might be a scenario our setup is suboptimal. would it be possible to schedule a short call? with you, maybe with Gabriel (or whoever happens to have experience with bigger bulk calls / optimization of that load), we are sure this will be one of the most crucial questions we will be asked next week. fyi @Michał Wójcik @Shyamalan Chemmery
m
@Or Weis Here you have more granular results for different page size/ RPS. We are talking here about dataset 2, which contains around 60,000 objects. Due to pod restarts, we see different error rates in the highest RPS, so the results are unreliable; they are simply hitting the OPA timeout limits. Do ABAC policies include the OPA evaluation, or only custom policies?
a
Hey @Michał Wójcik, @Gene I. - Thanks for following up - will make sure the team see this chat and will circle back
✅ 2
t
Hey @Gene I. and @Michał Wójcik I'm happy to jump on a call. https://calendly.com/thomas-permitio/1-hour
✅ 1
❤️ 1
🙏 1
@Michał Wójcik can you confirm that 
PDP_CACHE_STORE:in-memory
or 
PDP_CACHE_STORE:redis
(you'll have to point to a redis for it to work) is enabled. Also, it would probably be beneficial for 
PDP_DECISION_LOG_DEBUG_INFO:false
. Also, @Michał Wójcik what is the size of what you are running the PDP on?
m
Hi Thomas, our current setup for pdp:
Copy code
environmentVariablesPDP:
  - name: PDP_API_KEY
    value: "" 
  - name: PDP_DEBUG
    value: "False"
  - name: PDP_OPA_CLIENT_QUERY_TIMEOUT
    value: "30"
  - name: PDP_CACHE_STORE
    value: "in-memory"
  - name: PDP_CACHE_TTL
    value: "300"
  - name: PDP_CACHE_MEMORY_CAPACITY
    value: "256"
I will try to add now PDP_DECISION_LOG_DEBUG_INFO = false. Resource setup for pdp process:
Copy code
replica count: 3
pdpResourceLimitsCPU: "1250m"
pdpResourceLimitsMemory: "7Gi"
pdpResourceRequestsCPU: "500m"
pdpResourceRequestsMemory: "2Gi"
I scheduled meeting, see you tomorrow .
o
Hi I've asked multiple times if you tried changing the size of the batches for the bulk checks. is there a reason you are ignoring the suggested solution?
m
In the spreadsheet shared above the Page Size is a size of the batch, we've tested sizes: 10, 20, 50, 100.
o
so it's pretty clear the issue is resolved for dataset one by changing the batch size. Now I've also asked for an example of the entries to the bulk call. I'm guessing you're passing very large objects in dataset2. Please share dataset 2, and please evaluate: 1. if you actually need all that data for the authorization call (does the policy actually use it) 2. if some of that data can by synced asynchronously to the PDP, instead of being passed in the check call itself
m
The data set is indicative of the size of the data in the environment. Data set 1 is very small around 5k objects, Data Set 2 is around 60k objects (screenshot). To be clear we are not passing any attributes in the check request. Typical bulk-check is like that for batch-size 2 :
Copy code
### Bulk Check
POST {{url}}api/pdp/bulk-check
Authorization: Bearer {{auth.response.body.access_token}}
Content-Type: application/json

{
  "Checks": [
    {
      "user": "<mailto:user5@company-20251126115414-1.com|user5@company-20251126115414-1.com>",
      "action": "reply",
      "resource": "market_order:order-20251126115414-0011"
    },
    {
      "user": "<mailto:f3d81b93-506f-4cd1-a9f2-20b8496b7c12@company3.com|f3d81b93-506f-4cd1-a9f2-20b8496b7c12@company3.com>",
      "action": "edit",
      "resource": "negotiation:negotiation-order-20251128142333-0341-003"
    }
  ]
}
o
Wait, so objects is number of entries for bulk check, not number of objects within each check? Why would you need to evaluate 60K checks 250 times per second?
Are you trying to implement some sort of data filtering?
s
Hi @Or Weis we have seeded the environment with @60k objects (as @Michał Wójcik shared above) and we are running bulk checks against the data with batch size of 10, 20, 50 & 100 in each check. Hope this helps.
o
I’m sorry. Permit has no concept of “objects” - and I keep reading your messages to have different meaning for the word Is 60K • the number of checks passed in bulk • the number of resource instances preloaded into the system • the number of identities preloaded into the system • the number of something else ? You provided an example of a bulk check with 2 checks - that doesn’t reflect any of the batch sizes.
s
60k (or 57200 to be precise) is the sum total of identities & resources preloaded into the system. The example with 2 check was shared to give an idea of the bulk checks that we are running i.e., only with user key, resource key and action. No additional attributes are passed.
o
Okay, so how many checks do you need to pass per second? Seems that with 20 or less checks per bulk and/or RPS lower than 100; there is no problem. these sound like reasonable numbers to me, and that if you need can be scaled out with additional PDPs. We can go into maybe critiquing the policy you authored (though I'm guessing it's simple RBAC/ReBAC)
t
To emphasize Or's response: This load test seems to be reasonable for somewhere between 20 - 50 checks per bulk and lower than 150 rps that there are decent numbers. We may be able to get some better numbers with the flags I mentioned and the right balance of CPU/Mem but beyond that we would want to load balance the PDP which the architecture is already designed to support.
🙌 1
m
Hi team, thanks again for the recent conversation with Thomas and Dan — it clarified a lot for us and was very helpful 👍 Sorry to bother you again, but I wanted to share one more observation from our ongoing performance validation and ask for guidance on intended usage patterns. We started testing the 
/user-permissions
 endpoint and noticed the following behavior. When calling 
POST /user-permissions
and: • filtering only by 
resource_types
 (2 types), • where the total number of resource instances assigned to the user exceeds ~100, the request consistently returns 500, and the PDP logs:
Copy code
[ERROR pdp_server::api::authz::user_permissions]
Failed to send request to OPA: error sending request for url (<http://localhost:8181/v1/data/permit/user_permissions>)
However: • narrowing 
resource_types
to a single type works, • passing explicit resource instances works and is fast. So functionally everything behaves as expected — the failure appears only when the evaluation needs to expand a larger internal result set. At this point, we’re not trying to push extreme workloads. We’re trying to understand what is considered supported and recommended usage. Main question:
Are there any documented recommendations or known limitations regarding:
/user-permissions
result-set size per user
filtering by 
resource_types
vs explicit resource identifiers?
If the expectation is that 
/user-permissions
should be used only with narrow filters, that’s perfectly fine — we just need to design around it and set realistic performance expectations. Thanks again for your time and support.
p
Hey, thanks for reaching out ! It seems our team is a little busy right now, a human will be with you within the next hour or so. In the meantime, can I help you ?
t
Hey @Michał Wójcik, what is the p95 response time for 
/user-permissions
without any additional filtering? I am guessing that you are running into OPA timeout issues, which in turn points to memory constraints on the PDP. The number of concurrent 
/user-permissions
is lower than 
/allowed
. I'll also get @Dan to weigh in 🙂.
I assume you are running PDP_OPA_CLIENT_QUERY_TIMEOUT=30 still, correct?
s
Thanks for reverting @Thomas. We have not fully load tested user-permissions yet. The above were observed while doing some basic tests. And yes, the timeout is 30.
p
Hey, thanks for reaching out ! It seems our team is a little busy right now, a human will be with you within the next hour or so. In the meantime, can I help you ?
m
Shyam is right; we're just struggling to execute user permissions on our dataset. From the logs, I can see that for resource_type = market_order, 1262 permissions are being returned - this is working.
Copy code
2025-12-15T22:06:17.219471+0000 | opal_client.engine.logger               | INFO  | Sent response.       POST /v1/data/permit/user_permissions -> 200
[2025-12-15T22:06:17Z INFO  pdp_server::opa_client::user_permissions] permit.user_permissions("<mailto:6e400f86-95bd-4062-b918-dc63b49b0fb5@company2.com|6e400f86-95bd-4062-b918-dc63b49b0fb5@company2.com>", tenants=Some(["default"]), resources=None, resource_types=Some(["market_order"])) -> 1262 permissions
but when I asked about resource_type=negotiation (there should be more permissions than 1262) it is failing with ( I increased a timeout for PDP_OPA_CLIENT_QUERY_TIMEOUT)
Copy code
2025-12-15T22:15:17.395871+0000 | opal_client.engine.logger               | INFO  | Received request.    POST /v1/data/permit/user_permissions
2025-12-15T22:15:17.796363+0000 | uvicorn.protocols.http.httptools_impl   | INFO  | 127.0.0.1:49856 - "GET /healthy HTTP/1.1" 200
2025-12-15T22:16:57.121001+0000 | opal_client.engine.logger               | INFO  | Sent response.       POST /v1/data/permit/user_permissions -> 500
2025-12-15T22:16:58.619562+0000 | uvicorn.protocols.http.httptools_impl   | INFO  | 127.0.0.1:39686 - "GET /healthy HTTP/1.1" 200
2025-12-15T22:17:01.165040+0000 | opal_client.engine.logger               | INFO  | Logs uploaded successfully. {"plugin": "decision_logs", "time": "2025-12-15T22:17:01Z"}
Both of requests are for user 
"<mailto:6e400f86-95bd-4062-b918-dc63b49b0fb5@company2.com|6e400f86-95bd-4062-b918-dc63b49b0fb5@company2.com>"
a
Hey Michal - Thanks for following up Saw your message and we will circle back to you soon.
m
One more datapoint from our experiments that might help narrow this down. For context, our ReBAC model has three hierarchy levels: 
company → market_order → negotiation
. When calling(without resource instances specify only resource type) 
/user-permissions
: • for 
market_order
only, the endpoint returns data correctly, • for 
negotiation
, the request fails. Our assumption is that, in the negotiation case, the result set is much larger, because access is inherited through multiple group-based relationships. To validate this, we ran an additional test: • we passed ~2000 explicit 
resource_instances
 directly to 
/user-permissions
• these resources were assigned directly to the user (not via groups) In this scenario, the endpoint works without any issues and returns results. This suggests that the problem is not the absolute number of resources, but rather the explosion caused by group-based and hierarchical expansion during evaluation.
d
Hey @Michał Wójcik, thank you for the details. I'm happy to see the improvements in your benchmarks 💪✨ About the get user permissions with larger datasets, I suspect it is a memory issue and not (only) a latency / timeout issue. As observed from the logs you've shared, OPA is returning the 500 after 1:40min. Usually when getting a timeout, the response code would be 502 BTW. It might be that the amount of available permissions is so large that it consumes the entire available memory of OPA, and causing it to OOM (Out-of-Memory) crash and restart. You can check it in the PDP logs, seeing if OPA is restarting after sending the large request. Considering user permission queries require traversing the entire depth of the graph to find all of the granted permissions, the memory footprint of the request is the product (or explosion) of each depth of the derivation graph. (Good job, you've beat me to it and figured it out as well 👏) TL;DR - the memory footprint grows exponentially the larger the depth and number of relations in your chart. That does not necessarily mean you won't be able to do those queries, but it may require careful optimizations and resource adjustments for the PDPs. Try increasing your PDP memory until the query successfully pass, then asses the memory usage. Depending on your use-case, you may consider breaking down the checks to multiple queries, starting with query on permitted 
market_orders
then performing query/s on explicit 
negotiation
instances. This approach would allow you to split the work across your PDP cluster, and potentially increase your shared cache hit-rate. Let me know how it goes! 🌟
Open in Slack
PreviousNext
Powered by