We reported issue with condition sets and condition rules time out and slowness last week and it was fixed on Monday. Today we are experiencing issue with slowness. it times out 2-3 times even in UI and then it response back in 5-60 seconds. we have 64 condition sets which might grow to hundreds.

Hi @Yilmaz Alizadeh, I'm an AI assistant for Permit.io. I'll analyze available sources and get back to you shortly.

Hi @Yilmaz Alizadeh thanks for reaching out. Please provide the workspace/project/env in question, the URL of the API you’re seeing the delays with, and if you can what has changed between the “fixed” state and now Our team will investigate

Hmm our PM was in contact with you recently to upgrade our plan as we were receiving warnings. I'll ask them to get in touch. In the meantime here are our details: workspace = alex-trustflight-sandbox project = default env = dev

@Or Weis Just to confirm that our account is now upgraded to PRO but we still have the same issue. alex-trustflight-sandbox ->OneTrustFlight->Development is the one I am testing on.

We can pretty much no longer load your policy UI. For me it times out every time.

Thanks for letting us know, This is already queued up for the team to investigate. If you could address my question from before - of what has changed between the "fixed" state to now, it might help

Sorry, I missed that. At a guess it would be quantity of data. I think we have maybe 64 condition sets in there now whereas before it was probably ~20. I noticed it timing out occasionally yesterday but now it doesn't load at all.

The team thinks they found the culprit, a fix would like be in prod Sunday or Monday

Brilliant thank you!

Hi Lewis, we’ll investigate this new report and circle back to you. Are the changes you’re expecting policy (e.g. condition sets), or facts/data (e.g. role assignments, attribute values) ? If Data - as a intermediate side note, if you want to control/track how dynamic updates propagate to your PDP - we recommend using the “consistent updates” feature (note which APIs are covered by the feature)

It's policy data, specifically it's new resource sets we're creating that aren't propagating and neither are any of the condition set rules we apply to them. We're still seeing the delays.

CC: @Dan

Hey @Lewis Youl, can you please share the code you are using with the Consistent Updates, specifically the Permit instance initiation and the request/s lines that has the delays?

This is our permit initialization

import { Permit } from 'permitio';

const permit = new Permit({
  token: process.env.PDP_API_KEY,
  pdp: process.env.PDP_URL,
  proxyFactsViaPdp: true,
  throwOnError: true
});

export default permit;

This is the SDK call that we make to create a condition set.

await this.permit.api.conditionSets.create(conditionSet);

Thank you for the details 🌟 Notice that this API call is not supported by the Consistent Updates APIs https://docs.permit.io/how-to/manage-data/local-facts-uploader/#supported-apis Creating or changing a condition set is a schema update, therefor requires a new policy generation. Are you using Custom GitOps with your project? Can you please share the full PDP logs that had the long delay described?

Notice that this API call is not supported by the Consistent Updates APIs

Yes I figured that out earlier in the thread.

Are you using Custom GitOps with your project?

No we aren't.

Thank you for the details. It may very likely be the case, I'll try to find if the policy update may not have been propagated correctly. It may be an error or some race between them. I'll deep dive into the PDP logs to investigate what's going on there. About the Condition Sets page, the fix is ready and is now on testing and review. It will be available on production soon, I'll keep you posted about it 🙏

Sure, hopefully this helps.

Thank you. Those logs contain both logs from other containers in your system, like pgAdmin and NGINX. Would it be possible to filter only the PDP logs from your side? 🙌

Ah sorry about that, thank you for doing that.

Hey @Lewis Youl, the Condition Set page fix is available now on production. Can you please confirm it is working well for you now? 🌟

I'm afraid not, still getting a 502.

Hey @Lewis Youl, sorry for the inconveniences. I too been able to reproduce it still on the

lewis_development

env. It seem to be an issue with the number of condition sets and the amount of resources, actions and roles they have. This issue is the Cartesian fanout that cause the query to return hundreds of thousands of rows and leading to a timeout or even a crash. Even after the previous optimizations, this environment still leads to a very large response and timeout. I'm continuing to work on it to find the right way to overcome it. Thank you for your patience and I'll keep you posted about the solution 🙏

OK thanks. I appreciate that I don't know/understand the underlying implementation (that's why we picked permit right?!) but what I do notice is that there's no pagination on this page, it tries to show everything to you.

Roger that, I'll try to tackle them both ASAP

I'm getting the impression from your comment above that what this behaves like with larger datasets is unknown (I wouldn't consider the amount of data we've put into the system to be large yet). We've been facing these issues in our development (local) environments as well as our QA environment for a new tool. These environments currently have very little data in them compared to what we're likely to have once the product is used in anger so if it's struggling to handle this subset then I am rather concerned at how it's going to behave once we customers adding many of their own roles and resource instances.

That is a valid concern to raise. Your current environment has unusually large schema, consisting of hundreds of resources, roles and condition sets, while having very few data entires (users, resource instances, role assignments). Schema changes, compared to data, are translated by our platform into Rego files, uploaded to Github for SDLC management and synchronized in real-time to all PDPs connected to your environment. Schema changes does not tend to grow or correlate with the number of users. It can be considered as the "code logic" of your application - similar to deploying your code to new servers. The current schema you are managing is unusually high, taking the platform to new limits. That raise the question, if your current design is based on schema changes during production code, we should consider how we can make it more generic and use facts data as the stateful variability. Would you like please to describe your current design, what it is planned to include and how are you implementing it? We would help you plan how to design it to suit your needs best 🙂

@Dan Our design requirement is as follows: each Document X belongs to a Document Type Y. We want to grant a User U the ability to perform Action A either on a specific Document X or on all documents of a Document Type T. Permissions can therefore be granted at both the document level and the document type level. When fully implemented with all of our subsystems we might have 100s of actions and Document Types and Thousands of document instances. Users can receive permissions in two ways: Direct Permission assignment to the user. Indirect Permission assignment through membership in a user group. Additionally, each user should always have visibility over their own documents by default. To implement this, the pattern we’ve been following is: for each Document Type, we create a ResourceSet, The resource set Will have DocumentTypeId equals xxx . then we grant permissions between the UserSet with attribute groupId=xxxx-ggg-xxxx and the ResourceSet DocumentTypeId equals xxx by creating condition rule, consistent with the approach described in the Permit.io documentation. The direct permissions are being handled by using REBAC which works as expected. There will be additional attributes to have resource sets but this is the basic requirements we have for now. @Lewis Youl can you also check and add anything that I missed.

That pretty much covers it. Probably worth adding that the user groups/sets are user defined (fundamentally they relate to tenant specific scoped roles), with each tenant in our application being able to create their own roles and then scope those roles to different entities within our system (it is this scoping that reflects the users sets/groups in permit), hence us testing with potentially large numbers of them. Users are also able to define their own document types (this will also be the case for many other types of resources as the application grows), so in the case of my development environment I have ~30 document types, which as Yilmaz has described above will result in 2 resource sets per document type (so 60 resource sets).

I think it’s best that we schedule some time together to better understand your policy, how to model it better. It sounds like there might be other approach that might be more suitable for you but I don’t know for sure so I prefer to better understand your policy before I recommend anything. Do you have any time available tomorrow or on Thursday?

Tomorrow works, how is 10:30/11 GMT?

10:30 sounds good, who should I invite ( you can send the emails on PM if you prefer )

I have DM'd you