We are currently performing a large-scale migratio...
# permit-saas
g
We are currently performing a large-scale migration involving hundreds of thousands of accounts. This process includes tenant creation, user provisioning, and role assignment for each account. However, after processing a few hundred accounts, we begin encountering HTTP 429 (Too Many Requests) errors.
a
Hi @Girija shankar, I'm an AI assistant for Permit.io. I'll analyze available sources and get back to you shortly.
For large-scale migrations, Permit.io provides bulk operations that can help reduce the number of API calls and avoid rate limiting issues (1) . Here's how you can handle this more efficiently: Bulk User Operations Permit's API supports syncing and updating users en-masse : • Bulk `Create` User • Bulk `Replace` User • Bulk `Assign Roles` to Users • Bulk `Unassign Roles` to Users • Bulk `Delete` User SDK Support for Bulk Operations The SDKs provide methods for bulk role assignments: ​*Node.js:*​
Copy code
bulkAssign(assignments: RoleAssignmentCreate[]) : Promise<BulkRoleAssignmentReport>;
bulkUnassign(unassignments: RoleAssignmentRemove[]) : Promise<BulkRoleUnAssignmentReport>;
(2) ​*Java:*​
Copy code
BulkRoleAssignmentReport bulkAssign(List<RoleAssignmentCreate> assignments) throws IOException, PermitApiError, PermitContextError;
BulkRoleUnassignmentReport bulkUnassign(List<RoleAssignmentRemove> unassignments) throws IOException, PermitApiError, PermitContextError;
(3) Alternative: SCIM Integration For large-scale user provisioning, you might also consider using Permit's SCIM integration (4) , which allows seamless integration with identity providers like Entra ID, Okta, and others (1) . SCIM simplifies user management by automatically provisioning, updating, and de-provisioning users in Permit . To help our engineers investigate your rate limiting issue further, could you please provide: 1. Which Workspace / Project / Env you are using 2. What calls you are performing (as SDK code or CURL) 3. Which PDP version you are running 4. Any relevant logs from your PDP This information will help us understand your specific migration scenario and provide more targeted assistance.
To learn more, see the sources I considered: [[1] Sync your First User](https://docs.permit.io/overview/sync-your-first-user-with-sdk) [[2] permitio/permit-node/src/api/role-assignments.ts](https://github.com/permitio/permit-node/blob/main/src/api/role-assignments.ts) [[3] permitio/permit-java/src/main/java/io/permit/sdk/api/RoleAssignmentsApi.java](https://github.com/permitio/permit-java/blob/master/src/main/java/io/permit/sdk/api/RoleAssignmentsApi.java) [[4] SCIM Integration with Permit.io via Entra ID (Azure AD)](https://docs.permit.io/integrations/SCIM/EntraID)
a
Hey @Girija shankar, the AI is right. We also recently increased the number of requests per call. You're welcome to read more here.
g
Hi @Alon Boshi Boshi (Permit.io), We currently have a restriction that prevents us from using bulk APIs. The account data we are migrating is received from a third party through our API endpoint, and we synchronize each account individually by invoking the above-mentioned API. Since this is a synchronous process, we are unable to request the third party to provide the account information in bulk.
a
I'm checking with the team for a clearer answer and will get back to you.
o
The account data we are migrating is received from a third party through our API endpoint, and we synchronize each account individually by invoking the above-mentioned API
My recommendation would be one of: 1. poll the data on your side and write it with the bulk API in chunks 2. handle a queue on your side flushing the queue at the pace of the rate-limit
g
Hi @Or Weis i truly appreciate your recommendation but the stakeholders has insisted us to make the process synchronous which means at the same time when they are hitting our APIs the accounts should be synced with permit. Is there any way we can increase Rate or Quota Limits. CC : @Himanshi Gaur
o
Sorry, but this is not an option we provide outside of the rate-limit bounds within our Startup tier
g
does that mean we will have to upgrade the plan ?
o
We do consider special requests for enterprise customers. It would still require a technical review to understand the effort and cost to support such a special request
CC: @Shay Shany, @Natalia Edelson