What's the best way to throttle sending data to an API? The data will be a cfquery result to loop through, with >100k records (possibly up to 750k), with an API request for each. I saw some posts via search results using cfsleep, and others using cfthread, just want to understand what the best/safest way would be. Is anyone doing this successfully? Can anyone point me in the right direction? TIA!

If the API sends a slow down response like S3, I send as fast as I can and just retry with a sleep if the API tells me to chill.

It's the Mailgun API, btw.

i.e, total requests per second, time between requests, average requests per second, total concurrent requests, etc?

Cron and batched increments for stuff like mass emailing, and watch yourself they crack down if they think the system is abusing.

Yeah, and Mailgun doesn't really say exactly, just that there are rate limits. So, should I be sending 1000 at a time, then wait X min, send another 1000? Not sure how to approach.

I think that's a question for Mailgun, lol

And I will pursue those questions with Mailgun as well, just unsure how to approach the setup (even if the variables change, like max send count, send every X min, etc.)

Well mailgun’s api has batch sending and read the terms of whichever plan you are on to determine what is allowed. Like 5-10k an hour may be doable.

I would ask how you're batching them. Are you wanting to limit this inside of an SDK that's being used from multuple places (i.e., it's tracking all calls from various sources) or do you have a specific script using mailgun and you're just asking how to track the hits from inside of that specific script?

I'm pretty sure there is a mailgun provider for cbmailservices, but I don't know if it has any limiting built in

Built a script from scratch (I prototyped with just 3 records as a mic test, which is working...simple script). But the actual dataset will be quite large, so now some sort of batching or throttling needs to be considered. Thanks for those links, will look through those.

I would for sure start with the batch functionality Patrick linked to and see how far you get with that

Hi cf_gal, I would recommend creating a database queue and processing a defined number of records at set intervals through a scheduled task. It could look like this: 1. Have a database table (the queue), which contains key data of the records that need to be processed for sending. This includes a column which tells you if it has been sent or not. 2. Have an endpoint, which processes a number or records every time you call it. Once a record is processed, set the status to sent via the column mentioned in point one. 3. Have a scheduled task that calls the endpoint at a set interval. The combination of the number of records per request and the scheduled task interval will give you your processing limit (which should be based on your requirements). As with any engineering decision, there are tradeoffs. This solution has the following benefits. 1. You have a list of what is processed vs not processed. This makes it easy to monitor progress. 2. If you encounter an error, you know the item in which it failed. You can even catch the error, record it in logs or DB and then continue processing the queue. 3. If you do need to fix a bug in the process, you can continue where you left off. 4. If you need to reboot the server, the API process can continue where it left off once the server is back online. 5. You are less likely to run out of memory since you are processing a limited number of items per endpoint call. 6. This solution will limit the amount of resources used resulting in a predictable impact on other workloads. The drawback is that it will take longer to implement and is more complex than other solutions.

Thanks, @angel angel, that's really helpful.

Hey CF Gal, we use Mandrill by MailChimp which has a maximum hourly send limit... I'm sure Mailgun does something similar. They probably have a dashboard that shows your limit. With Mandrill, you can send them all at once if you want, providing you stay within your hourly limit. I am definitely not an email expert, but with that many sends your challenge is more than just sending on a schedule. You probably need to use different email accounts, and/or attach send times to schedule each individual email. Anything to keep out of someone's spam box. Hope that helps.