Hey all.. looking to understand how are you doing any batch processing work in ColdFusion? are there any frameworks you are relying on, and what are you using it for? would love to chat if this is of interest.. please comment or DM to talk more!

Can you clarify what you have in mind RE "batch processing"?

We just run them as scheduled jobs, mostly through the CF Admin.

@rstewart - processing of high volume / repetitive jobs

We also use scheduled jobs for the ones we handle in CF. We also have a lot of complex stored procedures and SIS packages that do somewhat similar kinds of things.

We use CF's scheduled task capability for regularly-scheduled (daily, weekly, monthly) processing, and all scheduled tasks still run within the application/security framework. High-volume tasks (e.g., processing relatively large user-uploaded files) are performed within the regular application framework and we work to optimize performance on both the application and database layers so those can occur within the normal web request/response cycle to provide feedback to the user. In the rare situations where we have to deal with larger data imports (e.g., more than a few hundred-thousands of records) which just aren't feasible through the web application, for example, we deal with those offline at the database level.

We utilize scheduled tasks for batch processing daily, hourly, etc. Some of those scheduled tasks have long running jobs (like account provisioning on desparate systems, creating big PDFs, etc) and use threads to avoid timeouts and free up the main threads. No frameworks are used.

RabbitMQ paired with CommandBox task runners on Docker Swarm. Gives us scalable, distributed processing not tied to any web server.

This was the area that led us to introduce non-CFML solutions at World Singles in the first place. The scheduled task stuff all still runs in the HTTP request/response context and we really wanted processes that could run 24x7 in the background, not tied to a web server at all. Back then, there really wasn't anything suitable within the CFML world so we wrote some background processes in Scala (and then switched them to Clojure). And that started us down the path of moving away from CFML altogether. I don't know whether the CLI machinery that exists now would have allowed us to write non-web processes that ran indefinitely. At this point we have a lot of non-web processes that run 24x7 and the web apps communicate with them either via the database or via pub/sub (Redis).

^^^ I'm with Sean, This is not something I'd be trying to solve with CFML. CFML is for servicing web requests. And just cos one can solve this sort of thing with CFML doesn't mean one should

I have several processes built on CommandBox task runners that run 24/7 as daemon processes. Both as docker containers or just Windows services as file system watchers, etc.

thanks all!

We use UC4 for our batch job scheduler. ACF and sometimes Lucee can get lazy with scheduled tasks where sometimes they don't run.... UC4 from Atomic is designed for this task and does reporting, notification and analysis of successful or failed jobs. It's also what we use to run batch jobs on our mainframes, etc.