UNOFFICIAL SURVEY: • Do you start testing ColdFusion updates immediately when they're released, or wait for feedback from the community? • Roughly how long do you test them before deploying them to production? • Is there an approval process that involves more than the dev team before you make changes to production servers? • How about Java updates, same questions? • Does any of this depend on the hotfix level, CF 2023 vs CF 2021 HF6? What I'm getting at is that narrowing the "not yet patched" window is clearly a good thing, but how do we balance that with possible breakage, and chain of responsibility? Thoughts?

We use Commandbox precisely so we can mitigate the cost of 'oh shoot this wasn't ready' (and we also use Lucee so we're not relying on Adobe's testing, though the extent to which that's an advantage is obviously subjective). We run our internal test suites on major updates for the CF engine. We don't do this for Java updates only because I don't remember the last time a Java update borked our app; we also don't update Java the moment a new release comes out.

So, as a development "team" of one, I develop locally using CommandBox, deploy to a testing server for QA by my larger workgroup "team", and then deploy to a production server. When updates come out, I let CommandBox automatically update to the latest hotfix, then run all of my tests (integration and unit) against the hotfix. If nothing breaks, I apply the hotfix on our testing server. If nothing breaks after a week or so, I apply the hotfix on our production server.

It all depends the criticality some patches could be applied to shared development, and QA servers pretty rapidly and then migration to production is normally done within 1-2 weeks, unless it is a in the wild vulnerability and then things can be quickened especially in any Internet facing servers.

Prior to Production you should have a QA team verify the application and prior to production verify with the Product Manager/Staff about a release date.

I like sleep too much to want to deploy something that hasnt been tested.

(For some reason Slack web version didn't show me these replies. I was wondering why nobody else was thinking about this.) The reason I was thinking about this yet again was that I was about to apply the most recent java and cf updates, and wondering what due diligence looks like for various devs and organizations. Not at all ironically, we got bit by this bouncycastle error, because we didn't test PDF generation. Clearly we should have, we've seen other java updates break it too. Side notes: • That issue and the download for the fix should have been listed on the CF downloads page. • That issue is marked as deferred, because it's a "Duplicate of CF-4215479" (not the usual meaning of "deferred"!), but that other issue can't be found, apparently. Anyway, this is quite a big app, written over many years by many people, legacy in many ways though not as spaghetti-d as some. The vast majority of it at least has methods to get the data that are separate from the code to render it, and those back-end methods should be more or less testable, with some work. Unmanned testing of front end stuff is much harder IMO, and that's where the PDF error would have shown up. But at least it's a crash, easier to trap for than incorrect HTML or broken interactions. We'll see...

* Do you start testing ColdFusion updates immediately when they're released, or wait for feedback from the community? We start testing CF updates soon after they are released. That is, without waiting for feedback from the community. Just as well, wouldn't you say? Imagine everyone waiting for feedback before testing. That will be a classic prisoner's dilemma resulting in nobody testing. 🙂 * Roughly how long do you test them before deploying them to production? Several days, for critical updates. A number of weeks, for non-critical updates. * Is there an approval process that involves more than the dev team before you make changes to production servers? Yes. The Service Desk team assists in testing. They act as proxy for the hundreds of thousands of users. They give the final nod of approval for roll-out from test to production. * How about Java updates, same questions? Yes. We treat these essentially as we would any non-critical ColdFusion updates. * Does any of this depend on the hotfix level, CF 2023 vs CF 2021 HF6? No. The above pertains to CF 2018 Update 16 and lower.