So I found myself needing to downgrade to CF2023 update 4 to test something, so I ran the uninstaller on update 5. It says it completed successfully, but I'm getting > java.lang.NullPointerException: Cannot invoke "coldfusion.server.LicenseService.isStandard()" because the return value of "coldfusion.server.ServiceFactory.getLicenseService()" is null when I try to go to the CF Admin. I googled that, and found @carehart’s article about needing to use the zip property when installing the update using Java 17.0.8. I did that in the first place. I tried running the uninstall with that property too, but it didn't fix it. I tried downgrading Java to 17.0.7, and uninstalling. Then I tried reinstalling Update 4. Still no dice. Does anyone know how to fix this?

First, when you say you added the jvm arg needed for 17.0.8, were you doing that from the command line or in the cf admin? It needs to be at the cmd line. Though in downgrading to 17.0.7, that can be done in either approach to avoid the new problem. Then again, whether that's the problem at all would be reflected in the cf update log. I detailed that in the first first post on the matter, in July (pointed to in the recent post). Indeed, are you checking the update log at all? Is its count of fatalerrors and nonfatalerrors 0? Btw, there's such a log for the install AND uninstall of an update. Do check both. I have another blog post on that: https://www.carehart.org/blog/2016/9/6/solve_common_problems_with_CF_updates_in_10_and_above

Semi-OT: I've always saved off a copy of the cfusion directory before installing updates, as I believe has been suggested, so in case of fubar I can just copy it back. I've never actually had to do that copying back though. So: 1. Is that strategy actually useful? 2. Would it have helped in this particular case?

Yeah, sorry. Let me clarify. So I installed Update 5 successfully from the commandline. No errors in that log. That worked fine. But then uninstalling update 5, I'm running into this error. In the uninstall log, it shows there were no errors. It just doesn't work when it's done...

Dave, I've never heard of or done that suggestion as a matter of course. No. Though not wrong, it could be huge, and is somewhat overkill since the update itself makes a backup of what's changed in the hf-updates folder. And it would be risky if not done immediately, as over time things will change that may not be safe to just be copied back.

@Dave Merrill I do the same thing and consider it best practice.

So I've got a bunch of errors in the coldfusion-error about not being able to load log4j. And then not much in coldfusion-out as a result, because I'm pretty sure it's log4j that writes to that... This is the one from coldfusion-out.

Unable to install Logging package: java.lang.NoClassDefFoundError: org/apache/logging/log4j/core/Layout

The error log has a bunch of stack traces.

Find the first error in the log. I'd not expect that to be it

I thought it was, but let me wipe it clean and then restart so I can know for sure.

that would be odd that the out log would have only 1 line. or am I misreading you? as for that first error in the error log, that's certainly odd to be the first error. more to the point, it's not an error I've seen before (of this sort, at that time, in that file). Not sure what to propose. Perhaps others will have ideas. I will say that it's possible there may be other practices you have which have led to this, which we can't know. by that I mean you may do something in your setup of CF, or in manual config once installed, which has led to this. any guesses about any such things? to be clear, I have helped many people install and even uninstall this latest update, and I've not seen that error. So it's not a generic problem common to all

well, it's good to know I'm weird at least....

sometimes understanding that is half the battle...er, you know what I mean 🙂

not exactly on topic, if I went from update 3 to update 5, when i uninstall 5, am I expecting to be on 3 again, or on 4?

so as for the coldfusion-out.log line, you referred earlier to "clearing" it. do you mean you do that in an editor, perhaps? or via the CF admin and its archive button? You may want to try the latter--assuming you can get into the CF admin. Or of course you could stop CF and then delete the file. For sure, on startup of CF, that coldfusion-out.log should have hundreds of lines...assuming you're running CF as a service, we should clarify. If instead you run CF from the command line (or start it in an editor), then instead the lines are written to the console (the command line, or that of the editor). Can you confirm how you're running cf? the answer may be in that other place

it's as a service. It does normally have hundreds of lines... Reapplying update 5 "fixes" it -- it starts normally, I can get into the cf admin, etc.

well, you said originally you wanted to go to update 4, from 5. but now you say you were first on 3 before updating to 5. so to be clear, uninstalling 5 should take you back to 3. and then you should be able to install 4. So did you try to install 4, once 5 was uninstalled? or could you not?

I did try installing update 4 after uninstalling 5. The install worked, but the coldfusion server still didn't.

well, what about "the cf server" didn't work? the service didn't start? the admin didn't load? it had errors?

Is this Windows or LINUX?

The server started with the errors above, and the cf admin didn't load.

and are you saying now that the errors are for your attempt to install update 4 after uninstalling update 5 (which should have reverted you to update 3)? Maybe it's best to leave it at update 5 (which you say works) and try all this on some other machine 🙂 Maybe it would not happen there

via the cfadmin, I'm pretty sure.

ok, but do heed all before that last question 🙂

Based on that error (NoClassDefFound), I suspect that one of the upgrades removed the log4j 1.2 JAR(s) -- remember all the security fuss around log4j 1.x and the updates to move us to log4j 2.x? -- and either they didn't get saved to the backup folder or the downgrade didn't restore them. @carehart Does that sound plausible? I would have thought all the log4j 2.x stuff was sorted out well before CF2023, so I'm a bit surprised this happens in the Update 3/4/5 range, but perhaps there was some residual code still using log4j 1.2 and perhaps Update 4 or 5 updated that to 2.x and removed the 1.2 JARs? See https://stackoverflow.com/questions/20909446/caused-by-java-lang-noclassdeffounderror-org-apache-log4j-logger for some supporting evidence

Sean, that's a sane observation--and again since I've not seen the problem with anyone else, I would again assert it may be a result (for Tim) of their having done some manual tweaks (recently or months ago), related to all this. Some folks were not satisfied with the log4j solutions Adobe offered over the past couple of years. I'd long warned about some of the practices, but folks will do what they feel they need to do. And in fact, I think I just realized what this is about: Tim, the problem is on uninstall, right? I'll bet you or someone removed from the backups folder a log4j jar that WAS put there by the install (into the backup), and some security scan flagged it, so it was "removed". This is a perfect example of what I warned folks about when all this was coming up, and some felt they "had to remove the jar". I said: you will rue the day if you forget to put that jar back in before doing the uninstall. I really think that's likely what's happening here. Let's see.

Ah, yes, that makes a lot of sense @carehart! I do vaguely recall some people mentioning that security scans flagged log4j 1.2 JARs in backup folders...

That's a reasonable theory. I don't remember going to that extreme, particularly on this machine. It's a build box, so it's used exclusively for compiling. Coldfusion isn't even running usually.... But I'm not about to discount it completely.

Which of those 5's is meant to be 3?

lol.... the first one. i'll edit.

Names of JAR files do not matter, only their contents. You don't want two versions of the same library on your classpath.

copy the log4j files from my working update 3 machine to my broken "upgraded to 5, and downgraded to back to 3, but has different log4j files" machine. Thinking that there's a version difference with those files that didn't get restored in the uninstall because they weren't present in the backup folder.

OK, that sounds plausible. Just the two version-numbered JARs? Although it would be interesting to know if the non-versioned file was restored correctly from the backup folder (i.e., is the working U3

log4j.jar

the same as the U5-to-U3

log4j.jar

version).

well, that didn't work. 😞 At this point, I think I'll just reinstall because I've spend more than enough time on it. But thanks for the ideas.

"didn't work"? The exact same error? And you only copied the two versioned JARs, not the unversioned one that got backed up? I wonder if there was some other library version change that depended on log4j 1.2 that also didn't get backed up / restored? (I guess that's me musing aloud since you're going down the reinstall path at this point)

yes, to all of those things... I tried the 3rd jar that did get backed up after the other 2, just in case, before giving up, and reinstalling.

Ugh.

I still stand by asserting this was likely about fiddling with the jars to exceed Adobe's dealing with log4j. I do realize you've run out of road/steam/time, Tim. Perhaps all this will help others if they may find it. BTW, Sean, that would be a yes to there being manifest files in most cf jars, having version info, from those I've observed. Sadly, not all. It's catch as catch can.

@carehart I didn't really mean the MANIFEST inside the JARs, I meant a file from Adobe stating which versions of which library JARs they actually ship, since they rename the JAR files to omit the version numbers (which makes upgrading easier since you only have one JAR for each library, not one for each version of each library). Hence "manifest" in quotes, as in, a manifest of the contents of the shipped product.

Ah, no. No manifest of that sort, at all, that I have ever seen. As some may know, when I do my "hidden gems" talk for each release, I trawl through the jars to find versions to report for folks, but it's neither official nor 100% reliable. Of course, some people are using "software bom" (bill of material) tools to analyze apps--which is harder with a complex app like CF, though not impossible. It may be easier with the CF container images that Adobe offers (for each of CF2023 and earlier versions, and a new one for each update). And Docker and other tools also offer security scanning tools which at least report on vulnerable library versions, which is better than nothing. But no, nothing from Adobe that I know of.

Thanks. I have a feeling this is something Lucee provides in the distro. Or at least did at some point. Maybe it would make sense for Adobe to start doing this? @Mark Takata (Adobe) Unless there are security reasons for not disclosing version numbers in the distro? Or maybe Adobe is shipping their own patched versions of some JARs?

@Tim i have a follow-up on this, which may or may not help you, but perhaps it may help others finding this. I have confirmed now that the errors you see are ones which appear in the logs during startup if you had previously tried to apply the CF update using the JVM from July of this year or the one released today, and that update had failed because of it. That would be Java 11.0.20 or 11.0.21 (for CF2021) or 17.0.8 or 17.0.9 (for cf2023). As some know, I have a blog post from July and from last week about this issue, discussing that if you change CF to use either of these later JVMs, then the install of the update will fail. (Same if you run it from the command line and the JVM there is one of these newer ones). And as I explain in my blog post, the solution is to run the JVM update from the command line, adding a new JVM arg added with those July JVM updates:

-Djdk.util.zip.disableZip64ExtraFieldValidation=true

Again, see the post for more. Now, I didn't think of this earlier today because you said that your update install log showed 0 errors. If oen tries to do an update with these newer JVMs but without that special argument, the install log WILL show you getting fatalerrors. So I didn't think this could be your issue. But I happened to look more closely at my startup logs after doing this intentionally, and sure enough the CF2023 log has the very

Unable to install Logging package: java.lang.NoClassDefFoundError: org/apache/logging/log4j/core/Layout

error that you quoted (as well as others). The CF2021 instead didn't have that

core/Layout

but it had some like

Unable to install Logging package: java.lang.NoClassDefFoundError: org/apache/logging/log4j/Logger

So again, while it may NOT be your situation (given the other oddities, like the 1-line out log, and being able to reinstall update 5), I share this as much for others (and to put a totally different take on the log4j aspect), and in the hopes that maybe it triggers an idea on your end.

Yikes, good detective work @carehart -- and, man, you've put a lot of time into this! Thank you! I see this version sensitivity and I really cringe. Since I left the CF world for the Clojure world, we're used to blithely updating the JDK every time there's a new version and it's exceedingly rare that we run into problems (a memory leak in early JDK 19 builds if

--enable-preview

was specified is about the only thing I can think of right now). Right now at work, we're on the latest JDK 20 everywhere and we're already testing on JDK 21 (we can't upgrade CI because compiling on JDK 21 won't always run on JDK 20, because of the new

SequencedCollection

interface that 21 added, which leaks into bytecode pretty easily and doesn't exist on 20 -- but even that version sensitivity is rare in the Java/JVM languages world as a whole, and only affects precompiled bytecode, not compile-on-demand).

Sorry, was slammed with people all day (totally unrelated issues). This is the life of a cf "batman". 🙂 So first, thanks. Second, as for this new jvm "sensitivity", I'll point out that the JVM change (from July) which is affecting CF this way is not limited to CF only--other vendors were burned by it. I discussed that at the bottom of my July post, with a list of posts from others about such other products or projects being hampered. Here's a link to that specific section. Again, Adobe is working on a fix for CF. It was not included in this most recent CF update from a week ago Friday, but they say (in the update technote) that it's coming in the next update...and the sooner the better! Finally, yep, I realize many moved to openjdk's. To be clear, Adobe still licenses Oracle Java for use with CF. Indeed FWIW, this issue that hit CF (due to the change in the July JVM update) happened with openjdk's as well (having nothing to do with CF, I mean). Here's the openjdk issue tracker for it, which is from last month, more pointing out how the same change in the July openjdk updates affected folks and how this fix is an option, and also listing tools that had to make changes to resolve issues they hit.

The change in the jdk affected a relative handful of products and libraries that were not producing valid zip/jar files and have mostly been fixed quickly.

understood on both points. the problem was not resolved by all of them within days, necessarily. i was just commenting that it was more widespread than "just cf".