We’ve been trying to track down an issue we’re seeing on one of our production servers, where we’re unable to use certain Java security providers due to a java.lang.SecurityException: “Invalid signature file digest for Manifest main attributes”. We’ve tracked this problem down to the bcprov-jdk18on-1.78.1.jar file that is downloaded from https://cfmodules.adobe.com/cf2023/repo/bcprov-jdk18on-1.78.1.jar when installing various bundles. From what we can tell, this Adobe-provided JAR file differs from the JAR published by Bouncy Castle and does have an invalid signature. Is there a reason that Adobe’s hosted version differs from the published version? Can we safely replace the invalid, Adobe-provided JAR with the valid, Bouncy Castle-published JAR?”

Any idea why only one server has this issue? Do they all have the same jar? Maybe the jar itself isn't the problem.

I believe there are tickets in the ColdFusion Tracker to update these but not sure when that will happen...

Given that the file is invalid, why does only one server have this symptom? Do they all have the same jar? Is there some other factor in the environment that makes it work when it shouldn't, or vice versa?

Hi, @Dave Merrill, I work with @John. Thanks for your questions.

Any idea why only one server has this issue?

The code path that results in the

JceSecurity.getInstance

invocation that verifies security provider JARs and ultimately results in the SecurityException being thrown is only invoked on one production server. We're unable to copy that code to other production servers to test if it's a configuration issue specific to that one server. We're in the process of replicating the production server's configuration in a test environment. But even if we replicate it, and even if replacing the invalid JAR with a valid one fixes it, we still want to understand why the Adobe provided one is modified from Bouncy Castle's so we can make an informed decision about whether to touch production.

Given that the file is invalid, why does only one server have this symptom? Do they all have the same jar?

That's an excellent question, and we're hoping that, if we can replicate the problem in a test environment, we'll be able to figure that out. Our development servers don't exhibit this behavior in spite of the faulty JAR being present. But the development servers have development-oriented configuration settings.

Is there some other factor in the environment that makes it work when it shouldn't, or vice versa?

That's still an open question. We noticed that the invalid JAR becomes placed under

bin/felix-cache

. (The SHA-256 of the corresponding

bundle.jar

matches the

bcprov-jdk18on-1.78.1.jar

under

bundles/repo

and that's hosted on cfmodules.adobe.com.) Our initial impression from that finding was that something had gone wrong in Felix or that the JAR simply became corrupt somehow. So, we stopped CF, purged the Felix cache and started CF back up. Our code that relies on the BouncyCastle provider worked again until the next CF restart, at which point the previous behavior resumed. Our assumption at this point is that when CF starts with a cold Felix cache, a different, valid bcprov JAR (that lives in a

lib

folder on the classpath) takes precedence and is registered as the "BC" security provider. For the lifetime of that JVM instance, the valid provider is the one that is used. After a restart, the invalid bundle takes precedence and is registered as the "BC" security provider. When JceSecurity attempts to verify that provider, the corresponding JAR is correctly identified as invalid and the exception is thrown. But that's just a barely-educated assumption and may not at all be what's happening. And, again, we don't yet have a clear idea of why this would affect prod but not dev. The only thing we've been able to definitively ascertain is that, of all of the signed JAR files under our CF folders, only the Conscrypt--which none of the bundles we use depend on--and BouncyCastle JARs are invalid. So, we figured it was decent lead to follow. Regardless, we'd also like to understand why the Adobe-provided JARs are modified so we can understand the implications of replacing them with the upstream vendor's version. (For what it's worth, the only change from Bouncy Castle's JAR is that Adobe's removes the line

Require-Capability: <http://osgi.ee|osgi.ee>;filter:="(&(<http://osgi.ee|osgi.ee>=JavaSE)(version=1.8))"

from the

MANIFEST.MF

file -- which, of course, changes the manifest's hash and causes the JAR to be marked as invalid.) We're more than open to exploring other possible causes if you have suggestions!

Interesting, seems like you're doing an excellent investigation, don't think I can really add anything beyond my drive-by questions. I do wonder why you can't copy the meat of the code that uses that jar to the other prod servers, and invoke with a temporary test file. Worst case maybe create some test code that basically only tries to load that jar, to see if it throws that security exception, try on on your different servers. But I'll leave you to it, you're clearly on top of it. Annoying situation, condolences 🙂

Thanks for giving it some consideration, at least! We're all ears if you have any ideas come to you!

I do wonder why you can't copy the meat of the code that uses that jar to the other prod servers, and invoke with a temporary test file.

Not my servers, not my call. 😕