Almost every morning (likely after the application...
# cfml-general
n
Almost every morning (likely after the application times out and resets on its own, not via a CF service restart or other manual intervention), our site crashes. The error is either: •
java.lang.ClassCastException
• or
class coldfusion.runtime.AttributeCollection cannot be cast to class java.lang.String (coldfusion.runtime.AttributeCollection is in unnamed module of loader coldfusion.bootstrap.BootstrapClassLoader @56a4f272; java.lang.String is in module java.base of loader 'bootstrap')
The
ClassCastException
has absolutely no other info in the stacktrace. No file or line number. The
AttributeCollection
error is breaking in an
init()
in a CFC. The CFC has a
property name="foo" type="cfc.models.bar";
and the failing line in the
init()
is
variables.foo = new cfc.models.bar();
. Bar.cfc's
init()
takes no arguments, so the
AttributeCollection
doesn't really make any sense. This all appears to be a race condition. If I get this error once, it never goes away until I restart the CF service or make a trivial change inside Bar.cfc like changing a method with
return something.booleanFormat()
to
return booleanFormat(something)
. (This is just one example... other trivial changes generally work too.) Once it's working, it never fails again until the next morning... again, likely after the application times out and CF automatically reinitializes it. Anyone know where to even start with something like this? I'm in ACF 2023, and I recently updated to jdk 17.0.14 with no change.
a
As a way of mitigating against this, can you up the timeout for your application ?
n
For sure.
a
As to the cause - very hard to say. Do you call onApplicationStart manually (so people do with a manual
?resetmapp=true
flag or similar) which is breaking the locking?
n
No
a
Is 'cfc/models' a mapping?
n
No
šŸ¤” 1
b
Hey Nick, you might need to reach out to Adobe. My assumption is the JDK you updated.
n
I have. They thought I should bandaid it: Turn off component caching, or manually clear the .class files when it happens. No offer of assistance with the root cause. Re: the JDK, we have the same issue on multiple servers, with 17.0.12, 13, and 14.
The CFCs are using inheritance, composition, abstract CFCs, etc. So, I know I'm pushing the technical envelope a bit more than the average CF'er (based on 20+ years of experience with multiple CF shops). I imagine something in CF is buggy, but because this issue is random, and buried in a complex architecture, it's a bit difficult to produce a bug report.
a
Any static variables in there?
n
No
a
Just wondering if it's a metadata thing
abstract functions or abstract components?
n
components
a
We've got similar stuff but not seen that error (just searched our logs). Hmm.
šŸ¤” 1
c
Nick, I have a few thoughts. Since the problem seems complex, bear with me while I share a few thoughts/questions here. 1) Since the problem happens on other servers (running cf2023 but with different versions of java 17), I see how you could think it's a seeming bug in CF. But clarify for us: did the problem happen before you moved to CF2023? (Has it been happening for years, perhaps, and you've just put up with it until now?) I press because whenever the problem started, it could be that some code change you implemented (then) has led to this. What code change? We can't possibly guess. 2) And if it can "go for weeks without the error" and then suddenly happen, you may be on to something with the fact that editing the code in question or restarting CF "fixes" things. CF will have stored in its template cache a compilation of your CFML that was based on the state of things when the request first ran after any previous edit. (Do you happen to turn off "save class files" in the CF Admin? If so, then CF also recompiles it on each restart.) There could be some configuration that's in play at the time CF compiles the template in question, which may be different at another time. 3) Then again, you say this problem just happens out of the blue each morning. I suppose you've confirmed that there's no change to the template in question...but then technically it could be that it's about some code that template includes, or CFCs it instantiates, and so on. We can get into the weeds here fast, of course. 4) One last thing, and it may be important to your consideration of things: you say that you wonder if the "every morning" nature of the occurrence might be "after the application times out and resets on its own". Let's clarify first that even if your applicationtimeout was set to 24 hours, that does NOT mean that the application will terminate itself after 24 hours. Like sessiontimeouts, these timeouts reset with each request (to that session or that application), so you'd have to have zero requests for 24 hours for the application to timeout. 4a) If you doubt that (or if I might be proven wrong, or if perhaps things have changed since I've been assuming that), note that you can (and should) add a cflog or writelog to your onapplicationstop method. That will prove when that happens. 4b) And to follow up on the earlier mention of people sometimes calling onapplicationstart manually, note that that does not technically reset the application: it merely re-runs whatever code is in that method. More to my last point, the onapplicationstop method is NOT called in such a case. 4c) Indeed, sometimes it can be helpful to add logging to the onapplicationstart...specifically to find out when (and perhaps how surprisingly often) such calls are being made to that method. You may want to log also whether it was being done due to such a manual call. 4d) It may even help you to see if your error might be immediately preceded by such a call to that method, which may be triggering some unexpected bahavior which leads to your error scenario. But that's just a guess. šŸ™‚ Let's hear what you may think or find.
n
Charlie, Thank you for taking the time to dig into this with me. To answer/respond to your questions/recommendations: 1) We built the site last year, so we've only tried it in CF2023. It's been happening on every hotfix (9-12) and Java version (17.0.12-17.0.14) since we started. 2) It usually fails every morning. Our application timeout is about 4 hours. We have "Component cache" and "Save class files" checked. 3) There is definitely no code change. We have 3 environments. DEV and UAT are on the same server. PROD is on its own. UAT is the one that fails most mornings. Oddly DEV and PROD only fail once a month or so. DEV has multiple daily updates to the code. But UAT and PROD are updated maybe 2-4 times per month. 4) I've bumped up the application timeout to 1 day. I'd say if by this time next week, we haven't seen the issue, that would confirm the application-reinit theory. I'll give an update next week. 4a) I've added the cflog to onApplicationEnd 4b) Makes sense. (We don't have anything that manually calls onApplicationStart().) 4c) Added logging to onApplicationStart 4d) Makes sense! I'll keep you posted on what happens!
c
Will be really interesting to hear what you find next. :-)
šŸ‘ 1
b
Copy code
@Nick: Our application timeout is about 4 hours.... Almost every morning (likely after the application times out and resets on its own,...
I think that that is the cause of the issue right there. Apparently, no one uses the application for long periods during the night. If no user interacts with the application within 4 hours, then the application will time out. It will no longer be active after that. Application timeout means that, after 4 hours, there is, in effect, no application to talk about. To borrow from Monty Python's Dead-Parrot sketch, the application "is no more", "has ceased to be", "bereft of life, it rests in peace" and "is an ex-application". Hence it won't have the resources to reset on its own . That would explain why you see little to nothing in the logs and why a restart of the application is necessary. If you want the application to live longer, then you will have to increase the applicationTimeout value.
An idea. It might perhaps help to do some cleaning up in Application.cfc's onApplicationEnd event-handler. For example, something like this:
<cffunction name="onApplicationEnd">
<cfargument name="ApplicationScope" required=true >
<cfscript>
var adminObj = createObject("Component", "cfide.adminapi.administrator");
adminObj.login("your_cf_admin_password");
var cfAdminRuntimeService = createObject("component", "cfide.adminapi.runtime");
// Clear caches
cfAdminRuntimeService.clearQueryCache();
cfAdminRuntimeService.clearTrustedCache();
cfAdminRuntimeService.clearComponentCache();
</cfscript>
<cflog file="#this.name#" type="Information" text="Application #this.name# has just ended">
</cffunction>
c
Thanks for the update, @Nick. As for fr, yes the defaults may suffice. But yes, there are additional settings you can consider. And I have a video on it in my yt fr playlist, https://youtube.com/playlist?list=PLG2EHzEbhy09al6Lx0JlIUcpbA8UrwPuo. See specifically "configuring Fusionreactor for action". While they're all a few to several years old, everything still applies. I do want to update them, both for the somewhat-changed ui and for new features. Looking forward to hearing what you find with what you're doing. And if you get to "just wanting it solved", or want help making best use of FR etc, I offer consulting in as little as 15 min increments and a satisfaction guarantee: you won't pay for time you don't find valuable. Carehart.org/consulting
n
Thanks @carehart. I'll check out that video, and we will definitely keep your consulting services in mind. Update: One thing FR has revealed already is that we have a client pinging our site once/minute. Therefore, I'm thinking it wasn't a bug in my logging in onApplicationEnd... It's that it never ends. Unfortunately for now, that's leaving me without a working theory. But hopefully FR will provide additional clues.
c
Good to hear, and yes, fr is so helpful. And yep, as I'd said in point 4 of my original reply I suspect for most people their apps don't reach their timeout. It takes only one request to reset that countdown. :-) Let's see how things pan out for you.