I’ve just had a good conversation with one of the ...
# lucee
p
I’ve just had a good conversation with one of the engineers at Fusion-Reactor, and in looking at my current server install he noticed a number of database threads that were stuck, notably with a “”ThreadDeath” error from the Lucee engine. He mentioned that ThreadDeath was a dangerous situation and that there is a fork of Lucee that doesn’t use it AND/OR a way to config to turn that off. Does anyone more knowledgeable than I (which would be everyone) have any thoughts on this? I should add that I’m using CommandBox and would REALLY like to continue using it rather than any kind of manual install of Lucee if at all possible!
b
@Peter Hoopes Yes, Lucee is aggressive in how it kills threads, but it also kills them in a way which can leave resources unaccounted for.
What you're looking for is not a "fork" of Lucee, it's just a setting.
Ideally, you'd have a pretty high request timeout and tune your pages so they never reach that timeout, but I understand that doesn't always happen.
lucee's request timeout setting cannot be turned off.
I believe the setting the FR engineer was telling you about is
lucee.async.request.handle
which can be set as a system property, or an env var with underscores and upper case.
It runs each request in a separate thread, which really only somewhat avoids the issues. That happens is Lucee kills the "place holder" thread, which prevents some issues like Tomcat's HTTP connection pool getting exhausted from zombie threads, but it doesn't actually • stop the real code running in the real thread • messes up tools like FR who only show you the code running in the main HTTP thread, which is just waiting to join to another thread the entire time.
Now, that said, the Pixll8 guys MAY have a fork where they've forcibly disabled request timeouts from kicking in, but I'm not sure if the FR peeps would know about that. @domwatson could fill us in on whether that existed or not.
And in case anyone is curious how I knew about this setting, there is a page of system property settings here in the docs https://docs.lucee.org/guides/Various/system-properties.html but it doesn't include this one. It is hidden in this Google doc which Micha has kept pretty up to date over the years as he added settings. https://docs.google.com/spreadsheets/d/10s-nn_FsoSD_RiLwjYZICacCoC386SjkEGT3pOfBJVU/edit#gid=0
d
Yes, discussed it informally with FR team so semi-aware at least.
👍 1
z
if you are hitting a problem like this, hammer out a minimal repo so it can be addressed
d
This was some time ago and after lengthy chats with Micha I believe both env vars added but also Lucee considerably refactored with its request timeouts. What version of Lucee was this running? And yes, we run with Lucee never attempting to detect timeouts and so never trying to kill threds.
💯 2
a
It would be nice if there was a formal way of turning it off rather than mitigating in 6 as would like to get on to formal packages
b
Agreed. Or at least, have a setting to change Lucee to give up after it interrupts the thread and not continue on to a thread.stop() (which throws thread death error). Interrupting a thread is not nearly as effective in killing it, but an interrupted thread is at least allowed to finish cleaning up any resources. In the past, Micha has said that any thread killed will always clean up its resources, but I'm still not convinced of that. One not need look far on the internet to find a plethora of advice telling you never to use thread.stop() https://stackoverflow.com/questions/16504140/thread-stop-deprecated
a
Yes i remember back in the day a conversation with Alan Williamson of Open BD and Open BD never supported request time out as it was determined it could never be something 100% reliable
b
I think it's find so long as you stick to interrupts-- those are the recommended java way to tap a thread on the shoulder and politely ask it to go. Some CF tags and Java methods (like
sleep()
) will respond (raise an interrupted exception) to interrupts, but many CFML tags and constructs will never check if they've been interrupted so it has limited use. I've had to take care in checking things like this in CommandBox CLI, otherwise Ctrl-C would be of no use 🙂
a
yeah i mean the killing rather than the interrupting
✅ 1
p
Wow… that was a lot more than I expected and WAY over my head. 😳 I’m a medium level sys admin with Lucee and CB so I’m wondering… my best bet is to change the timeout setting for the particular CF pages that are affected, or is something I can/should change on this instance of CommandBox/Lucee? Or, figure out how to update the
lucee.async.request.handle
setting? I know I have some slow DB issues that I’m working on, but I’m also trying to avoid any particular Lucee/Java Thread issues.
b
Honestly, I wouldn't change that setting unless you have a specific issue you're trying to solve. Generally speaking • set your request timeout higher if possible • Keep an eye on long requests in FR so you can address them