Hello Guys, Currently I have a problem with sudden...
# lucee
c
Hello Guys, Currently I have a problem with suddenly dying instance and I have no idea which process make this instance down. In FusionReactore, I can't find any information or log, and i dont even see long request too here. FYI, currently I use lucee 5.3.9.166 with docker-commandBox with image jdk11-3.6.3 When I upgrade to lucee 5.3.10.97 the the dying instance is worse, I get dying instance more often. Maybe someone here has an experience on this case? Thanks
z
first up, update to the latest java 11, i think that's a pretty old / early release https://adoptium.net/temurin/releases/?version=11
second,
dying
is the kinda error report client provide, what's the first question you'd ask as a developer to such a report?
👍 1
what error are you seeing in the logs?
c
Hi @zackster I dont see any log or error, suddenly the instance down without any info or log. Right night Im still investgate on this by downgrade 1 by 1 and see what is the problem here.
z
so it's stops responding to http requests, or do they hang but the process is still running, any cpu usage?
if the the server is still running, can you grab a thread dump https://www.baeldung.com/java-thread-dump
c
the instance is really stop, and in fusion reactor, the instance is really gone. FYI, yesterday I downgrade my lucee using 5.3.9.141 and so far the instance look stable. I will monitor with this lucee version in a week.
@zackster I found the last log before the instance down, maybe you familiar with this ?
I really no idea what process which caused that error
@zackster after I monitor in a week lucee 5.3.9.141 look very good and I never get dead instance anymore. It seem the problem since versuin 5.3.9.166
but till now I still have no idea which process caused the instance dead
z
you are also one release behind the latest stable commandbox. out of interest, do you have multiple servers instances on the same server?
c
@zackster sorry I just notice your message, FYI I use commandbox 5.70 and we use multiple instance in aws, so at the busy period, the instance can be 10-12
I have a feeling the problem with commandBox when I use commandbox 5.7 with lucee 5.3.9.141 I have this problem again
I can make sure now, the problem is not in lucee, but in commandBox 5.7 with commandBox 5.6 is fine
z
I think @bdw429s was working on this, Lucee made some changes to the osgi / felix config which affected how commandbox re-uses lucee as a downstream project
b
If we're referring to the error above, I don't see anything in there that indicates it's related to OSGI at all.
In fact, I don't even know what that error is related to at all! The stack trace doesn't really even show if this is part of Lucee or what. The exception is
Copy code
javax.xml.parsers.FactoryConfigurationError: Provider for class javax.xml.parsers.DocumentBuilderFactory cannot be created
Caused by: java.lang.RuntimeException: Provider for class javax.xml.parsers.DocumentBuilderFactory cannot be created
Caused by: java.util.ServiceConfigurationError: javax.xml.parsers.DocumentBuilderFactory: Error accessing configuration file
Caused by: java.io.FileNotFoundException: /tmp/com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl (No such file or directory)
which indicates file in the
/tmp/
is inaccessible. I'd look at permissions.
Also, it's worth saying here that "Correlation doesn't mean causation". Just because this was the last error in the logs, doesn't mean it is in any way related to the instance going down at some later point.
@cubortea Really needs to get FusionReactor in play so when his server goes unresponsive, he can look under the hood and see what it's doing.
z
yeah, this problem only happens randomly under commandbox, i wonder if it's related to this? https://luceeserver.atlassian.net/browse/LDEV-4119
b
To be honest, I've never seen that error myself so I wouldn't know if it happens.
Now, there have been other issues in the past that invovle Jaxb and classloading issues, but just because someone gets an error that mentions XML doesn't mean they're all related
z
i've seen it with the old axis extension in prod before
c
now I'm going to try my server with cimmandBox 5.8
FYI, with commandBox 5.8.0 I still have the same problem with 5.7.0 the instance is suddenly died with no reason, and also I cannot find any info in fussionRector because the instance is died suddenly. and in th elast request history, there is no long request which make CPU or Memory high too.
so for now I rollback again my commandbox using 5.6.1
I saw the killed log in the last log
another log maybe can help too:
1678780810972,"*** java.lang.instrument ASSERTION FAILED *: ""!errorOutstanding"" with message transform method call failed at ./src/java.instrument/share/native/libinstrument/JPLISAgent.c line: 873" 1678780816028,[INFO ] runwar.server: ****************************************************************************** 1678780816034,[INFO ] runwar.server: * starting 'stop' listener thread - Host: 0.0.0.0 - Socket: 39273 1678780816035,[INFO ] runwar.server: ****************************************************************************** 1678780816038,[INFO ] runwar.server: Server is up - http-port:8080 https-port:443 stop-port:39273 PID:7 version 4.8.3 1678780825466,[ERROR] runwar.context: [ERROR] LogFactory: Load of custom hashtable failed 1678781178299,[ERROR] runwar.context: [ERROR] LogFactory: Load of custom hashtable failed
b
@cubortea lease remove your FR license key and any passwords from the snippet above
I don't see anything out of order in those logs, they look perfectly normal. You'll need to debug what's happening on your app. I doubt it's related to the CommandBox version.
c
@bdw429s I have a feeling when there are massive request in the same time, it makes the CPU very high and make the instance down. FYI,right now I have set the worker-threads = 200
its hard for me to debug this now, because before I get the info, the instance already down
I still dont understand why in 5.6.1 is fine
I have plan to use jmeter to send massive request to test
b
@cubortea Did you configure FusionReactor to E-mail you a full thread dump when the instance starts to fail.
You need to be able to access the FusionReator details while the issue is happening
Right now, the description of the problem is just too vague and it could be anything at all
No other users have reported issues like this so I would be inclined to think it's related to your code base somehow.
c
Hi @bdw429s sorry for late reply, Im still investigate it in my application now. What make me strange with commandBox 5.6.1 I never get this problem, and all my instance very stable.
b
Right, you've been saying that for 2 months now and I get that, but you're also the only person having this issue so until you can catch some actual debugging information, we're just chasing rainbows here.