I have situation where a web server is having inte...
# box-productsr
ryan
08/25/2022, 6:20 PMI have situation where a web server is having intermittent issues with the site looking like it is down from a few minutes to a few seconds where Undertow is not accepting requests. This web server is using Commandbox, Lucee, and Undertow on an Ubuntu server. I do not have access to the server, yet, to look at logs and such, but I will have them in a day so and hopefully it will have a good indication as to what is going on. Has anyone experienced any issues similar to this before?
b
bdw429s
08/25/2022, 6:29 PMNope. I'd start by working your way in from the outermost proxy. What is the exact error message/status code and where is it coming from?
r
ryan
08/25/2022, 8:07 PMThanks, Brad
p
Peter Hoopes
08/25/2022, 9:11 PMI’m having this exact same problem, so it’s good to know we’re not alone. It just stops responding for a few seconds to a few minutes…
Peter Hoopes
08/25/2022, 9:11 PMIt’s as though Undertow stops hearing incoming requests entirely, then starts back up.
b
bdw429s
08/25/2022, 9:25 PM@Peter Hoopes Fusion Reactor would code in very handy in reporting
• The current web requests
• A full thread dump
p
Peter Hoopes
08/26/2022, 12:46 AMAgreed. FR would certainly be a big help and we may have to go that route. I was also wondering about finding logs for Undertow to start looking a possible issues…
b
bdw429s
08/26/2022, 5:44 PM@Peter Hoopes The CommandBox logs are all documented here:
https://commandbox.ortusbooks.com/embedded-server/server-logs
p
Peter Hoopes
08/26/2022, 9:28 PMI knew about those, but are they for Undertow or just the Lucee engine or both?
b
bdw429s
08/26/2022, 9:28 PMUndertow itself has no log files-- it's a service layer
bdw429s
08/26/2022, 9:28 PMRunwar has logs however
bdw429s
08/26/2022, 9:29 PMAnd that page of docs covers both Runwar's and the CF engine's logs
p
Peter Hoopes
08/27/2022, 1:20 AMAwesome sauce. Thanks!
b
bdw429s
08/27/2022, 1:53 AM@Peter Hoopes Did you send me a message somewhere about the access logs? I saw the notification on my phone, but now that I'm on my computer, I can't find it.
bdw429s
08/27/2022, 2:05 AMDo you see that file on disk at all?
bdw429s
08/27/2022, 2:05 AMrun
Copy code
restart --debugp
Peter Hoopes
08/27/2022, 2:06 AMNo. In that folder are just server.out.txt logs - nothing called access.txt
b
bdw429s
08/27/2022, 2:06 AMIf access logs are enabled, you should see the following JVM args being passed to the server when it starts
Copy code
--logaccess-enable
--logaccess-basename
--logaccess-dirp
Peter Hoopes
08/27/2022, 2:07 AMNope - those are args are not present…
b
bdw429s
08/27/2022, 2:07 AMAlso, are you sure you enabled the access logs for the correct server?
p
Peter Hoopes
08/27/2022, 2:07 AMIt’s the only one I have running on this machine…
b
bdw429s
08/27/2022, 2:08 AMI see the current working dir of your shell is a
wwwp
Peter Hoopes
08/27/2022, 2:08 AMserver show
{
“app”:{
“libDirs”:“lib”
},
“name”:“Socrates49e”,
“openBrowser”:“false”,
“profile”:“staging”,
“web”:{
“accessLogEngable”:“true”,
“blockCFAdmin”:“false”,
“directoryBrowsing”:“true”,
“host”:“0.0.0.0",
“http”:{
“port”:“80”
},
“SSL”:{
“certFile”:“certs/_.standrews-de.org.2022.crt”,
“enable”:“true”,
“forceSSLRedirect”:“true”,
“keyFile”:“certs/_.standrews-de.org.nopass.key”,
“port”:“443"
},
“webroot”:“socrates49e”
}
}
b
bdw429s
08/27/2022, 2:08 AMShow me the output of
server listp
Peter Hoopes
08/27/2022, 2:08 AMSocrates49e (running)
http://0.0.0.0:80
https://0.0.0.0:443
CF Engine: lucee 5.3.9+141
Webroot: /home/phoopes/www/socrates49e/
Last Started: 26-Aug-2022 220709
Peter Hoopes
08/27/2022, 2:08 AMI do have the webroot as a subfolder on purpose
b
bdw429s
08/27/2022, 2:09 AMHmm, and you restarted the server for sure.
p
Peter Hoopes
08/27/2022, 2:09 AMMultiple times
b
bdw429s
08/27/2022, 2:09 AMCan you show me the JVM args you see?
bdw429s
08/27/2022, 2:09 AMLike, paste them here
bdw429s
08/27/2022, 2:09 AMOr screenshot them
p
Peter Hoopes
08/27/2022, 2:10 AMI know. It’s odd.
b
bdw429s
08/27/2022, 2:10 AMThat's all of them in the screenshot?
p
Peter Hoopes
08/27/2022, 2:10 AMI did notice that when this deafness happens, my VM also kills my SSH connection, so I’m wondering if it’s a bigger issue.
Peter Hoopes
08/27/2022, 2:10 AMFull list:
Runwar: args ->
/usr/lib/jvm/java-11-openjdk-amd64/bin/java
-Djdk.io.File.enableADS=true
-jar /home/phoopes/.CommandBox/lib/runwar-4.7.7.jar
--processname Socrates49e [lucee 5.3.9+141]
--log-dir /home/phoopes/.CommandBox/server/1A76B70ECEBCBA8C15DD6B6E9A621B23-Socrates49e/lucee-5.3.9.141/logs
--server-name Socrates49e
--tray-enable false
--dock-enable true
--directoryindex true
--timeout 240
--proxy-peeraddress false
--cookie-secure false
--cookie-httponly false
--pid-file /home/phoopes/.CommandBox/server/1A76B70ECEBCBA8C15DD6B6E9A621B23-Socrates49e/lucee-5.3.9.141/.pid.txt
--tray-icon /home/phoopes/.CommandBox/cfml/system/config/server-icons/trayicon-lucee.png
--debug true
--gzip-enable true
--cfengine-name lucee
-war /home/phoopes/www/socrates49e/
--web-xml-path /home/phoopes/.CommandBox/server/1A76B70ECEBCBA8C15DD6B6E9A621B23-Socrates49e/lucee-5.3.9.141/WEB-INF/web.xml
--lib-dirs /home/phoopes/www/lib/
--http-enable true
--ssl-enable true
--ajp-enable false
--http2-enable true
--open-browser false
--open-url https://0.0.0.0:443
--ssl-port 443
--ssl-cert /home/phoopes/www/certs/_.standrews-de.org.2022.crt
--ssl-key /home/phoopes/www/certs/_.standrews-de.org.nopass.key
--urlrewrite-enable false
--predicate-file /home/phoopes/.CommandBox/server/1A76B70ECEBCBA8C15DD6B6E9A621B23-Socrates49e/lucee-5.3.9.141/.predicateFile.txt
--host 0.0.0.0
--background false
--port 80
--stop-port 38379
b
bdw429s
08/27/2022, 2:10 AMThat sounds network related
p
Peter Hoopes
08/27/2022, 2:11 AMIt’s a VM in our cluster, but that’s the only other clue I have. I was trying to get these access logs up to compare to some pings, etc.
Peter Hoopes
08/27/2022, 2:11 AMBut the logaccess lines aren’t there despite the item in the server.json file
b
bdw429s
08/27/2022, 2:12 AMHmm, can you run
Copy code
server info property=accesslogenablebdw429s
08/27/2022, 2:12 AMOh, you typed it wrong
Copy code
accessLogEngablebdw429s
08/27/2022, 2:12 AMThis is why we use tab complete 🙂
bdw429s
08/27/2022, 2:12 AMRemove the extra
gp
Peter Hoopes
08/27/2022, 2:12 AMACK!!!
Peter Hoopes
08/27/2022, 2:12 AM<wiping egg off face>
b
bdw429s
08/27/2022, 2:13 AMType
Copy code
server set web.accp
Peter Hoopes
08/27/2022, 2:13 AMGot it. Me = idiot.
b
bdw429s
08/27/2022, 2:13 AMno worries
bdw429s
08/27/2022, 2:13 AMI never trust myself to spell anything correctly
p
Peter Hoopes
08/27/2022, 2:14 AMHow to remove bad property?
b
bdw429s
08/27/2022, 2:14 AM Copy code
server clear xxxp
Peter Hoopes
08/27/2022, 2:14 AM👍
b
bdw429s
08/27/2022, 2:15 AMYou'll find the same three trio of commands all over CommandBox
• xxx set
• xxx show
• xxx clear
bdw429s
08/27/2022, 2:15 AMThat's the same for
packageservercfconfigpropertyFilep
Peter Hoopes
08/27/2022, 2:15 AMRight. I love this whole system. Not trying to butter you up, but it’s awesome.
Peter Hoopes
08/27/2022, 2:16 AMFollowing the access log now. Thanks.
Peter Hoopes
08/27/2022, 3:49 AM@ryan Just wanted to let you know that I’m 96.4% sure that this is a Linux issue. I’ve enabled the logs (as you can see above) and several times as soon as my web monitor detected the site being down my SSH connection died with a broken pipe as well. My Linux-fu isn’t good enough so I’m getting help, but there could be something messed up in the networking stack, or…? Just a head’s up.
r
ryan
08/27/2022, 5:17 PMSounds good! Thank you, Peter!
3 Views