Is the "shape" of memory usage on 5.4 different from 5.3? IE: does it intrinsically use a bunch more, or grab a lot more up front or anything? This could be either in Lucee, or something about the changes to the docker image, I guess. I just rolled out a change from 5.3.9.166-nginx to 5.4.1.8-nginx in our UAT environment, and it refused to start with:

#
# There is insufficient memory for the Java Runtime Environment to continue.
# Cannot create worker GC thread. Out of system resources.
# An error report file with more information is saved as:
# /usr/local/tomcat/hs_err_pid21.log
2023-07-19 07:57:26,585 INFO exited: lucee (exit status 1; not expected)
2023-07-19 07:57:29,591 INFO spawned: 'lucee' with pid 25
[...]
[0.006s][warning][os,thread] Failed to start thread "GC Thread#0" - pthread_create failed (EPERM) for attributes: stacksize: 1024k, guardsize: 4k, detached.

In the docker logs. We had no such issue in dev, ad the only change in this release was bumping the Lucee version in the Docker image. The container doesn't even start, so it's not like it's due to unexpected load or anything like that. NB: I've only just started to look at this now, so this is very early in my troubleshooting, but figured I'd raise it here and now in case someone goes "oh yeah, that's because [reasons]"

nothing really that major has changed between 5.3.10.120 and 5.4.1.8, aside from java libraries, there was a lot since 5.3.9, but if the container doesn't even start, that's weird... @justincarter any ideas?

I found this https://stackoverflow.com/questions/72841549/container-fails-to-start-insufficient-memory-for-the-java-runtime-environment-t interesting bit around Ubuntu versions vs version of docker being used i don't know whether this is relevant but while you wait for more informed answers

@zackster nothing in the catalina log. I mean literally nothing. I guess it was not getting sufficiently through start-up to log anything. The pid logs mentioned in the docker log listing are no help cos they're written to the container filesystem, so... aren't persisted across container rebuilds (which is what we do when a container becomes unhealthy)

which specfic docker image?

As per above: 5.4.1.8-nginx

just trying to spin up that image on windows

yeah the linked GitHub issue that Alex linked does not fill me with confidence

that image starts for me

That looks like a Docker Desktop version to me. It's the server version that's relevant here. 20.10.5, anyhow. Which is before the version mentioned in the Stack Overflow answer. Have asked our hosting provider what's involved in getting that up to date. Probably can do that overnight, and try again tomorrow. I wonder if I can roll back my local from 24.x to 20.x somehow...

those images are prewarmed when they build, so they do start sometimes....

Interestingly I've seen a similar thing today using 5.3.10.97 and base image

ortussolutions/commandbox:3.6.3

though it's intermittent.. my favourite kind.

7/24/2023, 5:35:55 PM GMT+10	OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000a0000000, 1610612736, 0) failed; error='Not enough space' (errno=12)
7/24/2023, 5:35:55 PM GMT+10	#
7/24/2023, 5:35:55 PM GMT+10	# There is insufficient memory for the Java Runtime Environment to continue.
7/24/2023, 5:35:55 PM GMT+10	# Native memory allocation (mmap) failed to map 1610612736 bytes for committing reserved memory.
7/24/2023, 5:35:55 PM GMT+10	# An error report file with more information is saved as:
7/24/2023, 5:35:55 PM GMT+10	# /var/www/hs_err_pid6.log