Hello, I need help troubleshooting an issue with Lucee where the Tomcat service stops with an error of event ID of 1000 (this is from Windows Event Viewer). This has happened 4 times over the last month. I've checked the Lucee logs, Lucee web context logs, IIS Logs, Windows logs, and Tomcat logs. I have found no information as to why Tomcat stops. According to Fusionreactor, the CPU and RAM are normal before the issue happens. The level of requests per second is normal. Recent changes have been: There is a new app using complete null support. We started using static component functions. We started generating larger PDFs with more content and via merging. We started using a global libs folder to share code between applications. Unchecked the "Use time server" setting in Lucee due to the recent timestamp issue. System: OS: Windows 2022 64bit Web server: IIS 10 Lucee Info: Lucee: Lucee 5.4.6.9 (Original install 5.4.3.15) Servlet Container: Apache Tomcat/9.0.80 Java: Java 11.0.23 (Eclipse Adoptium) 64bit Null Support: Complete Null support Lucee Extension Applications: Microsoft SQL Server (Vendor Microsoft) 12.4.2jre 11 S3 Resource Extension 2.0.1.25 PDF Extension 1.2.0.10 Compress Tags 1.0.0.15 Image extension 2.0.0.26 EHCache 2.10.0.36 BonCode Settings (from the Lucee AJP13 folder): <Settings> <Port>8008</Port> <Server>localhost</Server> <MaxConnections>200</MaxConnections> <LogLevel>0</LogLevel> <LogDir>c:\temp</LogDir> <FlushThreshold>0</FlushThreshold> <EnableRemoteAdmin>False</EnableRemoteAdmin> </Settings> BonCode Settings (from the Windows folder): <Settings> <Server>localhost</Server> <Port>8008</Port> <EnableRemoteAdmin>False</EnableRemoteAdmin> <EnableHeaderDataSupport>True</EnableHeaderDataSupport> <ForceSecureSession>False</ForceSecureSession> <AllowEmptyHeaders>False</AllowEmptyHeaders> <ModCFMLSecret>xxx</ModCFMLSecret> <RequestSecret>xxx</RequestSecret> </Settings> Tomcat server.xml changed settings from install: Set address to "::1" to fix the long response time issue. Changed AJP port to "8008". An additional thing to note is that the CFML apps throw a lot of unhandled exceptions. This is not new but I thought I'd mention it just in case. I'm in the process of upgrading to the latest version of Tomcat 9 and the MSSQL extension (just testing them on a staging server). If anyone has encountered an issue like this before or has a suggestion on what I should try next, your help would be appreciated.

Seems like a memory leak of some kind. How much RAM are you giving it, and have you tried to give it some more, or change the garbage collection?

An OOM error should show up in the Catalina.out log

Have you run hardware integrity checks on the server? Online sources seem to suggest event ID 1000 is often associated with filesystem or other hardware-related problems.

And as @seancorfield alluded to, it might not be hardware related, but if you're in a Cloud environment and the file storage system is having issues, you could get the same thing. For example, we had issues in one of our datacenters once where the file storage array in their cloud infrastructure was having issues and this causes us weird issues because the file access wasn't always working correctly.

Thank you all for your input. I appreciate the ideas. The last time the issue happened was July 16 at 15:03 UTC. @denny Lucee is set with 2048MB for initial and 9216MB for max memory pool. The memory is normal before the issue happens as you can see in the screenshot below. @bdw429s There is nothing to note in the catalina.out (see attached). There are no logs for July 16 until the service is started after it stopped. I did a system-wide search for hostspot files, there were no results. @seancorfield @dswitzer The hardware is on the cloud. I will look into the hardware integrity, thanks for the idea.

Something storage related would track as you added more storage-related changes — if you have application logs maybe you can find some sort or correlation to which of the things could be causing it