Attempt number, I've already stopped counting: moving from puppet7 server to openvox8. Narrowed down performance woes of version 8 (also affects puppet8) to the Java running just hitting OOM every few minutes and restarting.

May 20 2251 puppetserver_err_pid992969.log
May 20 2253 puppetserver_err_pid1006954.log
May 20 2254 puppetserver_err_pid1017281.log
May 20 2258 puppetserver_err_pid1029901.log
May 20 2301 puppetserver_err_pid1061443.log
May 20 2304 puppetserver_err_pid1088613.log
May 20 2306 puppetserver_err_pid1109684.log
May 20 2309 puppetserver_err_pid1125754.log
May 20 2311 puppetserver_err_pid1143964.log
May 20 2319 puppetserver_err_pid1164164.log
May 20 2322 puppetserver_err_pid1224264.log
May 20 2323 puppetserver_err_pid1246039.log
May 20 2325 puppetserver_err_pid1257227.log
May 20 2328 puppetserver_err_pid1273593.log
May 20 2330 puppetserver_err_pid1292134.log

Have doubled the memory on these servers from 16->32G, tried incrementing

-Xms

and

-Xmx

from 2G up to 24G, higher memory assignment just makes it worse as it completely stalls (GC suspending all threads, I assume). Similarly tuning reserved code cache size alongside. Doubled vcpu count 6->12, different max-active-instances between

4

and

12

, switched model from x86-v3 to v4 instruction set, doesn't help. Tried differents GC's

-XX:+UseZGC -XX:+ZGenerational

to no observed effect. Trebled server count from 2->6 severs, behind a load balancer to even out the management of some ~500 nodes, server still on its knees. Meanwhile 2x Puppet7 servers running 7.17.3 "just work" with the same load hitting them. They are spec'd with: 6 vCPU, 16GB mem,

-Xms2G -Xmx2G -XX:ReservedCodeCacheSize=512M

,

max-active-instances: 4

, and Openjdk 17. I'm just going to walk away now as it's very, very late here. But I am beginning to think my codebase is cursed against puppet 8, or just blame everything on Java 21. Only last thing I can do is increase the

runinterval

on the agents to reduce the chance of thundering herd from the agents. 🤷