@Justin Wyker, similar setup, about 3500 agents, one server. Around the time we crossed the 3000 agent threshold, I had the same problem. I set max-active-instances to

CPU - 1

and heap size to

1GB + (max-active-instances * 1GB)

and everything ran perfectly smoothly from then on. This is on a 16 CPU/ 64 GB RAM server. I could potentially downsize it after those tunings, but if it ain't broke...