is there a chance that you've a thundering herd problem?

like a backoff induced flood?

no im directly setting --server based on time of day so its an even distribution

there's a chance that at a specific time frame more agents hit the server than usual

okay

do you run https://github.com/puppetlabs/puppet_operational_dashboards to get some metrics?

yep

how does requested/available jruby instances look like?

here's a recent one

it seems catalog compilation is really laggy at first

that really looks like a thundering herd problem 😄

yeah the problem is if this happens without a server restart its fine

you should distribute your agent requests more

this is what it looks like if the server didnt do a restart

are all your agents coming in this 20min window?

also you said 'moving load back'. is that a load balanced setup? when do you move the agents back? I guess too early

i have 2 servers that i migrate between every 30 minutes, at :15 and :45. primary server runs a script that can reload crl/restart puppetserver at :00 and secondary does it at :30, this way there is a 15 minute window before and after the potential server restart to give agents enough time to finish their runs.

agents are hard coded to each server based on this time frame using a script

agents run hourly according to fqdn_mod_by distribution

and within each minute of that hour they're distributed by the second according to fqdn_mod_by

its a very good distribution (i looked at one point)

so all agents check in within 1 hour

14000 agents

acording to your graph it's quite bad 😄

200 of them are on a 10 minute run interval though (canary env)

lol

you do over 28k catalog compilations with two servers, each 30 jruby instances?

that bad graph is what it looks like when the puppetserver container has restarted... the 2nd graph is the same flip but when cache is intact

14k compilations per hour, 30 jrubies yes

so on the second graph it looks like your agents do the first requests at ~8:15 and finish at 8:45, but nothing before/after