Puppet Community

That is _extremely_ helpful information. At first glance it looks like it stops responding until it does that CRL check, but in reality, the CRL check _itself_ is running late (it should happen every 60 seconds, but it happens 128 seconds after the previous check). So that just happens to be the moment the service “wakes up” and carries on processing _anything_.

ah, I didn't notice that CRL was late as well

I did further debugging on the node with our internal monitoring and there was zero IOWait, cpu load was normal. I couldn't spot anything on the box that would effect the performance of the pcp-broker

a full java GC could maybe cause this? but I don't know if the clojure code is even effected by GC. the puppetserver_gc didn't list any long full GCs.

I saw one GC that was like 11 seconds, but that’s much less than the pause in the logs

I think gceasy said there were two in total, one with 8s and one with 11s. but I'm really bad at reading those logs

Given the pcp-broker on the compiler lives in process with puppetserver, it’s possible that something puppetserver is doing is causing that behavior. I wonder if the puppetserver logs *also* indicate a lengthy pause at that same time. That might tell us whether it’s a java issue (GC, whatever) or a pcp-broker specific issue.

There wasn't anything, it was just compiling catalogs as usual

let me check if there was also a pause, but I don't think so

I got access to the logs. It _did_ pause at the same time

oh we submitted puppetserver.log as well? good

but hard to tell if there was no activity or if it was actually sleeping

It looks like the pause was 89 seconds long

I see a GC that took almost 70 seconds. It’s really tough to correlate times between the GC log and the regular log, but I assume that’s it

I have a meeting now, but I’ll keep looking at it today

CC <@U17J6MKV5>, we did the debugging together

I am very curious what caused the behavior 