https://www.puppet.com/community logo
Join Slack
Powered by
# voxpupuli
  • c
    That does sound like a resource that is adding a lot of new generated resources to the catalog. Files with 
    recurse => true
    are the usual suspect.
  • b
  • b
    I diffed the catalog, the only diff I could see was was the ordering of profiles:b/profilesc/profiles:d. amount of resources stayed the same
  • b
    mhm autogenerated resoures would be generated on the fly by the provider/type and won't be present in the catalog?
  • c
    Pretty much. The resource present in the catalog returned by Puppet Server "expands" to generate additional resources when the agent prepares the catalog for evaulation. 
    recurse => true
    is the prototypical example as it generates a new resource for each file present a directory tree on the agent. The 
    resources
    resource is another common source.
  • b
    I think the enterprise modules also generate a lot of resources
  • b
    it would help a bit when I could get more debug output about what puppet is doing
  • c
    As for the chaining arrow, I'm not sure what's going on there. What should be happening is just one additional relationship between the end whit of 
    profile::b
    and the begin whit of 
    profile::c
    and one additional relationship between the end whit of 
    profile::c
    and the begin whit of 
    profile::d
    .
  • b
    it always freezes at a different resource, but always with 
    Adding autorequire relationship
  • b
    yes that's what I saw in the diff
  • c
    It's also quite likely this is an area where there isn't any additional debug output.
  • b
    yeah 😞
  • c
    Puppet probably drops into a loop that doesn't have any additional logging statements inside of it and ends up exhausting memory inside that loop.
  • b
    My naive thought was that it processes everything in the catalog and one of those elements triggers an expansion. but It doesn't log the elements that will be processed but only those that are processed
  • b
    so basically it feels like it's logging after the loop, not before
  • b
    but I didn't find the correct place in the soruce code where I could patch around
  • c
    Just had a thought: 
    strace
    could be useful. There might be a pattern in what the Puppet process is looking at via syscalls leading up to the OOM.
  • b
    finding the correct thing in strace is always... challenging :D
  • c
    Sometimes. If this was a 
    recurse => true
    issue, then there should be something like a ton of 
    stat()
    calls going to the same directory tree leading into the OOM.
  • c
    If that sort of repeating pattern isn't there, then that's evidence of something purely inside application memory --- like the chaining arrow somehow producing an explosion of resource relationships that consume memory.
  • b
    I assume it's the latter, I just nodd to prove that
  • s
    o/ Some flamegragh might help to see the call stack where most CPU time is used: https://www.brendangregg.com/flamegraphs.html I never tried it with Ruby, but I think I saw ways to do it.
  • c
    rbspy
    can do that, I haven't tried to use it with 
    puppet-agent
    though: https://rbspy.github.io/
  • b
    @smortex @csharpsteen even wrote about it!
    👍 1
  • b
  • b
    reads it again
  • b
    totally forgot about it
  • c
    That's using flame graphs at a different layer.
  • c
    Specifically, taking the 
    PROFILE
    logs produced by Puppet Server when compiling catalogs and transforming them into something that 
    flamegraph.pl
    can draw a picture of. So, you'd still need something like 
    rbspy
    to look at Ruby operations.
  • b
    ah thats only for catalog requests on the agent side
1...570571572...648Latest