https://www.puppet.com/community logo
Join Slack
Powered by
This message was deleted.
# puppet
s
This message was deleted.
b
@csharpsteen maybe that's something that interest you
y
I’d blame autorequires but just guessing 🙂
b
yeah probably
but why 😄
an how
creating directories isn't anything special and done in thousand places
so what can be so special that it goes OOM
y
is it just 1 node or reproducible in multiple places?
b
I currently don't have a second node where I could test this
y
my idea was to try with different puppet versions.. e.g. with latest 7.x
b
this isn't an nfs mount or something, it's a virtualbox instance just with a local disk
y
ah, so you can snapshot/backup it
just curious if valgrind can be used with ruby 🤔
b
"it's complicated" 😄
it's a VM running on a workstation from another developer in another building and I only have ssh access into the VM
🙈 2
my options are.. limited
y
memory_profiler at least.. though the article is a bit old
just to be sure.. /opt is not managed recursively, right? 🙂
and btw.. puppet agent is in /opt/puppetlabs, right?
b
yes it's in /opt/puppetlabs. official AIO packages from Puppet
I don't think /opt is managed. I suspected something with recurse => true as well. let me disable the file resource to get a working run and then check resources.txt
y
Just found
Copy code
➜  ~ find /opt/puppetlabs | wc -l
   94220
ah.. nvm.. it’s bolt and pdk there.. I have no puppet agent there
b
the paren't directories don't have recurse => true up to 
/opt
, which isn't managed by puppet. I will remove the directory resources and just try it with the file resource
j
I'd 
strace -f -e trace=file /opt/puppetlabs/puppet/bin/puppet agent -t > agent.log 2>&1
b
will give that a try next week
I attached strace to the running puppet process and only saw 
brk()
a lot
and I think I found the issue, but cannot explain it. let me hack an example
https://gist.github.com/bastelfreak/8c727a72042dc0e31698e5f4ee27a4bc that's heavily simplified. A coworker added the 
/opt/foo/first/something/file.txt
file resource and then puppet agent ate up all ram
I reviewed it and noticed that 
purge => true
and 
recurse => true
doesn't make sense for 
/opt/foo/second/something
because 
/opt/foo/second
already has it
so I removed 
purge => true
and 
recurse => true
. But I also had to remove the 
/opt/foo/first
and 
/opt/foo/first/something
resources. only then I can create 
/opt/foo/first/something/file.txt
and that doesn't make sense to me
mhm I need to check if I can create 
/opt/foo/first
and 
/opt/foo/first/something
if I also remove purge/recurse/force from 
/opt/foo/second
I assume there are some really weird side effects from other resources
aha! so a bit of progress here. originally the resources from my gist are spread about multiple puppet classes. Moving the 
/opt/foo/first
file resource to another class fixes the memory problem 🤔
c
purge => true
and 
recurse => true
are both prime suspects for causing this sort of issue. Both of those parameters add resources to the catalog and if they end up walking over a large directory tree, then they could easily add so many resources that memory becomes exhausted and the kernel OOMkiller awakens from it's slumber to punish the hubris of mankind.
b
yeah 😞
but some debug output would be helpful
Open in Slack
PreviousNext
Powered by