Puppet Community

<https://github.com/theforeman/puppet-puppet/blob/16.2.0/manifests/server/config.pp#L193-L213>

can be path to a file, which can be a policy implementation

Sure. Use the `runtimeout` setting:

<https://puppet.com/docs/puppet/6/configuration.html#runtimeout>

Thanks <@U11MDJNGZ> I found it already :-) thanks again 

check this: <https://forge.puppet.com/modules/puppet/cron>

also consider using a systemd timer because that's teh cron successor :slightly_smiling_face:

which works with the puppet/systemd module

See the `puppet-systemd` module at <https://forge.puppet.com/puppet/systemd?src=slack&amp;channel=puppet>

Any attribute to create cron jobs under /etc/cron.d  ?

<https://github.com/voxpupuli/puppet-cron#cronjob>

My code looks like this
cron { 'sa2':
    command = '/usr/lib64/sa/saw -A',
    user           = 'root',
    hour          = '23',
    minute      = '53',
}

```cron::job { 'sa2':
    command =&gt; '/usr/lib64/sa/saw -A',
    user           =&gt; 'root',
    hour          =&gt; '23',
    minute      =&gt; '53',
}```

+1 for systemd timers
<https://forge.puppet.com/modules/themeier/systemd_cron>
:grin:

right, that needs to be incorporated to puppet/systemd

because now one need to manage two components and it's a pita

Good idea. I will send a pr when I find some time

<https://themeier.github.io/themeier-systemd_cron/|https://themeier.github.io/themeier-systemd_cron/> in the README.md is dead :(

Oh. Didn't notice. But the reference.md should be ok

```cron::job { 'sa2':
    command =&gt; '/usr/lib64/sa/saw -A',
    user           =&gt; 'root',
    hour          = &gt;'23',
    minute      =&gt; '53',
}```

I'd have a look on the node's catalog first :slightly_smiling_face:

but looking for what? There are at least no resources that I could drop

but a huge amount of functions. they might eat up the memory

`puppet agent -tdv` and look when the memory usage on the same time ?

If that use lot of time too, you have `/opt/puppetlabs/puppet/cache/state/last_run_report.yaml` with execution time

so cumbersome with &gt; 2000 resources and &gt;5min agent runtime :smile:

you should see the ones with 6000 resources

look the yaml and use `evaluation_time` key when it's &gt; to 10sec, that can be help you maybe

I ran it through one of the ruby profilers once but it wasn't all that helpful. Ideally, puppet itself would instrument tracing. BUT, you can also dump `--evaltrace` and see if you've got any long processes piling up.

You might also be able to hook `Benchmark` around some of the guts and print out what object is being processed prior to the application.

I can assume that stuff that runs long also consumes a lot of memory, but that doesn't need to be the case

actual profiling would be a really cool feature :disappointed:

I think <@U11HA7VJ7> might have been tinkering with that at one point but it would be AMAZING to be able to trace all the way through the server and into the clients.

168 exec resources with &gt;2 minutes runtime in the report summary could be an indicator for performance issues :smile:

but again I would assume that the agent would free up memory if one resource used it and is done

Pretty sure the objects hang out until the end of time. One of the reasons we only run via scheduled tasks

I migrated to that but it's still so much memory for a single run

I proposed a stack/heap model way back in the day that would have remedied that but it didn't go anywhere

Do you have any recursive file objects/tidies? That could do it

I mean, have you seen the oracle installers? :face_vomiting:

A profiler might be useful to see if you're spawning bajillions of objects of a given type

you mentioned the awesome evaltrace option. do you know how I should interpret the resource count it prints? I've ~4k resources accourding to the option, resources.txt lists ~1.3k. are the missing resources in other branches, like if conditions/where statements and do not apply to the node?

I assumed the puppetserver evaluated the code and won't add those to the catalog

there is a 120s sleep in an exec resource during the oracle installation...

you use `--tags` for look if this really an oracle issue

It's only what's being applied at the time so anything not compiled in won't show (but also won't affect your client RAM usage)

I'm at 2G memory usage acording to ps_mem, before the first resource is applied :thinking_face:

that's during the `info: caching catalog for $fqdn`

Running `facter -p` by its self and measuring the high-water mark of that can help split the problem space a bit.

Otherwise, it’s difficult. Best approach is probably to line a timestamped log of memory usage samples up with a timestamped log of `--debug --evaltrace` messages to see where the growth picks up.

OpenTracing would be great, but won’t provide visibility into memory usage on it’s own. It would have to be paired with some specific instrumentation using Ruby’s `ObjectSpace` module or something similar.

do you by any chance know if someone at Puppet works on that / would it help if a PE customer submits a feature request for this?

Noone working on it at the moment that I know of. A feature request would definitely be welcome.

Hrm..might want to check Jira, could have sworn there was something in there

At some point I had the agent emitting profiling info using the `opentracing` ruby library. It's in a feature branch somewhere, but needs to be rebased due to other agent changes

The existing profiling info is just timing data though. We’d need to add something new to capture memory usage.

sure, for that you can use the memory_profiler gem (we use that for our benchmarks) like <https://github.com/puppetlabs/puppet/blob/c0f692088789217a35486714de2e9d53bfaf6622/tasks/benchmark.rake#L105-L118>

OpenTrancing is a heck of a good idea as a standardized reporting format. But, it needs to be fed with data to report.

sounds like a good friday afternoon hacking session :D

sooo I did some further debugging on the agent with the huge memory usage. I did `rm -rf /opt/puppetlabs/puppet/cache/state/*` and kicked off 7 runs so far, none of them consumed more than 500mb according to ps_mem.

now it would have been helpful if I just moved it instead of rm. not sure if there was any old file that wasn't updated on a puppet run or any file where runs are appended. but usually that's not the case, so I cannot explain why the memory usage dropped. I've a few more nodes with high memory usage. I will check those next week

My guess would be `/opt/puppetlabs/puppet/cache/state/state.yaml` was large. That stores the last state of every resource enforced in the past month, which enables `audit` and corrective vs. intentional change reporting.

The retention period can be adjusted:

<https://puppet.com/docs/puppet/7/configuration.html#statettl>

oh I thought only for the last run. thanks!

We fixed the "store state forever" issue in <https://tickets.puppetlabs.com/browse/PUP-3647> By default `statettl=32d`  but can be shortened if needed

this is puppet 6.25.1, so it should be at 32d. I will check next week if we use schedule anywhere and reduce the ttl

hey people. Anybody debugged Puppet Agent memory usage in the past? I'm running the agent as a systemd timer and it uses up to 4gig of memory during a run. seems to be a bit high. There is probably an awesome way to profile this

look the yaml and use `evaluation_time` key when ith &gt; to 10sec, that can be help you maybe

how I supposed to achieve this with puppet module?

iterate on the array, execute each query and figure out an sql statement to identify if they were already executed. or create a lock file for each query

or investigate if puppet is the right tool for this

I did not choose puppet for just this task, this process is just part of a dns server setup and Puppet is  a perfect tool for this )

I see you have written a lot of modules, what is the best way of handling credentials in modules? I do not want to push them to git. We have vault too

how do you handle credentials personally?

most people use eyaml or vault as hiera backends for encrypted data

can you provide any module implemented one of them? preferable vault.

<https://github.com/petems/petems-hiera_vault|https://github.com/petems/petems-hiera_vault> its just a hiera backend

```Traceback (most recent call last):
        13: from /opt/puppetlabs/puppet/bin/facter:10:in `&lt;main&gt;'
        12: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/facter/framework/cli/cli_launcher.rb:23:in `start'
        11: from /opt/puppetlabs/puppet/lib/ruby/gems/2.7.0/gems/thor-1.1.0/lib/thor/base.rb:485:in `start'
        10: from /opt/puppetlabs/puppet/lib/ruby/gems/2.7.0/gems/thor-1.1.0/lib/thor.rb:392:in `dispatch'
         9: from /opt/puppetlabs/puppet/lib/ruby/gems/2.7.0/gems/thor-1.1.0/lib/thor/invocation.rb:127:in `invoke_command'
         8: from /opt/puppetlabs/puppet/lib/ruby/gems/2.7.0/gems/thor-1.1.0/lib/thor/command.rb:27:in `run'
         7: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/facter/framework/cli/cli.rb:124:in `query'
         6: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/facter.rb:458:in `to_user_output'
         5: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/facter/framework/formatters/legacy_fact_formatter.rb:16:in `format'
         4: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/facter/framework/formatters/legacy_fact_formatter.rb:26:in `format_for_no_query'
         3: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/facter/framework/formatters/legacy_fact_formatter.rb:63:in `hash_to_facter_format'
         2: from /opt/puppetlabs/puppet/lib/ruby/2.7.0/json/common.rb:286:in `pretty_generate'
         1: from /opt/puppetlabs/puppet/lib/ruby/2.7.0/json/common.rb:286:in `generate'
/opt/puppetlabs/puppet/lib/ruby/2.7.0/json/common.rb:286:in `to_json': source sequence is illegal/malformed utf-8 (JSON::GeneratorError)```


I suspect cloud-init is doing something as I cannot replicate this when I launch an instance without using it.

```amI9ቷj(mg*W'&gt;q{q~}P0{^v^P٠( ] DEBUG Facter::FactManager - fact "ec2_userdata" has resolved to:UQo6}p?@hIکH,     IZ4ހ&lt;u%sHRl?~ -'Ndsk
"!-hD'#enFO&amp;٥R)]h!I&amp;\vnT߉{ݓ^tΜǀ
e$۪nzv{~98yKeV%1R-ڎ`s"Wp&gt;95st\
}5=zr6aJH慂#/+-X-|&lt;;,pnzv}}]Wns&amp;iFYcz\uIC@b5H!c,
                                       t߾Iz24*kA5^T +Y&lt;X\%T0*9.x=_U$:@l3#EigA@EDKNcci[HuHfi՘ű&lt;-&amp;f2t?-xl`:5ɮOvajV8&gt;l7: owgIf'h^@pJSfR2.Y!@
}B񻣣kE EqzɅhXc^h*Ҷ,Yx
fo
  }1v
     ٠3kP&amp;C9zƗW|?(B
Bwfe&lt;cVzc&gt;&lt;/gL```

It looks like it doesn't like the binary/base64 encoded ec2_userdata cloud-init uses.

Fixed with `/etc/puppetlabs/facter/facter.conf`
```facts : {
  blocklist : [ "ec2_userdata"]
}```

Is this the sort of thing that warrants a bug report/Jira ticket or is the work-around I found the correct solution?

<https://tickets.puppetlabs.com/browse/FACT-3120|FACT-3120> created. :crossed_fingers::skin-tone-3:

Thanks <@U7NHPPYSJ> Does facter 3 display the fact (as base64)? or does it omit the fact?

It displays the encoded data
```# /opt/puppetlabs/puppet/bin/facter ec2_userdata
amI9ቷj(mg*W'&gt;q{q~}P0{^v^P٠(?~ -'Ndsk
"!-hD'#enFO&amp;٥R)]h!I&amp;\vnT߉{ݓ^tΜǀ
e$۪nzv{~98yKeV%1R-ڎ`s"Wp&gt;95st\
}5=zr6aJH慂#/+-X-|&lt;;,pnzv}}]Wns&amp;iFYcz\uIC@b5H!c,
                                       t߾Iz24*kA5^T +Y&lt;X\%T0*9.x=_U$:@l3#EigA@EDKNcci[HuHfi՘ű&lt;-&amp;f2t?-xl`:5ɮOvajV8&gt;l7: owgIf'h^@pJSfR2.Y!@}B񻣣kE EqzɅhXc^h*Ҷ,Yx
fo
  }1v
     ٠3kP&amp;C9zƗW|?(B
Bwfe&lt;cVzc&gt;&lt;/gL```


```# /opt/puppetlabs/puppet/bin/facter -v
3.14.23 (commit 46e6e6ffdecff72b75853af43a11acdf2d36752c)```


How about  `/opt/puppetlabs/puppet/bin/facter --json | grep ec2` ?

```# /opt/puppetlabs/puppet/bin/facter --json | grep ec2
      "serial_number": "ec214863-280f-126f-0aa8-32f824cca8ac",
      "uuid": "ec214863-280f-126f-0aa8-32f824cca8ac"
  "ec2_metadata": {
    "hostname": "ip-10-10-4-7.ec2.internal",
      "ec2": {
    "local-hostname": "ip-10-10-4-7.ec2.internal",
            "local-hostname": "ip-10-10-4-7.ec2.internal",
    "domain": "ec2.internal",
    "fqdn": "i-09a69a.ec2.internal",
Binary file (standard input) matches```


Never mind. v4 throws the traceback without expressly blocking ec2_userdata

What would cause this on a clean Alma Linux 8 server (EC2)?

Stack trace moved to :thread:

What would cause this facter error on a clean Alma Linux 8 server (EC2)?

Stack trace moved to :thread:

Hi Everyone,

cem_linux module, `cem_linux::utils::packages::linux ::ssh:` Applies ssh configuration on all the hosts,

And, I wanted to apply 2 different values only for 3 specific nodes.

For ex: 
I want to change 
AllowTcpForwarding and max_startups
On 3 servers only

Any suggestions for this?

Bumping this, for great justice, and also because I feel like I’m losing my mind with not being able to figure out where a particular virtual resource is getting realized in my codebase.

To be honest, that’s one big reason I’ve stayed far, far away from virtual resources. Too much spooky action at a distance.

Same, but one doesn’t always get to chose the quality of one’s codebase.

Do you have a stack trace for that error?

i dont. I found the issue after 2 of us spent about 8 hours on this. it turned out to be a misplaced capital letter in the lookup that went unnoticed by two eyes.

but it begs the question as to WHY this doesn't provide any better information ( like what property is the null showing up in ) .

but what i think it actually boils down to is an issue with hashes. not the actual property. i bet this error is exposed far before the property is being set.

$facts['hash'] vs. $facts['Hash']

the crappy part is it provides no line number or anything. just the resource name followed by that error.

so you know its in there, but its not clear where or why.

How is that hash used? Are you calling create_resources function? Also, does the error occur when compiling or when applying?

That is also why I asked for a stack trace, to make it possible to figure out where in the puppet code base the error comes from.

its compile time i think.

do you have a link on how to do stack traces?

I would like to write up a better way for our team to troubleshoot this one. its not the first time I chased this dragon

The stack trace is in the log for that compilation.

i.e a Ruby stack trace, showing where ”in Ruby” the error was raised.

I am not at all a ruby expert. i am running this on windows and I am only aware of what comes out of the event log when I run puppet agent or puppet apply

<@U012DPETBKM> could you run `puppet agent -t --trace` from the `Start Command Prompt with Puppe*t*` option?

```Error: /Stage[main]/Main/Dsc[COSCRole-GenetecAuthRole]: Could not evaluate: unsupported type NilClass of value ''
C:/ProgramData/PuppetLabs/puppet/cache/lib/pwsh/util.rb:131:in `format_powershell_value'
C:/ProgramData/PuppetLabs/puppet/cache/lib/puppet_x/puppetlabs/dsc_lite/powershell_hash_formatter.rb:18:in `format'
C:/ProgramData/PuppetLabs/puppet/cache/lib/puppet_x/puppetlabs/dsc_lite/powershell_hash_formatter.rb:36:in `block in for
mat_hash_to_string'
C:/ProgramData/PuppetLabs/puppet/cache/lib/puppet_x/puppetlabs/dsc_lite/powershell_hash_formatter.rb:36:in `each'
C:/ProgramData/PuppetLabs/puppet/cache/lib/puppet_x/puppetlabs/dsc_lite/powershell_hash_formatter.rb:36:in `map'
C:/ProgramData/PuppetLabs/puppet/cache/lib/puppet_x/puppetlabs/dsc_lite/powershell_hash_formatter.rb:36:in `format_hash_
to_string'
C:/ProgramData/PuppetLabs/puppet/cache/lib/puppet_x/puppetlabs/dsc_lite/powershell_hash_formatter.rb:24:in `format_hash'

C:/ProgramData/PuppetLabs/puppet/cache/lib/puppet_x/puppetlabs/dsc_lite/powershell_hash_formatter.rb:14:in `format'
C:/ProgramData/PuppetLabs/puppet/cache/lib/puppet/provider/base_dsc_lite/powershell.rb:125:in `format_dsc_lite'
(erb):36:in `block in ps_script_content'
(erb):35:in `each'
(erb):35:in `ps_script_content'
C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/2.5.0/erb.rb:876:in `eval'
C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/2.5.0/erb.rb:876:in `result'
C:/ProgramData/PuppetLabs/puppet/cache/lib/puppet/provider/base_dsc_lite/powershell.rb:152:in `ps_script_content'
C:/ProgramData/PuppetLabs/puppet/cache/lib/puppet/provider/base_dsc_lite/powershell.rb:143:in `ps_script_content'
C:/ProgramData/PuppetLabs/puppet/cache/lib/puppet/provider/base_dsc_lite/powershell.rb:67:in `exists?'
C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/property/ensure.rb:82:in `retrieve'
C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/type.rb:1115:in `retrieve'
C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/type.rb:1143:in `retrieve_resource'
C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:307:in `from_reso
urce'
C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:20:in `evaluate'
C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:267:in `apply'
C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:287:in `eval_resource'
C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:191:in `call'
C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:191:in `block (2 levels) in evalua
te'
C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/util.rb:546:in `block in thinmark'
C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/util.rb:545:in `thinmark'
C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:191:in `block in evaluate'
C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:122:in `traverse'
C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:178:in `evaluate'
C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:240:in `block (2 levels) in a
pply'
C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/util.rb:546:in `block in thinmark'
C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/util.rb:545:in `thinmark'
C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:239:in `block in apply'
C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/util/log.rb:165:in `with_destination'
C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/transaction/report.rb:152:in `as_logging_destinat
ion'
C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:238:in `apply'
C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:196:in `block (2 levels) in apply_c
atalog'
C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/util.rb:546:in `block in thinmark'
C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/util.rb:545:in `thinmark'
C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:195:in `block in apply_catalog'
C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/util.rb:233:in `block in benchmark'
C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/util.rb:232:in `benchmark'
C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:194:in `apply_catalog'
C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:412:in `run_internal'
C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:251:in `block in run'
C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet.rb:310:in `override'
C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:221:in `run'
C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:361:in `apply_catalog'
C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:273:in `block in main'
C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet.rb:310:in `override'
C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:212:in `main'
C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:182:in `run_command'
C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/application.rb:382:in `block in run'
C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/util.rb:735:in `exit_on_fail'
C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/application.rb:382:in `run'
C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:143:in `run'
C:/Program Files/Puppet Labs/Puppet/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:77:in `execute'```

not seeing anywhere in there where it points to the line / call where the error occurred but i might be missing it.

That looks like an error when applying the catalog, not when compiling. The problem being that the powershell ”dsc_lite” tries to operate on a hash with nil/undef value.
Suggest logging a ticket to at least get a better error message.