Puppet Community

The default group resource won’t remove users - it can only add users. The gpasswd module adds calls to gpasswd on platforms that support setting the members with a list which includes most Linux. No idea why that has never been pulled into the default handler.

I think because it's anti-pattern for Puppet DSL. One should manage what group members should be, not who shouldn't be there

Well you can’t manage a group at all if you don’t have a way to remove members 

If you enforce groups' membership, all extras will be removed, I must be missing something

oh. I would file a bug, that is a core functionality

The core module doesn’t support it.  The purpose of the gpasswd module was to add it.  We’ve already created an issue on the SIMP github as they have apparently taken ownership of the module, but my post here was to see if anyone else is actually using it.

If you are somehow able to set groups with the core module I’d love to see it.

Specific error is “gpasswd does not support features manages_local_users_and_groups; not managing attribute forcelocal”.  The code seems to add that feature but I can’t see why puppet isn’t picking that up.

Well nevermind - was told out of band that module is now OBE and that core can do group management properly with the right flags.

Hey folks, here is the latest blogpost from the Content-and-Tooling team. Hope you have an amazing weekend! <https://puppetlabs.github.io/content-and-tooling-team/blog/updates/2023-04-28-status-update/>

Hiera’s `lookup()` converts the result into String afaik

i.e. you cannot use it like “puppet.%{alias()}”

though I might be wrong and this usage is allowed.. easier to try and see

you can do `"fix_string.%{alias('anotherkey')}"`

of course this makes sense with string data only and not with array or hash

additionally you need un unwrap the sensitive value.

maybe this is working: `"puppet.%{alias('k8s::server::resources::bootstrap::secret').unwrap}"`

so the unwrap function is working in hiera too?

nope, this doesn't work

```Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Syntax error in string: puppet.%{lookup('k8s::server::resources::bootstrap::secret').unwrap} on node <http://worker-0.example.com|worker-0.example.com>```
and also alias is not working
```Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: 'alias' interpolation is only permitted if the expression is equal to the entire string on node <http://worker-0.example.com|worker-0.example.com>```

You need to write a hiera backend function. This function would return the concatenated result of the two sensitive values it would look up, unwrap, concatenate and then make into a Sensitive value.

maybe you can try to `alias` into a intermediate key and then `lookup` it.. though I’m not sure if String conversion of `lookup` wouldn’t break it

Can only be done with a backend function as hiera will interpolate ”redacted” into the result.
It would not be unreasonable to modify hiera itself such that interpolation of a Sensitive would unwrap it, and then make the resulting string Sensitive. A similar thing was done to templates.

Thanks for the report, bringing it up internally.

Just as a quick followup, it seems this is an issue with old versions of Beaker. ewoud and I chat about it in <#C0W1Y5VL0|>. The key part starts around <https://puppetcommunity.slack.com/archives/C0W1Y5VL0/p1682690200588239> and is basically that this is an issue that will be cleaned up on the Vox Pupuli side.

No, puppetserver does not yet support RHEL9.  That's not until Q3.  Where in the release notes do you see it?

Ah, I have it backwards. The release notes omit it, sysreqs include it. Let me check to ensure I'm not looking at the _agent's_ reqs ...

Ah, yep. Just the agent. <https://www.puppet.com/docs/puppet/8/system_requirements.html#system_requirements> has a tiny line under the heading that I missed.

IMO, the "Packaged platforms" header should be where 'puppet-agent' is called out, not the line below.

Or, I could read the larger heading just above it ... :laughing: :facepalm:

I'm guessing the server reqs portion of that doc still needs updating since it refers the reader to v7 instead of including a list of supported OS releases.

right - my thoughts exactly.  I guess I can bash it and use exec

I tried while ago but there was an error in Puppet CA which is fixed these days.. didn’t try after

well.. actually I tried to use Vault PKI to make the intermediate CA and import it to the Puppet CA

ah, we're thinking of using vault directly as the intermediate ca

and not have any puppet ca infrastructure

though when you have a Vault PKI you don’t need the Puppet CA

but.. you should manage puppet certificates yourself then

I have this in my TODO but didn’t try so far

we've been able to sign-verbatim a csr from puppet with the trusted extensions, but getting the puppetservers to actaully accept the cert has been the hard part

I should try this :slightly_smiling_face:

drawback here is that you should distribute the agent certs and CA cert yourself

yeah, pain in the butt especially since we're trying to have both current ca and new certs work side by side

```# Item 1
Mar 24 10:37:45 ipp-puppet-03 puppetserver[1685762]: [2645.635s][warning][codecache] CodeHeap 'non-profiled nmethods' is full. Compiler has been disabled.
Mar 24 10:37:45 ipp-puppet-03 puppetserver[1685762]: [2645.635s][warning][codecache] Try increasing the code heap size using -XX:NonProfiledCodeHeapSize=
Mar 24 10:37:45 ipp-puppet-03 puppetserver[1685762]: OpenJDK 64-Bit Server VM warning: CodeHeap 'non-profiled nmethods' is full. Compiler has been disabled.
Mar 24 10:37:45 ipp-puppet-03 puppetserver[1685762]: OpenJDK 64-Bit Server VM warning: Try increasing the code heap size using -XX:NonProfiledCodeHeapSize=
Mar 24 10:37:45 ipp-puppet-03 puppetserver[1685762]: CodeHeap 'non-profiled nmethods': size=520292Kb used=520089Kb max_used=520291Kb free=202Kb
Mar 24 10:37:45 ipp-puppet-03 puppetserver[1685762]:  bounds [0x00007fd7f7bdf000, 0x00007fd8177f8000, 0x00007fd8177f8000]
Mar 24 10:37:45 ipp-puppet-03 puppetserver[1685762]: CodeHeap 'profiled nmethods': size=520292Kb used=498346Kb max_used=499213Kb free=21945Kb
Mar 24 10:37:45 ipp-puppet-03 puppetserver[1685762]:  bounds [0x00007fd7d7fc6000, 0x00007fd7f7bdf000, 0x00007fd7f7bdf000]
Mar 24 10:37:45 ipp-puppet-03 puppetserver[1685762]: CodeHeap 'non-nmethods': size=7992Kb used=1619Kb max_used=4017Kb free=6373Kb
Mar 24 10:37:45 ipp-puppet-03 puppetserver[1685762]:  bounds [0x00007fd7d77f8000, 0x00007fd7d7bf8000, 0x00007fd7d7fc6000]
Mar 24 10:37:45 ipp-puppet-03 puppetserver[1685762]:  total_blobs=142670 nmethods=141774 adapters=802
Mar 24 10:37:45 ipp-puppet-03 puppetserver[1685762]:  compilation: disabled (not enough contiguous free space left)
Mar 24 10:37:45 ipp-puppet-03 puppetserver[1685762]:               stopped_count=1, restarted_count=0
Mar 24 10:37:45 ipp-puppet-03 puppetserver[1685762]:  full_count=0
# Item 2
Apr 02 02:17:56 ipp-puppet-03 puppetserver[1685762]: [750255.667s][warning][codecache] CodeHeap 'profiled nmethods' is full. Compiler has been disabled.
Apr 02 02:17:56 ipp-puppet-03 puppetserver[1685762]: [750255.667s][warning][codecache] Try increasing the code heap size using -XX:ProfiledCodeHeapSize=
Apr 02 02:17:56 ipp-puppet-03 puppetserver[1685762]: OpenJDK 64-Bit Server VM warning: CodeHeap 'profiled nmethods' is full. Compiler has been disabled.
Apr 02 02:17:56 ipp-puppet-03 puppetserver[1685762]: OpenJDK 64-Bit Server VM warning: Try increasing the code heap size using -XX:ProfiledCodeHeapSize=
Apr 02 02:17:56 ipp-puppet-03 puppetserver[1685762]: CodeHeap 'non-profiled nmethods': size=520292Kb used=456061Kb max_used=520291Kb free=64230Kb
Apr 02 02:17:56 ipp-puppet-03 puppetserver[1685762]:  bounds [0x00007fd7f7bdf000, 0x00007fd8177f8000, 0x00007fd8177f8000]
Apr 02 02:17:56 ipp-puppet-03 puppetserver[1685762]: CodeHeap 'profiled nmethods': size=520292Kb used=465697Kb max_used=510728Kb free=54594Kb
Apr 02 02:17:56 ipp-puppet-03 puppetserver[1685762]:  bounds [0x00007fd7d7fc6000, 0x00007fd7f7bdf000, 0x00007fd7f7bdf000]
Apr 02 02:17:56 ipp-puppet-03 puppetserver[1685762]: CodeHeap 'non-nmethods': size=7992Kb used=3863Kb max_used=4036Kb free=4128Kb
Apr 02 02:17:56 ipp-puppet-03 puppetserver[1685762]:  bounds [0x00007fd7d77f8000, 0x00007fd7d7bf8000, 0x00007fd7d7fc6000]
Apr 02 02:17:56 ipp-puppet-03 puppetserver[1685762]:  total_blobs=118789 nmethods=117889 adapters=802
Apr 02 02:17:56 ipp-puppet-03 puppetserver[1685762]:  compilation: disabled (not enough contiguous free space left)
Apr 02 02:17:56 ipp-puppet-03 puppetserver[1685762]:               stopped_count=794, restarted_count=793
Apr 02 02:17:56 ipp-puppet-03 puppetserver[1685762]:  full_count=812```

&gt; compilation: disabled (not enough contiguous free space left)


I know surprisingly little about how JVM works...

```-Xms2g -Xmx8g -XX:ReservedCodeCacheSize=1g -Djruby.logger.class=com.puppetlabs.jruby_utils.jruby.Slf4jLogger -XX:+CrashOnOutOfMemoryError```
If you need more info, please let me know.

how many jruby instances did you configure?

max-active-instances in /etc/puppetlabs/puppetserver/conf.d/puppetserver.conf usually

max-active-instances: 16

```CPU(s):                          40
On-line CPU(s) list:             0-39
Thread(s) per core:              2
Core(s) per socket:              10
Socket(s):                       2
NUMA node(s):                    2
Vendor ID:                       GenuineIntel
CPU family:                      6
Model:                           79
Model name:                      Intel(R) Xeon(R) CPU E5-2640 v4 @ 2.40GHz

root@ipp-puppet-03:~# free -h              total        used        free      shared  buff/cache   available
Mem:           62Gi        21Gi       535Mi       0.0Ki        40Gi        40Gi
Swap:         127Gi        10Mi       127Gi```


```-Xms16g -Xmx16g -XX:ReservedCodeCacheSize=2g -Djruby.logger.class=com.puppetlabs.jruby_utils.jruby.Slf4jLogger -XX:+CrashOnOutOfMemoryError```

I've got a massive amount of nodes and a huge catalog. FYI

• ReservedCodeCacheSize supports a maximum of 2G. always use that
• plan for at least 1G of heap per jruby instance.
• if you have many environments or huge catalogs, switch to 1.5-2G per instance
• Always set Xms and Xmx to the same value

what's your definition of "massive amount"

Good question. PuppetDB shares this info. Is that useful?

finally a bigger environment to play with :smile:

most people seeking advise here run smaller setups,  less fun for me :disappointed:

I was better at tuning puppet servers back when I ran it under mod_rails. Yes, that long ago :sweat_smile:

so avg resources are 541 resources per nodes. that's okay. I would go with 1G per jruby instance for now

I highly highly recommend running <https://forge.puppet.com/modules/puppetlabs/puppet_operational_dashboards>

that's the new monitoring standard for puppetserver/puppetdb performance. with the dashboards tuning is way easier

that will also tell you how much of your heap is used, if you've memory leaks. if you've puppetdb/postgresql performance problems

do you have any performance problems or did you just notice the errors in the log you posted?

I get a "rare" (about 150 agents at any given moment out of ~14000) throwing

```Failed to generate additional resources using 'eval_generate': SSL_connect returned=1 errno=0 state=unknown state: sslv3 alert certificate unknown```
But might could be my haproxy server in front hitting a limit? It's doing dumb tcp hand off.

so usually the required CPU power for haproxy can be delivered by a toaster or tamagotchi, at least in layer 3/4 mode

could be that haproxy has a bottleneck, but that error usually appears when a puppetserver doesn't answer fast enough or not at all

So 4 servers. Every server is a full puppet master. Simply share certs via NFS...

```ipp-puppet-04:/var/nfs/puppetssl  117G   25G   87G  23% /etc/puppetlabs/puppet/ssl
ipp-puppet-04:/var/nfs/puppetca   117G   25G   87G  23% /etc/puppetlabs/puppetserver/ca```


how is the CA configured? Do all of them act as CA?

All are a CA. I've heard rumors that we can do a separate CA server but never looked into it.

yeah. so there's a chance that two puppetservers want to update the serial or CRL file, which will corrupt them

because of that it's common to have a single puppetserver acting as CA only

so setup a puppetserver, small box. you can set ca_server in puppet.conf in your agents to point to that box

use an srv record that points to that box

What kinda size do you think I need for CA server?

or! let haproxy terminate tls, check for the requested url and redirect the 2 CA endpoints to that single box

benefit of this: you don't need the NFS mounts

so a common pattern is a "MoM" or master of masters

that's a puppetserver that acts as CA for everybody and the puppet agents on the other puppetservers talk to that MoM

and then you need a single jruby instance and 2GB ram maybe. really tiny

I've this haproxy acl that matches for urls with `puppet-ca`, those are routed to a MoM, all other requests are loadbalanced to a normal puppetserver (also called compiler). but that won't work in layer 3/4 mode

there's a chance that your puppetserver are just overloaded sometimes. the error you posted indicates that

maybe the ssl error correlates with that. if you setup the operational dashboard I linked, it's easy to  identify

because you will see the jvm max heap size, how much it's used, and the amount of requests jruby instances (a.k.a agents hitting the server)

and a thundering herd problem is common where agent runs aren't distributed evenly over time which load to such errors. but also your heap is way to small

is your haproxy doing least connection or round robin?

it's worth testing least connection. that usually works better. but I suggest to setup the dashboard first

Will do. Gives me something to run with. Thanks a ton!

Enable the systemd service :slightly_smiling_face:

use theforeman-puppet module to manage it and use systemd timer to run it periodically

See the `theforeman-puppet` module at <https://forge.puppet.com/theforeman/puppet?src=slack&amp;channel=puppet>

```  systemd::unit_file { 'puppet-run.service':
    enable  =&gt; $enable_boot_run,
    content =&gt; epp("${module_name}/puppet-run.epp"),
  }```


there is option in the module to do this (IIRC this is default mode though I’m not 100% sure)

we use splay normally, and that `oneshot` service during boot

<@U1PC2RAH0> that worked like a charm, thanks

All the other responses are appreciated too

I recommend managing the puppet agent and its config with theforeman/puppet

<https://forge.puppet.com/modules/zack/exports/readme> is your friend

or query the PuppetDB API. It knows about exported resources

See the `zack-exports` module at <https://forge.puppet.com/zack/exports?src=slack&amp;channel=puppet>

wow, all great minds :slightly_smiling_face: Thanks!

Oh, “This module is compatible with Puppet Enterprise and Puppet Bolt.” I hope it’s not the case.

just making sure it can work with the open source version

Personally, I prefer puppetdb_query to exported resources

you don't have to :slightly_smiling_face:

zack-exports will install a face on the server, so you can examine exported resources by logging to puppet server

bolt lets you to do it from your workstation

sorry, i feel like a total idiot.  Is this what you’re talking about:
```bolt task run exports resources=Host --nodes <http://master.corp.net|master.corp.net>```


will it work on it’s own or do i need zack-exports?

<https://forge.puppet.com/modules/zack/exports/tasks>

Thanks!  got what I needed.  Appreciate your help.

sorry, our manager got us pizza for lunch.  My brain is… blah.

I guess a follow up question.  If i’m upgrading all of our servers to a new puppet server/version, what is the best way to figure out an order to make sure i “keep” all exported resources…?  I hope my question makes sense.

exported resources are stored in the puppetdb

or in the postgresql behind the puppetdb service

Yes, understood. I am upgrading puppetdb as well.

do it without dropping  the database then you're fine

but i’m “starting fresh” with the new database.  Should I export/import data from old to new databases?

if you want to keep your data you should do that :smile:

you can dump the postgresql data and restore it.

but for that you need matching puppetdb versions

so setup a second system, install same version, dump on old, restore on new etc

if you don't care about your histroy, you can trigger puppet runs with --noop to populate the new database

that might show some changes due to missing exported resources

yes that’s what i’ve been doing to see the changes

if every node does a second noop run, there shouldnt be changes because exported resources are back

so should i run --noop on ALL nodes (against new puppet server)  and then do an actual upgrade?

usually an older puppet agent can talk to newer puppetserver/puppetdb

so you could setup a new puppetserver/puppetdb, move agents to new infra, do puppet agent upgrades

no i’ve noticed they are failing, but i can upgrade agents, not an issue.

i already have new puppetserver/puppetdb.  puppet-agent (old ones) have been failing with the new infra, so i’ve been updating them (agents)

Can I just say how amazing this channel is and how grateful I am for everyone’s help.  Seriously.  Thank YOU.

so this is going to sound really dumb, but ... does the yaml parser in hiera "recognize" date/time values somehow? i'm getting different results between
```param_name :
  start_time : 2022-01-01 00:00:00```
and

```param_name :
  start_time : "2022-01-01 00:00:00"```

Yeah, those should both be strings. What difference in result are you getting?

without the quotes, the defined type which receives the values as parameters is receiving an empty string as the value (the `start_time` parameter is just declared as `String`)

i.e. there's a class which receives the `param_name` hash as a parameter, and it calls the defined type as
```  xxx::type { $x :
    * =&gt; $param_name ,
  }```

(after making sure all the keys in `param_name` actually _exist_ as parameters in the defined type)

i'm okay with telling the people who write the hiera files to always quote that value, mostly i just want to understand _why_ this is happening.

also, it's entirely possible that this is an issue with puppet 4.5.2 ... we're stuck on PE 2016.2.0, i've been trying to upgrade for several years but i'm the only puppet dev and this is only _part_ of my job, so i haven't had time to convert things over to work with r10k and the other new sexy

Very possible. 4.5 is early enough that bugs were still being worked out of the Puppet 4 type system.

i'm willing to accept that, because it adds to the list of reasons for which i _really need_ to be given the time to upgrade (also i need to edit that ticket, change "PE2018" to "PE2021" since that's the newest LTS version)

There is some attempted conversion to `Time` occurring with latest 7.x:

```# puppet --version
7.25.0

# cat /etc/puppetlabs/code/environments/production/data/common.yaml
---
foo :
  start_time : 2022-01-01 00:00:00
bar :
  start_time : "2022-01-01 00:00:00"

# puppet apply &lt;&lt;'EOF'
$foo = lookup('foo')
notice("foo is: ${foo}")
notice("Type of foo is: ${foo.type}")

$bar = lookup('bar')
notice("bar is: ${bar}")
notice("Type of bar is: ${bar.type}")
EOF

Error: Unable to parse (/etc/puppetlabs/code/environments/production/data/common.yaml): Tried to load unspecified class: Time```
Likely occurring on 4 as well. So, "if it is a string, quote it" seems like a good rule to follow.

YAML has some other oddities that make that a good rule as well. For example, `yes` and `no` are coerced to booleans if not quoted.

some of them like to be cute, i.e. "the yaml spec says strings don't always have to be quoted", like saving two bytes in a file is a huge victory ... i just tell 'em "you don't need to convince me, you need to convince hiera", even though i know when they run into it they're just gonna come back to me with "why doesn't this work"...

Yeah, if two bytes per string buys you consistent behavior, it's worth the price.

magic is convenient up until the point where it costs you a chunk of debugging time.

It did take quite a few scars before I learned that lesson though :wink:

anyway, thanks for answering, much appreciated. i'm calling it, have a good weekend