Puppet Community

and you could query it through puppet facts show... with a custom fact

hmmmm.
&gt; X509v3 extensions:
&gt;       1.3.6.1.4.1.34380.1.3.13: 
&gt;         ..pe_compiler

this isn't necessarily the most elegant solution, but:
```#!/bin/bash
FQDN=$(hostname).$(dnsdomainname)
CERTPATH=/etc/puppetlabs/puppet/ssl/certs/$FQDN.pem
if openssl x509 -in $CERTPATH -text | grep -q "pe_compiler"; then
  echo "is compiler"
else
  echo "is not compiler"
fi```

trusted hash is exposed to puppet, I think

and
```CERTPATH=$(puppet config print hostcert)```

I should not be relying on absolute paths

And, you can use jesse/certificate_extensions_facts module, it will convert certificate to facts

See the `jesse-certificate_extensions_facts` module at <https://forge.puppet.com/jesse/certificate_extensions_facts?src=slack&amp;channel=puppet-enterprise>

```# facter -p certificate_extension_1_3_6_1_4_1_34380_1_1_9812
puppet/server```

<https://www.puppet.com/docs/pe/2023.1/commands_endpoint.html#post_v1_commands_unpin_from_all> that api endpoint is indeed a bit slow with a few node groups

How many nodes are pinned to the group in question?  It is certainly an area that would benefit from some optimization.  That seems like an unusually long time.

All the groups must be interrogated for the "unpin from all", which can be expensive depending on how many rules a given group has.

customer has environment node groups which usually have one rule that matches a fact and then one node group per role

those groups have no rules, just pinned nodes. 120 of those groups with 5 to maybe 100 pinned nodes

I deleted some environment node groups, maybe 15. before that the api endpoint took 4min30s to cleanup a single node

do you know if the time is spent calculating the resilts of the match rules in groups?

because thats something that isnt required for unpinning?

The API isn't very smart, currently.  Also, pinned nodes are not stored in an optimal way currently.  The algorithm has to get the group rules from the database, and analyze the rules to determine if any fit the pinned node scheme.  If they do, it checks to see if they match the unpinning request.  If it matches, it rewrites the rules back to the database.  This is done for each group.  This is an area we are hoping to improve on as we recognize it is a pain point.

ah so it doesnt differentiate between actual rules and pinned nodes

Correct.  They are just "or" clauses matched on names currently.  We would like to separate them internally for a few reasons:  It will improve node classification times if we can avoid fact-checks for groups that match through the node-name, it will improve the performance and behavior around pinned nodes.

Yeah, "pinned nodes" are just a separate box in the UI. Under the hood, they're just more rule entries that match against certname.

why do you need different logic in the modules?

is that actually a requirement or do you only use it to set different default values

Maybe someone says that a server in production needs an additional file which the nonprod server does not

But yes you're right, we don't always need different logic and have modules with branches for production and nonprod just for the sake of it.
I want to go away from this principle.

so from the perspective of one of your teams, they have a production environment with important systems and a non-prod environment with less important systems

but from a puppet point of view, that's not the case.  you don't have the less-important env to test puppet code. it's used to test an application I guess?  So puppet needs to work on both environments, it just does different things

if so, I would not implement such an environment logic in the modules, but rather features flags like $manage_file_foo in the mentioned module. and you manage that parameter via hiera. it's true by default for nonprod and false for prod

that allows you to use the same branch/commit/tag of a module in both environments

and if that works you can think about if you actually need two puppet code environments

Yes that's correct! Interesting I haven't thought about this one yet. This might be something to consider. Oh and I totally left out Puppet profiles, which of course also have production and nonprod named branches just because.

think about if that's actually required or if it just makes stuff more confusing

Feature flags sound good, but I worry from knowing the people in my organization and the servers they run that this could get out of hand.

and now the branches dont get out of hand?

how often do you merge from one branch to another? how big is the gap between nonprod and prod?

most customers I support end up with two (or more) completely different environments over time because most of their stuff doesn't understand git enough to do merges/cherry-picks, so we try to reduce the git complecity

Oh yes, it is absolutely confusing because you have servers which are to some IT teams "production" but actually run in the nonprod datacenter and are thus a nonprod system in the puppet environment but the module and profile are on a production branch.
It's SO complicated that I want to just rip all this out.

Disclaimer: I inherited this setup a couple years ago and am now rebuilding everything from the ground up

Yup, the people here also don't want to learn git and the ways on how to bring in changes etc. I'm happy if they do Pull Requests. But most just edit online on Bitbucket.

Regarding merging: We don't
Because we have logic in each branch which cannot be brought over to the other because this would break the server

ah so if people want to implement something for both environments they have to do it twice?

And then they forget to do it on the other branch

So I want to have logic in one branch where they can do exactly this in the SAME place so they don't have to think about it

They can still to PRs (which is good for reviews etc)

All this is a huge PITA and I hate everything about it :smile:

I would try to get rid of the multiple branches and reduce that

Agreed. This is the first action I'm taking. The layer of complexity is too much and does not bring any benefits.

in case you need and puppet or git training for the team, let me know :smile:

Thanks for your feedback and input! :slightly_smiling_face:

may I ask how many nodes you have? any performance issues?

Sure, so we have roughly 500 nodes and not really any performance issues. Why do you ask?

ah just curious. many people report  performance problems

Interesting. Long run times or other things?

Hmm I did find this:  <https://support.puppet.com/hc/en-us/articles/13823637065239-Copy-hiera-eyaml-encryption-keys-from-the-primary-server-to-the-compliers-and-replica-in-Puppet-Enterprise|https://support.puppet.com/hc/en-us/articles/13823637065239-Copy-hiera-eyaml-encryptio[…]ry-server-to-the-compliers-and-replica-in-Puppet-Enterprise>

Yes, the compilers each need a copy of the key.

every system which uses your data and which compiles a catalog needs the keys. this might also affect your ci.

Correct. Location of the key is configurable, and the key is somewhat highly sensitive and might be something that you would prefer to restrict to just the provisioning process that sets up new VMs that will eventually host compilers.

But, you could write a profile that manages the key as a `file` resource. But, that can lead to wanting to store the eyaml key in eyaml which makes it easy to set up a circular dependency...

...which is all fine when it works and BITES YOU HARD when there is a problem.

TL/DR compartmentalizing management of this key into a separate process (like a Bolt task) that exists outside of PE might be good :wink:

one can use the file resource type and specify content =&gt; file(&lt;path, file to key on puppetserver&gt;),

Which version of the `peadm` module are you using?

Hmm, oddly I don't see a query in the `peadm` module that matches against both the role and cluster. Closest is this, but that is missing the `role = "A"` bit:

<https://github.com/puppetlabs/puppetlabs-peadm/blob/v3.12.0/manifests/setup/node_manager.pp#L83-L88>

Regardless, the problem is the empty array match: `certname in []`

If this is an XL installation, was a value passed for the `puppetdb_database_host` parameter of the `peadm::upgrade` plan?

<@U11MDJNGZ> sorry for the late reply. It's a large installation. We converted the installation again and will check with the next update what will happen. 

I believe you need the "Nodes" "View node data from PuppetDB" permission in order to run PQL queries.

ah yeah. I had the operator role and assumed thats enough

because I can start tasks/plans with that role

The reason is that it is possible to generate PQL queries that could create high CPU load and deny services, so we wanted to make sure that we had a gate around it.

There was an update to jGit in 2021.7.3 which is why it might be new.  Thanks for the report.

ah good to know. let me know if you need more infos. but it should be easy to reproduce

In 2021.7.x we also added more agressive GCing to counter some scaling issues in HA. You can try to turn off that more aggressive GC and see if that helps (would be good to know). It's configurable with: `profile::master::cert_data_discard_history` it defaults to `true` (which is the more agressive behavior), you can disable it by setting it to `false`.

`cert_data` isn't a name I would have expected in that context

I guess I should mention that it should only be an issue if you have HA enabled

ah that's a single primary, no compilers and no replicas

I also saw that in 3 other setups with additional compilers. I'm not sure if I saw it on the compilers itself or only on the primaries

thanks for the info, I got excited because I saw essentially the same error on primaries when replicating the CA data and the above setting helped, but it may just be how we're using JGit is causing this and the toggle for CA data just lessened our usage enough to stop those warning from appearing.

ah the CA data is exchanged via filesync and internal git repos, like normal puppet code?

never had a customer that wanted a replica :sadluke:

yep, but file-sync is honestly a really bad fit for the cadata usage, so now we discard its history on every update which causes a lot of git activity and causes similar errors to what you saw. JGit seems to do no synchronization of its shared objects, but will ignore  most of those errors and end up healing itself, for the most part, now.

yeah every time I saw the error it complained about a lock on the logfile, not on anything important.

class caiso_unix_firewalld {

  package { 'firewalld':
    ensure =&gt; present,
  }

  service { 'firewalld':
    ensure  =&gt; running,
    enable  =&gt; true,
  }

  file { '/etc/firewalld/zones':
    ensure =&gt; directory,
    owner  =&gt; 'root',
    group  =&gt; 'root',
    mode   =&gt; '0750',
  }
}

class caiso_unix_firewalld::nrpe {

  require caiso_unix_firewalld

  $zonefile = $facts['caiso_environment'] ? {
    'caiso_dev' =&gt; '<http://nrpe_zone.dev|nrpe_zone.dev>',
    default     =&gt; 'nrpe_zone.prod',
  }

  file { '/etc/firewalld/zones/CAISO-nrpe.xml':
    ensure =&gt; file,
    owner  =&gt; 'root',
    group  =&gt; 'root',
    mode   =&gt; '0640',
    source =&gt; "puppet:///modules/caiso_unix_firewalld/${zonefile}",
    notify =&gt; Service['firewalld'],
  }
}

Error: Found 1 dependency cycle:
(File[/etc/firewalld/zones/CAISO-nrpe.xml] =&gt; Service[firewalld] =&gt; Class[Caiso_unix_firewalld] =&gt; Class[Caiso_unix_firewalld::Nrpe] =&gt; File[/etc/firewalld/zones/CAISO-nrpe.xml])\nTry the '--graph' option and opening the resulting '.dot' file in OmniGraffle or GraphViz
Error: Failed to apply catalog: One or more resource dependency cycles detected in graph

a) there's a firewalld module, I highly recommend using that instead of reinventing the wheel

b) in `caiso_unix_firewalld::nrpe` you require caiso_unix_firewalld but also send a notify to a service in that class. that doesn't work

I don't see why the `require` is requried. just drop that

<https://github.com/voxpupuli/puppet-firewalld> I recommend that, if you need to use firewalld

require is so that the package &amp; service is there

no. require means that the package and service is defined before caiso_unix_firewalld::nrpe. that doesn't make sense in your usecase

yea, now that you mention that, include would be better

Yup, `require caiso_unix_firewalld` means all resources declared in `caiso_unix_firewalld` have to be synced before any resources in `caiso_unix_firewalld::nrpe`.

So anything in `caiso_unix_firewalld::nrpe` that tries to move before something in `caiso_unix_firewalld`, like `notify =&gt; Service['firewalld']`, is going to be a circular dependency.

I was looking at the firewalld module, and had started with a rich rule, rather than a zone.  first time dealing with firewalld/nftables in rhel9, other admins preferred defining in zones rather than rich rules

ahhh, thanks for the explanation there <@U11MDJNGZ>

`include caiso_unix_firewalld` just ensures the compiler has evaluated that class so that things like `Service['firewalld']` are available to form dependencies against.

re: firewalld, so needed a bit more time to figure out the module, when I  have a couple dev/test hosts that "need it now"

You should be able to use classifier api group endpoint for changing parameters.  

<https://www.puppet.com/docs/pe/2019.8/groups_endpoint.html#post_v1_groups_id|https://www.puppet.com/docs/pe/2019.8/groups_endpoint.html#post_v1_groups_id>

yeah. I was hoping I can reuse an existing task :D

can anyone tell me what `puppet_admin_certs` means in this part of the docs?
<https://www.puppet.com/docs/pe/2021.7/config_puppetserver.html#add_certificates_to_the_puppet_admin_certificate_allowlist>

Is there a plug in for ingesting reports to ELK stack or are most people just pulling from logs?

When following the documentation <https://www.puppet.com/docs/puppet/7/ssl_regenerate_certificates.html#regenerate_agent_certs_and_add_dns_alt_names|Regenerate the agent certificate of your Puppet primary server and add DNS alt-names or other certificate extensions> on the latest Puppet PE primary server I am running into some issues.
When following it step by step i am running into the following output:
```root@puppet-primary:[~] #: puppetserver ca clean --certname puppet-primary.home.arpa
Certificate for puppet-primary.home.arpa has been revoked
Cleaned files related to puppet-primary.home.arpa
root@puppet-primary:[~] #:


root@puppet-primary:[~] #: puppet ssl clean
Error: Could not run: Failed to connect to the CA to determine if certificate puppet-primary.home.arpa has been cleaned
Wrapped exception:
certificate verify failed [certificate revoked for CN=puppet-primary.home.arpa]
root@puppet-primary:[~] #:```
This makes it seem that I need to manually remove the files in `/etc/puppetlabs/puppet/ssl/` . (Also the docs should say `pe-puppetserver` instead of `puppetserver` but this something I might bring up with support.

You are in PE channel, PE has special infrastructure plans

<https://www.puppet.com/docs/pe/2021.7/regenerate_certificates.html>

Ah, haven't thought about looking at the PE tasks and plans. That link above was just was Google listed at the top when searching for it.