Puppet Community

I have went through reading nooks, created some modules for internal use, did practice labs do not know what do to else beside reviewing reading nooks again

they are normally generated together, what do you have for:
```ls /etc/puppetlabs/puppet/ssl/*/*myserver*```

thank you so much <@U3ZVA4ND6> you was right, the answer is no!  it was test server so I learn from my mistake :slightly_smiling_face: thanks again

do you habe iowait on yout postgresql box?

did you check the puppetdb metrics already?

image.png

Hey <@U1PC2RAH0>, regarding the iowait, the PuppetDB is RDS at AWS and this is the monitoring view of the last 3 days:

and could you please let me know how to check the puppetdb metrics?

btw, the drop in iops write and read is due to the command `VACUUM FULL VERBOSE` we ran on the DB

maybe autovacuum isn't enabled or not agressive enough

<https://puppet.com/docs/puppetdb/7/api/metrics/v1/mbeans.html> puppetdb has some builtin metrics

on the 18th, we ran the autovacuum command I mentioned above, is it not suppose to be the cause of that dropdown?

for the drop, maybe. but why does it go up afterwards

I guess that's not a counter, those are current values

it goes up if old tuples aren't deleted frequently

postgresql usually has a logfile where it writes down information about the vacuum and the amount of dead tuples

okay thanks, any idea on what the postgres log path might be?

`/var/lib/pgsql/13/data/log/` if you use the postgresql packages

<https://gist.github.com/bastelfreak/58c22a611ba5bc3bc096501b7716a44a> I think those are the relevant options from the postgresql.conf

on which postgresql version are you? and which puppetdb?

okay I see in the logs in AWS these errors:
```puppetdb@puppetdb:[19511]:ERROR: could not serialize access due to concurrent up date```
```puppetdb@puppetdb:[1701]:ERROR: set-returning functions are not allowed in CASE at character 559```

you think we can rollback the DB to the point it worked (before the 18th)?

I recommend tuning the autovacuum and planning a DB upgrade

as a start, enable/ensure logging works, run a manual vacuum, ensure autovacuum is enabled, check for wal_buffers, work_mem, max_wal_size, max_worker_processes

and keep in mind that puppet 5 is dead and EoL :slightly_smiling_face:

if you do the manual vacuum now: note that it will do an exclusive lock on the tables and no other queries will run during that time

btw, regarding making sure that the autovacuum is enabled I see it is since we:  `Error during garbage collection` every hour
I guess it means it runs

is that from the puppetdb log? then it's not the vacuum from postgresql

puppetdb does some cleanup for old reports/catalogs/nodes

you can also check how many days of reports you store and reduce that, to reduce the database size

sounds good, is that configured in the `puppet.conf` file?

I think it's /etc/puppetlabs/puppetdb/conf.d/config.ini

`puppetserver ca` can be used to generate certificates like that

I don't know that there's a specific puppet internal resource for that

I would also think carefully about reusing the puppet ca for anything not puppet (you might be exposing internal puppet data to those client certs)

It's easy enough to use openssl to generate a self-signed certificate, or you can use something like certbot and letsencrypt to create an externally-signed certificate.

yeah, i could do a self-signed cert, but there's always the problem of distribution and trust with a CA, all of that

in this case trying to generate a cert for mutual TLS of a client/server

<https://github.com/archlinux/svntogit-community/blob/packages/facter/trunk/PKGBUILD> if it helps, this is how I build the official facter arch linux package

So I downloaded it from git (`<https://github.com/puppetlabs/facter/archive/refs/tags/3.14.23.zip>`), unpacked it (`unzip 3.14.23.zip`), and started to build it as suggested in the `README.md`: `cd facter-3.14.23`, `bundle install`. Here's what I get:

```Fetching source index from <https://artifactory.delivery.puppetlabs.net/artifactory/api/gems/rubygems/>

Retrying fetcher due to error (2/4): Bundler::HTTPError Could not fetch specs from <https://artifactory.delivery.puppetlabs.net/artifactory/api/gems/rubygems/>                                                                           

Retrying fetcher due to error (3/4): Bundler::HTTPError Could not fetch specs from <https://artifactory.delivery.puppetlabs.net/artifactory/api/gems/rubygems/>                                                                           

Retrying fetcher due to error (4/4): Bundler::HTTPError Could not fetch specs from <https://artifactory.delivery.puppetlabs.net/artifactory/api/gems/rubygems/> ```

try <https://downloads.puppet.com/facter/>

On FreeBSD, puppet was recently updated from version 6 to 7. The puppet server still uses 6, so I compiled Ruby 2.7.x manually and installed puppet and facter as a gem.

you can use puppet agent 7 to talk to puppetserver

Which does not work very we;;:
```sudo puppet agent --test --noop
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Error: Failed to apply catalog: Resource type 'Ssh_authorized_key' was not found```
Even though I have installed the `puppetlabs-sshkeys_core` module.

See the `puppetlabs-sshkeys_core` module at <https://forge.puppet.com/puppetlabs/sshkeys_core?src=slack&amp;channel=puppet>

just puppetserver 6 and puppet agent 7 on the same box won't work

OK, re-trying that. For some reason, that caused problems...

some core types got moved into their own modules and the puppetlabs rpm vendors them

I fix this in the arch linux PKGBUILD I linked above

```sudo puppet agent --test
Error: Could not autoload puppet/indirector/report/rest: undefined method `use_server_setting' for #&lt;Class:0x0000000806d5c600&gt;                                                                                                          
Error: Could not prepare for execution: Could not autoload puppet/indirector/report/rest: undefined method `use_server_setting' for #&lt;Class:0x0000000806d5c600&gt;```

mhm, do you have a custom report processor?

I suggest to head over to <#C6CK0UGB1|> and wait for <@U6B5Q9BSA> to wake up

puppet tries to load a lib that can process reports from each agent run. that fails. could be an issue with puppet or with a processor (e.g. parsing them and posting into slack)

Puppet is quite messed up on that server; I already tried many things. Maybe I should clean it up first and retry.

Puppet 6 and 7 are available in the ports, but since we now ship Ruby 3.0 by default, if you use Puppet 6 you have to build the packages yourself

I would recommend you set DEFAULT_VERSIONS=ruby=2.7 and build your own packages if you want to continue using Puppet 6 for some time, but I think it's worth migrating to Puppet 7

Puppet 6 is expected to be EOL by May 2023 according to <https://puppet.com/docs/puppet/6/platform_lifecycle.html>

matt@vm020:274% puppet module list
/opt/puppetlabs/puppet/modules
├── puppet-selinux (v3.2.0)
├── puppetlabs-apache (v5.4.0)
├── puppetlabs-concat (v6.2.0)
├── puppetlabs-stdlib (v6.3.0)
└── puppetlabs-translate (v2.2.0)
matt@vm020:275%

Thanks Matt. Recently I have ran into an issue where pupper server CA cert expired. While checking, came to know about some orchestration tools like bolt. Wanted to check if any of those are installed in my environment. Hence the question. Will it show that as a module?

apt list --installed | grep bolt || rpm -qa | grep bolt

Hello all - Is there a way to check, list of plugins that are used/installed in a puppet environment?

No timeline published at the moment, but I believe it is planned for the new LTS release in August.

DeepinScreenshot_Alacritty_20220719180358.png

DeepinScreenshot_Alacritty_20220719180403.png

looks like:

a) puppet isn’t running or
b) a firewall is blocking the port

Those are the most common I can think of.

Is port 8140 open (firewall/selinux denial)?  Can you connect to the port with netcat to verify network connectivity?

If you can't connect you can (temporarily!!!) disable firewall and selinux to see which is causing the problem.  If those are not the cause, is your puppetserver running or is it configured to respond on a non-standard port?  Does 'puppet' resolve to the IP address of your server?

On your puppetserver, what’s the full contents of `/etc/puppetlabs/puppet/puppet.conf`?

i have checked and puppet is running and checked the firewall with ufw and it is allowing connection from the server. however, there is not a port listening on 8140 on the agent server

Only the `puppetserver` services listens on TCP 8140.

Agent’s don’t have any listening services.

i am confused on why the server is trying to connect to the agent on port 8140 then

Please share the entire contents of `/etc/puppetlabs/puppet/puppet.conf` from your Puppetserver.

`root@mezzanotte-01:~# cat /etc/puppetlabs/puppet/puppet.conf`
`[main]`
`server = puppet`
`# This file can be used to override the default puppet settings.`
`# See the following links for more details on what settings are available:`
`# - <https://puppet.com/docs/puppet/latest/config_important_settings.html>`
`# - <https://puppet.com/docs/puppet/latest/config_about_settings.html>`
`# - <https://puppet.com/docs/puppet/latest/config_file_main.html>`
`# - <https://puppet.com/docs/puppet/latest/configuration.html>`

The server should not be connecting to the agent, the agents reach out to the server for their configuration

<https://puppetcommunity.slack.com/files/U03Q2UQLRC5/F03QQ2CRQ73/deepinscreenshot_alacritty_20220719180358.png>

Do you have DNS or an `/etc/hosts` entry that resolves `puppet` to the IP of your Puppetserver?

K, good.
And is the `puppetserver` service up and running?

What does `netstat -tlnp | grep 8140` show on the puppetserver?

`curl -kv <https://puppet:8140/status/v1/services>` on the server node would be interesting as well.

root@mezzanotte-00:~# netstat -tlnp | grep 8140
tcp6       0      0 :::8140                 :::*                    LISTEN      5117/java

root@mezzanotte-00:~# curl -kv <https://puppet:8140/status/v1/services>
*   Trying AGENT_IP:8140...
* connect to AGENT_IP port 8140 failed: Connection refused
* Failed to connect to puppet port 8140: Connection refused
* Closing connection 0
curl: (7) Failed to connect to puppet port 8140: Connection refused

have you actually run `puppetserver ca sign --certname mezzanotte-01` on your puppetserver?

<https://puppetcommunity.slack.com/files/U03Q2UQLRC5/F03QBB1FA4T/deepinscreenshot_alacritty_20220719180403.png>

oh, well then… if that’s what it’s doing, then as I initially noted, it can’t reach itself.

or if you don’t have netcat, telnet works too

`telnet puppet 8140`

ip6tables? firewalld? iptables?

your port is clearly up (netstat)

I think your DNS is probably messed up. Can you confirm that the IP address of your puppet *server* is actually in /etc/hosts, and that it’s not the IP of your agent?

or puppet hostname is pointing to wrong IP in /etc/hosts or in DNS somewhere

but anything trying to come off the wire (local or remote) can’t get there due to “connection refused”

`puppet` should resolve to the IP _of your Puppet Server_ on both the agent and the server.

(another option is to just use full FQDN’s for everything, and change to that instead of `puppet`… it’s usually less confusing that way)

That fixed the problem, i had the agent as puppet in /etc/hosts on the server

If you don't mind it being a bit hacky, you could do it with `exec`, `grep`, and `sed`:

```exec { comment-$somefile:
   path =&gt; $somefile,
   onlyif =&gt; "grep -qEv '^#' ${somefile}",
   command =&gt; "sed -i -e 's!^!#!g' ${somefile}",
}```


It assumes `#` as the comment character, depending on the syntax of the file you might have to get more complicated.

At this point i do not mind lighting something on fire, so being a bit hacky is well within my current limits :smile:

I would want to rethink what you are doing here.  `exec()` is something to stay away from if at all possible.  If you want to comment out every line of a file, wouldn't it be better to just replace the file with a 0 length, blank file instead?

Not an option. The original content needs to be preserved atm.

Yeah, I couldn't think of a better way than `exec` short of writing a custom type, given that it would need to be aware of the state of the file on the client

(tbh; I’d rather that `epp()` and `template()` would auto-defer themselves all on their own, but this is a useful stepping point to get there)

I was watching that PR with interest.  Is the idea to demonstrate the PoC before building this into the core functions at a later date? Or was there some other blocker?

there are conversations about whether core functions (`epp()` and `template()`) should be “magic” themselves or not. This is a step that allows users to try out the idea in the meantime.

would have made the work i did on Vault PKI certs a lot easier!

Notice: Local environment: 'development' doesn't match server specified environment 'production', restarting agent run with environment 'production'

on puppet agent /etc/puppetlabs/code/environments this directory only has production however on puppet master /etc/puppetlabs/code/environments directory has all the envrionments including development

Check if the ENC is forcing it prod
Also here's the agent docs <https://puppet.com/docs/puppet/7/environments_creating.html#env_assign_nodes_agents_config_file|https://puppet.com/docs/puppet/7/environments_creating.html#env_assign_nodes_agents_config_file>