Puppet Community

you get that from an agent? It tries to download the certificate revocation list from the compiler/primary. sounds like that failed

running it with --debug or --http_debug should show the the http calls that fail

running puppet 7.16.  So it would communicate to a compiler not the primary directly.  Maybe the updated CRL doesn’t exist on the compiler?

that would be odd. you can run the agent on the compiler itself

```
Debug: Resolving service 'ca' using Puppet::HTTP::Resolver::Settings
Debug: Creating new connection for <https://puppet:8140>
Debug: Starting connection for <https://puppet:8140>
opening connection to puppet:8140...
opened
starting SSL for puppet:8140...
Conn close because of connect error SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get certificate CRL)
Error: certificate verify failed [unable to get certificate CRL for CN=<http://puppetserver01.nwops.io|puppetserver01.nwops.io>]
Error: Could not run: certificate verify failed [unable to get certificate CRL for CN=<http://puppetserver01.nwops.io|puppetserver01.nwops.io>]```


that looks odd. is `puppet` expected to be resolved, or didn't you specify the correct server?

that part works as we use it on other systems

okay. and that points to your primary and puppserver01 is your compiler? Or the real FQDN from the primary?

I had to hide the real names.  But initially we were using a specific compiler in the server setting and saw that error.  I then tried to use puppet which resolves to the primary.

blew away the ssl directory on the agent.  ran puppet same error.  Downloads the crl.pem but still complains.

did it get corrupted on the primary maybe? you could try to do puppetserver ca generate bla and puppetserver ca revoke bla to get it updated

I am able to fetch it though so not a comm issue.  I’ll validate with other nodes.

if nodes already have one they won't update it I think

ok, this is a new server so that computes

The CRL on old system is way bigger than on new system

so I assume the ca api reads a file and doesn’t cache it?  Why would the shorter file be transmitted

I manually copied the crl.pem and ran puppet.  Signing request was sent.  So this seems like the crl.pem is not being sent out on the first puppet run.

<https://puppet.com/docs/puppet/7/configuration.html#certificate-revocation>

do you have en_US.UTF-8 locale available on that box?

at least on ubuntu 20.p4 the PE installer doesnt depend on the correct package to pull in the locale, and the installer will fail if its missing

to be fair I later learned its a documented dependency :D

The docs don’t detail how exactly to run it

<https://puppet.com/docs/pe/2021.6/supported_operating_systems.html#supported_operating_systems|https://puppet.com/docs/pe/2021.6/supported_operating_systems.html#supported_operating_systems> I thought I saw the locale documented there, but it isnt

Been looking here <https://puppet.com/docs/pe/2021.6/dr_configuring.html>

so people, anybody see increased IO caused by postgresql on the primary, after upgrading PE 2019 to 2019.8.11?

The "Run Puppet" job in Puppet Enterprise, is there an option to icrease the limit of 8 nodes in parallel?

<@U17J6MKV5>
```puppet_enterprise::profile::orchestrator::global_concurrent_compiles: 8```

I rely on `puppet infrastructure tune --compare`  you can check what changes it recommends

<@U1PC2RAH0> sounds like table bloat. You can run pg_repack on those tables to shrink them.  I think that even runs automatically now

<https://puppet.com/docs/pe/2019.8/pe_database_maintenance.html#enable_pe_database_module>

2019.8.9. pg_repack was already enabled and didnt  do a difference

postgresql processes make roughly 120MB/s avg since the upgrade, which is odd

it’s possible that pg_repack is failing to run on those tables

the systemd services at least didnt log errors

I can’t remember how it works… do you see repack activity in the postgresql logs?

mhm no. it logs to stdout. output looks fine

how did you acquire the size of the tables?

there is continious IO from the postgresql checkpointer now, we didnt see that before the upgrade

with <https://wiki.postgresql.org/wiki/Disk_Usage|https://wiki.postgresql.org/wiki/Disk_Usage> . used that in the past with great success on other databases

total db size is ~93GB. I cannot acccess the environment until Sunday. But <@U17J6MKV5> can probably provide some more details tomorrow

puppetdb has a preformance regression on postgresql 11 with jit enabled, there is a ticket for that, but we confirmed jit is off.

<https://tickets.puppetlabs.com/browse/PDB-5450|https://tickets.puppetlabs.com/browse/PDB-5450> thats the one. I was hoping we hit that as well, but I couldnt confirm that

okay yea a support ticket is best and they can just give you the right queries to run and more up-to-date advice than I have.

I expected a successful repack to shrink those down.

or maybe it has errors but doesnt log them. I need to check if verbosity can be tuned.

you could try <https://wiki.postgresql.org/wiki/Show_database_bloat>

no but I think through a combination of APIs calls someone has a script to get to that info

The `pe_status_check` will report `pxp-agent` connection status as a fact:

<https://github.com/puppetlabs/puppetlabs-pe_status_check/blob/main/lib/facter/agent_status_check.rb#L17-L18>

Otherwise, PuppetDB doesn’t have visibility into that. The closest it would get is a list of agent `certnames` that have reported within the past `node-ttl` that could be compared against the list of connected `certnames` returned by the Orchestrator API.

<https://www.sickchirpse.com/wp-content/uploads/2011/01/bill-and-ted-excellent.jpg>

What is the easiest way to put a trusted cert (internal) on code manager service?

thinking of using netcat and piping to curl

how do the pe_repos get synced to other compilers?

the LB should pass through SSL to the compilers

this should help <https://puppet.com/docs/pe/2021.6/installing_compilers.html#using-load-balancers-with-compilers-load_balancing_for_orchestration_scale>

Select an unresponsive node, perform a manual run (or restart the service to trigger a service run) and verify via exit code and local system log. Then verify on the server side, starting with the logs of the compiling server. Then verify that reports are passing through the PuppetDB server(s) to the PE PostgreSQL server(s).

I want to run a task on 800 nodes but I dont’ want to run all at once is there a max_concurrent setting?  What is the default?  This is a puppet agent upgrade task so I don’t want to DDOS my puppet servers.

the limit is already set
```puppet_enterprise::profile::orchestrator::global_concurrent_compiles```

with bolt you can be more granular it has
```bolt task run  --concurrency```

`# @param task_concurrency [Integer] The maximum number of task runs to allow at once, across all jobs.`