This message was deleted Puppet Community #puppet-enterprise

Join Slack

This message was deleted.

# puppet-enterprise

Slackbot

10/27/2022, 5:41 PM

This message was deleted.

Jeremy Mill

10/27/2022, 5:44 PM

hey Kelvin, fairly positive this is a feature since the UI cert usually comes from a "normal" trusted CA

bastelfreak

10/27/2022, 5:48 PM

by default, the console certificate comes from the Puppetserver CA

bastelfreak

10/27/2022, 5:48 PM

but you can provide your own certificate

bastelfreak

10/27/2022, 5:50 PM

@kelvin you need to import the puppetserver CA cert (and the intermediate) into your truststores

kelvin

10/27/2022, 5:51 PM

I have instructions (and have implemented) custom certificate on 443, but that doesn’t seem to get installed on TCP8140.

Jeremy Mill

10/27/2022, 5:53 PM

that's the intended behavior, the cert on 8140 should be from the Puppet CA

kelvin

10/27/2022, 5:53 PM

Well, then clients will not trust it without bypassing trust, which we would prefer not to do

bastelfreak

10/27/2022, 5:55 PM

you have a few options: • import the CA • use the official Puppet mirrors instead • create your own mirror • setup a proxy on your primary

kelvin

10/27/2022, 5:55 PM

I can certainly supply that certificate to curl, but was hoping to simply add the certificate to the OS trust store on Linux, but Linux processes ignore non-CA certificates. Took me a while to figure it out, because our older puppetserver did not have this issue.

bastelfreak

10/27/2022, 5:55 PM

you can still import the puppetserver CA

kelvin

10/27/2022, 5:55 PM

Not using linux update-ca-trust processes.

bastelfreak

10/27/2022, 5:56 PM

which file did you import?

kelvin

10/27/2022, 5:56 PM

https://unix.stackexchange.com/questions/525601/update-ca-trust-extract-not-adding-certificates-to-ca-bundle

kelvin

10/27/2022, 5:57 PM

TL;DR The

update-ca-trust

won’t extract your certificate file to the

ca-bundle.crt

unless this succeeds:

Copy code

openssl x509 -noout -text -in <cert_file> | grep --after-context=2 "X509v3 Basic Constraints" | grep "CA:TRUE"

Because the puppetserver certificate served on 8140 does not match CA:TRUE (it’s FALSE) Linux update-ca-trust fails to copy/extract to the tls-ca-bundle used by cURL on linux.

bastelfreak

10/27/2022, 5:58 PM

again, you need to import the puppetserver CA cert, not the server certificate from port 8140 🙂

bastelfreak

10/27/2022, 5:58 PM

Copy code

# openssl x509 -noout -text -in /etc/puppetlabs/puppetserver/ca/ca_crt.pem | grep CA
        Issuer: CN = Puppet Root CA: 7cf5d98cdb92bc
        Subject: CN = "Puppet Enterprise CA generated at +2022-10-22 17:42:42 +0000"
                CA:TRUE

kelvin

10/27/2022, 5:58 PM

Yeah, I was shooting toward using a one-time setup.

kelvin

10/27/2022, 5:59 PM

Hrm. Let me check out the ca_crt.pem file.

kelvin

10/27/2022, 5:59 PM

That would work hopefully

kelvin

10/27/2022, 5:59 PM

brb

bastelfreak

10/27/2022, 5:59 PM

That will work 😄

bastelfreak

10/27/2022, 5:59 PM

I do that on a regular basis

Jeremy Mill

10/27/2022, 6:01 PM

the default for puppet is to trust at time of first use (ToFU) but you're welcome to distribute the CA ahead of time and manage how you trust the puppet CA

kelvin

10/27/2022, 6:39 PM

The certificate served on TCP8140 is self-signed; it is not issued by any other root ca.

kelvin

10/27/2022, 6:39 PM

Copy code

X509v3 Subject Key Identifier:
                49:92:A6:FB:5A:67:BF:C3:84:26:A6:CE:B1:04:D5:AC:C1:57:7B:70
            Netscape Comment:
                Puppet Server Internal Certificate
            X509v3 Authority Key Identifier:
                keyid:49:92:A6:FB:5A:67:BF:C3:84:26:A6:CE:B1:04:D5:AC:C1:57:7B:70

kelvin

10/27/2022, 6:39 PM

Subject Key and Authority Key identifiers are identical.

kelvin

10/27/2022, 6:42 PM

Essentially the certificate installed on 8140 is a self-signed, non-CA certificate, which Linux will not trust by default (not even sure if I can hack it in, but even so, that’s another hack I don’t want to do). So my only choice at this point is to tell curl to fetch the RPMs from the puppetserver on TCP8140 passing --cacert option for trust:

curl --cacert /etc/pki/ca-trust/source/anchors/puppetserver.crt

csharpsteen

10/27/2022, 6:45 PM

puppetserver

defaults to using the end-entity certificate of the

puppet

agent service on port 8140 which is signed by the Puppet CA certificate.

puppetserver

never has used or presented the Puppet CA certificate directly.

bastelfreak

10/27/2022, 6:56 PM

it's technical impossible for Puppet to ship a TLS certificate that matches your configured FQDN and is signed by a trusted CA. Puppetserver by default will create its own CA + intermediate. you can add that to your truststore, or tell curl to trust the certificate, or provide your own intermediate CA to puppetserver

kelvin

10/27/2022, 6:56 PM

This is not the behavior I see on 2021.7

bastelfreak

10/27/2022, 6:57 PM

which part is different in your environment?

bastelfreak

10/27/2022, 6:58 PM

I still have the impression that you mix up a random server certificate, as used on port 8140, with the CA cert that's used to sign server/agent certs

kelvin

10/27/2022, 6:59 PM

I’m doublechecking my sanity here…

bastelfreak

10/27/2022, 7:01 PM

As I posted above, the ca_crt.pem is indeed a CA. And that's the default setup in PE 2021.7

csharpsteen

10/27/2022, 7:03 PM

Backing up to the original question:

Discovered something that I’m trying to determine if its a feature or a bug. The certificate that puppet enterprise (2021.7) uses on 8140 (separate cert for 443) is not a CA certificate

That is the expected behavior and the default configuration going all the way back to PE 3.7.

csharpsteen

10/27/2022, 7:04 PM

If port 8140 was returning a certificate with

CA:TRUE

to the TLS handshake, that's something someone would had to have gone out of their way to configure.

csharpsteen

10/27/2022, 7:09 PM

It's also expected for the Puppet CA to be missing from the system trust store. Clients that need to communicate with the Puppet API, like the

puppet

daemon, are expected to use the certificate bundle returned by

puppet config print localcacert

to validate the Puppet Server certificate. That bundle could be added to the system trust store, but we don't recommend it as it gives the Puppet CA more privilege than needed for

puppet

operations.

kelvin

10/27/2022, 7:09 PM

Ok, I got it figured out I grabbed the wrong certificate out of the /etc/puppetlabs/puppetserver/ca/ca_crt.pem, and yes the cert served on TCP8140 is a leaf certificate signed by the Puppet Root CA:

subject=CN = Puppet Root CA

kelvin

10/27/2022, 7:11 PM

Trusting the Puppet Root CA on client systems is a bad thing???

csharpsteen

10/27/2022, 7:12 PM

No, it's a least privilege thing. The

puppet

daemon on client systems needs to trust the Puppet CA. That is not true of all HTTP/TLS clients that may be running on the system.

bastelfreak

10/27/2022, 7:14 PM

I would say it depends on who trusts it. The Puppet Agent needs to trust the CA. The rest of the Linux system usually doesn't have such a requirement. you can use the install script/repos from the primary. For that usecase, your curl/wget statement needs to trust the CA once. you can add the Puppetserver CA to the local truststore for that, but you don't have to

bastelfreak

10/27/2022, 7:14 PM

If you want to download the agent from a location that has a certificate from a known CA, use https://apt.puppet.com/ / https://yum.puppet.com/

kelvin

10/27/2022, 7:14 PM

Ok, then I’ll opt to perform a one-time “use the puppet root CA certificate” to fetch the client RPMs from the puppetserver.

bastelfreak

10/27/2022, 7:15 PM

The open source and enterprise rpm/debs are the same

kelvin

10/27/2022, 7:16 PM

we are using the puppetserver to source our RPMs for now, matching the version of the server, which gets patched monthly or quarterly (I forget puppet’s patch cadence)

kelvin

10/27/2022, 7:34 PM

Technically, the cert served on 8140 is a leaf certificate, signed by intermediate “Puppet Enterprise CA” signed by Puppet Root CA. The service on TCP8140 should be serving both the leaf and the intermediate certificate, up to, but not including the Puppet Root CA certificate. This permits me to use curl with --cacert with just the Root CA certificate.

csharpsteen

10/27/2022, 8:03 PM

--cacert

still works with the server providing just the leaf when the bundle has a full chain:

Copy code

# curl -v --cacert $(puppet config print localcacert) https://$(puppet config print server):8140/status/v1/simple
* About to connect() to clspetst-928bbc-0.us-west1-c.c.customer-support-scratchpad.internal port 8140 (#0)
*   Trying 10.253.3.252...
* Connected to clspetst-928bbc-0.us-west1-c.c.customer-support-scratchpad.internal (10.253.3.252) port 8140 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/puppetlabs/puppet/ssl/certs/ca.pem
  CApath: none
* NSS: client certificate not found (nickname not specified)
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
* 	subject: CN=clspetst-928bbc-0.us-west1-c.c.customer-support-scratchpad.internal
* 	start date: Oct 25 18:40:37 2022 GMT
* 	expire date: Oct 22 18:40:39 2037 GMT
* 	common name: clspetst-928bbc-0.us-west1-c.c.customer-support-scratchpad.internal
* 	issuer: CN="Puppet Enterprise CA generated at +2022-10-26 18:39:34 +0000"
> GET /status/v1/simple HTTP/1.1
> User-Agent: curl/7.29.0
> Host: clspetst-928bbc-0.us-west1-c.c.customer-support-scratchpad.internal:8140
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Thu, 27 Oct 2022 20:03:21 GMT
< Content-Type: text/plain;charset=utf-8
< Vary: Accept-Encoding, User-Agent
< Content-Length: 7
<
* Connection #0 to host clspetst-928bbc-0.us-west1-c.c.customer-support-scratchpad.internal left intact

kelvin

10/27/2022, 9:35 PM

yeah, I guess I’m really just thinking narrowly ala CAB forum that you really only anchor your roots, and TLS services serve the full chain upto, but not including the root.

3 Views

Open in Slack

Previous Next