Hello all! Curious how folks here handle TLS certi...
# general
a
Hello all! Curious how folks here handle TLS certificate ownership across teams? We recently had a renewal silently fail and the tricky part wasn’t expiry — it was figuring out who owned the endpoint and where else the same cert was reused. Are you just relying on cert-manager/ACME + alerts, or do you track ownership/runbooks somewhere?
a
Personally, combination cert-manager & certbot & caddy & tracking expiry with uptime kuma dashboard with webhooks to slack.
a
Nice that’s a very real-world stack Do you have a good way today to answer: “who owns this endpoint/cert?” and “where else is the same cert reused?” We found expiry monitoring was easy, but those two questions were what slowed us down during incidents.
a
i treat it as part of infra, so its all coalessed under infra. app teams "shouldnt" be dealing with certs. for alot of it, cert-manager issues a wildcard for that arena, so *.nonprod.app.example.com is isolated to that environment, which tracks to a github repo, with ownership descriptions (repo would be named app.example.com, and it has terraform noting all the clusters. slightly different for some metal hosts, because i'm lazy, and dont want to move that to k8s, and fight with getting things to work nicely (its mail servers)
a
That’s a pretty clean setup. Do you ever run into issues when the same wildcard or cert ends up reused across clusters or metal hosts and something changes (issuer/chain/etc)? We found that the infra repo told us intent, but during incidents we still had to manually confirm where the cert was actually deployed.
a
not really, each deployment using cert-manager is quite explicit
Copy code
apiVersion: <http://cert-manager.io/v1|cert-manager.io/v1>
          kind: Certificate
          metadata:
            name: "default-ingress-cert"
            namespace: company-core-cert-manager
            annotations:
              <http://argocd.argoproj.io/sync-wave|argocd.argoproj.io/sync-wave>: "1"
          spec:
            dnsNames:
            - "*.<http://apps.nonprod.app.example.com|apps.nonprod.app.example.com>"
            - "*.<http://nonprod.app.example.com|nonprod.app.example.com>"
            - "*.<http://platform.nonprod.app.example.com|platform.nonprod.app.example.com>"
            - "*.<http://internal.nonprod.app.example.com|internal.nonprod.app.example.com>"
            - "*.<http://public.nonprod.app.example.com|public.nonprod.app.example.com>"
this along with restrictions to where DNS lives (route53 account/zone) means that this cluster cannot deploy certs for
<http://nonprod.app.acme.com|nonprod.app.acme.com>
, unless its an http-01 cert, and that means someone has access to create the dns entry, which (1) we can track, and (2) is usually only for a specific subset of apps within a known environment
there are definitely loopholes in current processes, but its not the highest burning fire 😛 i suspect you have some form of CMDB? the only thing i could think of is that the information you're looking for should be mandated in your cmdb. but that gets very stagnant very quickly
depending on how you use your certs, wouldnt the sans of a given cert point you to where its used?