OK getting back to that. I tried something new: I logged in onto the puppetserver container, and regenerated the certificates (after deleting them all):

root@puppet:/# /opt/puppetlabs/server/bin/puppetserver ca setup
Generation succeeded. Find your files in /etc/puppetlabs/puppetserver/ca