After migrating the pki infrastructure to puppet7 we started to see issues with certificate renewal. It seems the main error seen was
```
2023/10/31 10:10:37 [INFO] generate received request
2023/10/31 10:10:37 [INFO] received CSR
2023/10/31 10:10:37 [INFO] generating key: ecdsa-256
2023/10/31 10:10:37 [INFO] encoded CSR
2023/10/31 10:10:37 [INFO] Using client auth with mutual-tls-cert: /etc/cfssl/mutual_tls_client_cert.pem and mutual-tls-key: /var/lib/puppet/ssl/private_keys/pki2002.codfw.wmnet.pem
2023/10/31 10:10:37 [INFO] Using trusted CA from tls-remote-ca: /etc/ssl/certs/wmf-ca-certificates.crt
{"code":7400,"message":"failed POST to https://pki.discovery.wmnet:443/api/v1/cfssl/authsign: Post \"https://pki.discovery.wmnet:443/api/v1/cfssl/authsign\": x509: issuer name does not match subject from issuing certificate"}
```
which is caused by the [[ https://github.com/ikapelyukhin/go-x509-issuer-name-does-not-match-subject, | strict processing in go ]] and the lax ssl implementation in puppet [insert puppet bug here]
It was also noticed that the ocsp refresh process was failing with
```
ERROR:root:debmonitor issue with SQL query: (2003, "Can't connect to MySQL server on 'm1-master.eqiad.wmnet' ([SSL:CERTIFICATE_VERIFY_FAILED] certificate veri>
```
which was fixed by [[ https://gerrit.wikimedia.org/r/c/operations/puppet/+/970267/ | updating the ca trust bundle ]]
To fix the issue i have now de-pooled pki2002 which is still using puppet7 so we can debug and rolled back pki1001 to puppet5
The first issues in codfw seems to have occurred at [[ https://puppetboard.wikimedia.org/report/restbase2022.codfw.wmnet/4ee0ec27d439b6474731202e567f54bf05fb27ec | 00:05:33 ]] and in eqiad at [[ https://puppetboard.wikimedia.org/report/ganeti1029.eqiad.wmnet/edf7d8a811f15f161e2617c60e2a6ad9abefc9e9 | 00:23:53 ]]