today, @Andrew accidentally hit an interesting corner case on acme-chief. He committed the following config as part of https://gerrit.wikimedia.org/r/c/operations/puppet/+/496785:
ldap-labtest:
CN: 'ldap-labtest.eqiad.wikimedia.org'
SNI:
- 'labtestservices2001.wikimedia.org'
challenge: dns-01
authorized_regexes:
- '^labtestservices2001\.wikimedia\.org'as you can notice, the CN is not included in the SAN/SNI list, but Let's Encrypt always includes the CN in the SNI list. On the first attempt. acme-chief was able to issue the certificate as expected. But this triggers the following behaviour on certificate status re-evaluation:
Mar 15 15:00:01 acmechief1001 acme-chief-backend[14291]: Certificate ldap-labtest type ec-prime256v1 has SANs {'labtestservices2001.wikimedia.org', 'ldap-labtest.eqiad.wikimedia.org'} but is configured for {'labtestservices2001.wikimedia.org'}, moving back to re-issue
Mar 15 15:00:01 acmechief1001 acme-chief-backend[14291]: Certificate ldap-labtest type rsa-2048 has SANs {'labtestservices2001.wikimedia.org', 'ldap-labtest.eqiad.wikimedia.org'} but is configured for {'labtestservices2001.wikimedia.org'}, moving back to re-issueof course this triggers an infinite loop, hammering Let's Encrypt rate limits and every request related to that configured certificate is rejected by Let's Encrypt driving acme-chief crazy