Page MenuHomePhabricator

cloudinfra mx certificate fails to renew
Closed, ResolvedPublic

Description

the cloudinfra acme-chief cert is failing to push the updated DNS records:

Traceback (most recent call last):
  File "/usr/local/bin/acme-chief-designate-sync.py", line 60, in <module>
    zone = potential_zones[0]
           ~~~~~~~~~~~~~~~^^^
IndexError: list index out of range

Event Timeline

taavi triaged this task as High priority.
Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Change #1174686 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] acme_chief: designate: Include problematic zone in error message

https://gerrit.wikimedia.org/r/1174686

With the above applied:

RuntimeError: Did not find zone for domain '_acme-challenge.mx-out.wmflabs.org.'

The list of domain names for this certificate is absolutely massive, and I feel like some of these could be dropped:

profile::acme_chief::certificates:
  mx:
    CN: mx-out.wmflabs.org
    SNI:
    - mx-out01.wmflabs.org
    - mx-out02.wmflabs.org
    - mx-out.wmcloud.org
    - mx-out01.wmcloud.org
    - mx-out02.wmcloud.org
    - mx-out01.cloudinfra.eqiad1.wikimedia.cloud
    - mx-out02.cloudinfra.eqiad1.wikimedia.cloud
    - mx-out03.wmflabs.org
    - mx-out04.wmflabs.org
    - mx-out03.wmcloud.org
    - mx-out04.wmcloud.org
    - mx-out03.cloudinfra.eqiad1.wikimedia.cloud
    - mx-out04.cloudinfra.eqiad1.wikimedia.cloud
    - mx-out05.wmflabs.org
    - mx-out06.wmflabs.org
    - mx-out05.wmcloud.org
    - mx-out06.wmcloud.org
    - mx-out05.cloudinfra.eqiad1.wikimedia.cloud
    - mx-out06.cloudinfra.eqiad1.wikimedia.cloud
    - mx-out-a.wmcloud.org
    - mx-out-b.wmcloud.org
    - mx-out-a.cloudinfra.eqiad1.wikimedia.cloud
    - mx-out-b.cloudinfra.eqiad1.wikimedia.cloud

Change #1174686 merged by Majavah:

[operations/puppet@production] acme_chief: designate: Include problematic zone in error message

https://gerrit.wikimedia.org/r/1174686

Mentioned in SAL (#wikimedia-cloud) [2025-07-31T10:31:47Z] <taavi> cleanup SNIs on mx-out certificates T400873

Removed domains that do not resolve from that list of SNIs.