tools-acme-chief-01 is attempting to validate DNS challenge against cloud authdns IPv6 addresses
Open, MediumPublic
Actions

Assigned To

None

Authored By

	Krenair
	Feb 23 2020, 2:30 AM

Description

I noticed strange behaviour while running puppet on a tools acme-chief client and noticed the logs on tools-acme-chief-01 show this:

Feb 23 02:18:21 tools-acme-chief-01 acme-chief-backend[23899]: Handling pushed CSR event for toolforge / rsa-2048
Feb 23 02:18:25 tools-acme-chief-01 acme-chief-backend[23899]: DNS server 2620:0:861:2:208:80:154:135 (ACMEChallengeValidation.UNKNOWN) failed to validate challenge Challenge type: ACMEChallengeType.DNS01. _acme-challenge.toolforge.org TXT ILmKGKtTIpwadEcimFmGXsb-Eb9JJAx9yV-7nY4Kllo
Feb 23 02:18:25 tools-acme-chief-01 acme-chief-backend[23899]: DNS server 2620:0:861:1:208:80:154:11 (ACMEChallengeValidation.UNKNOWN) failed to validate challenge Challenge type: ACMEChallengeType.DNS01. _acme-challenge.toolforge.org TXT ILmKGKtTIpwadEcimFmGXsb-Eb9JJAx9yV-7nY4Kllo
Feb 23 02:18:25 tools-acme-chief-01 acme-chief-backend[23899]: Unable to validate challenge Challenge type: ACMEChallengeType.DNS01. _acme-challenge.toolforge.org TXT ILmKGKtTIpwadEcimFmGXsb-Eb9JJAx9yV-7nY4Kllo

Any kind of IPv6 traffic from within labs would obviously fail, so why is it trying?
/etc/acme-chief/config.yaml contains:

challenges:
  dns-01:
    issuing_ca: letsencrypt.org
    ns_records:
    - cloud-ns0.wikimedia.org.
    - cloud-ns1.wikimedia.org.
    resolver_port: 53
    sync_dns_servers:
    - cloud-ns0.wikimedia.org
    - cloud-ns1.wikimedia.org
    validation_dns_servers:
    - cloud-ns0.wikimedia.org
    - cloud-ns1.wikimedia.org
    zone_update_cmd: "/usr/local/bin/acme-chief-designate-sync.py"

And with our puppetisation there's no way of putting IPv4 addresses in there.

>>> from acme_chief.dns import Resolver
>>> Resolver.resolve_dns_servers(['cloud-ns0.wikimedia.org', 'cloud-ns1.wikimedia.org'])
['208.80.154.135', '2620:0:861:2:208:80:154:135', '208.80.154.11', '2620:0:861:1:208:80:154:11']

Resolver.resolve_dns_servers calls socket.getaddrinfo but does not specify family and ignores the family in the results.

Important to get this fixed or worked around by 19th March as that's the expiry date of the live toolforge cert this instance manages.

Details

	Subject	Repo	Branch	Lines +/-
	Allow configuration of AddressFamily used for DNS validation	operations/software/acme-chief	master	+31 -12

Customize query in gerrit

	Title	Reference	Author	Source Branch	Dest Branch
	Allow configuration of AddressFamily used for DNS validation	repos/sre/acme-chief!1	brett	change-574221-allow-config-of-addressfamily-dns-validation	main

Customize query in GitLab

Related Objects

Mentioned Here: T176891: DNS resolution chosing IPv6 addrs on hosts with only link-local IPv6 addresses

Event Timeline

Krenair created this task.Feb 23 2020, 2:30 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 23 2020, 2:30 AM

Krenair renamed this task from tools-acme-chief-01 is attempting to validate DNS challenge against authdns IPv6 addresses to tools-acme-chief-01 is attempting to validate DNS challenge against cloud authdns IPv6 addresses.Feb 23 2020, 2:30 AM

I'd be interested to know why this has not been a problem before by the way - those cloud-ns hosts have had AAAA records since creation AFAIK

Change 574221 had a related patch set uploaded (by Alex Monk; owner: Alex Monk):
[operations/software/acme-chief@master] Allow configuration of AddressFamily used for DNS validation

https://gerrit.wikimedia.org/r/574221

gerritbot added a project: Patch-For-Review.Feb 23 2020, 3:29 AM

T176891: DNS resolution chosing IPv6 addrs on hosts with only link-local IPv6 addresses may be related.

for the traffic cloud instances I bypassed this issue with this ugly hack:

authdns_servers:
  208.80.154.11: 208.80.154.11
  208.80.154.135: 208.80.154.135

In T245937#5910553, @Vgutierrez wrote:
for the traffic cloud instances I bypassed this issue with this ugly hack:
authdns_servers:
  208.80.154.11: 208.80.154.11
  208.80.154.135: 208.80.154.135

I assumed with https://gerrit.wikimedia.org/r/c/operations/puppet/+/554292/5/modules/profile/manifests/acme_chief.pp requiring Stdlib::Fqdn this would not work. Interesting. New cert has successfully been issued now.

bd808 edited projects, added cloud-services-team (Kanban); removed cloud-services-team.Mar 8 2020, 4:44 AM

Andrew triaged this task as Medium priority.Mar 10 2020, 4:07 PM

@Krenair, can you summarize the results here? It looks resolved but it's not clear if or how :)

In T245937#6185676, @Andrew wrote:

@Krenair, can you summarize the results here? It looks resolved but it's not clear if or how :)

worked around with an ugly hiera hack :)

fnegri edited projects, added cloud-services-team; removed cloud-services-team (Kanban).Jan 18 2023, 7:32 PM

fnegri moved this task from Kanban to Inbox on the cloud-services-team board.

Pppery edited projects, added Patch-Needs-Improvement; removed Patch-For-Review.Apr 2 2023, 12:54 AM

brett opened https://gitlab.wikimedia.org/repos/sre/acme-chief/-/merge_requests/1

Allow configuration of AddressFamily used for DNS validation

Restricted Application removed a project: Patch-Needs-Improvement. · View Herald TranscriptSep 28 2023, 9:16 PM

Change 574221 abandoned by BCornwall:

[operations/software/acme-chief@master] Allow configuration of AddressFamily used for DNS validation

Reason:

Migrated to Gitlab: https://gitlab.wikimedia.org/repos/sre/acme-chief/-/merge_requests/1