Page MenuHomePhabricator

deployment-mx02 puppet failures, deployment-acme-chief03 gives a 403
Closed, ResolvedPublic

Description

taavi@deployment-mx02:~$ sudo run-puppet-agent
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for deployment-mx02.deployment-prep.eqiad1.wikimedia.cloud
Info: Applying configuration version '(9dbe9c68ed) root - profile::etcd::v3: use puppet certs for standalone cluster'
Notice: The LDAP client stack for this host is: classic/sudoldap
Notice: /Stage[main]/Profile::Ldap::Client::Labs/Notify[LDAP client stack]/message: defined 'message' as 'The LDAP client stack for this host is: classic/sudoldap'
Error: /Stage[main]/Profile::Mail::Mx/Acme_chief::Cert[mx]/File[/etc/acmecerts/mx]: Failed to generate additional resources using 'eval_generate': Error 403 on SERVER: access denied
Error: /Stage[main]/Profile::Mail::Mx/Acme_chief::Cert[mx]/File[/etc/acmecerts/mx]: Could not evaluate: Could not retrieve file metadata for puppet://deployment-acme-chief03.deployment-prep.eqiad.wmflabs/acmedata/mx: Error 403 on SERVER: access denied
Info: Stage[main]: Unscheduling all events on Stage[main]
Notice: Applied catalog in 7.10 seconds
taavi@deployment-mx02:~$ ping deployment-acme-chief03.deployment-prep.eqiad.wmflabs
PING deployment-acme-chief03.deployment-prep.eqiad.wmflabs (172.16.6.209) 56(84) bytes of data.
64 bytes from deployment-acme-chief03.deployment-prep.eqiad1.wikimedia.cloud (172.16.6.209): icmp_seq=1 ttl=64 time=0.530 ms
64 bytes from deployment-acme-chief03.deployment-prep.eqiad1.wikimedia.cloud (172.16.6.209): icmp_seq=2 ttl=64 time=0.473 ms
^C
--- deployment-acme-chief03.deployment-prep.eqiad.wmflabs ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 0.473/0.501/0.530/0.036 ms

Event Timeline

Mentioned in SAL (#wikimedia-releng) [2021-03-06T08:09:20Z] <Majavah> deployment-acme-chief change authorized regex for mx to use .eqiad1.wikimedia.cloud domain to fix T276652

taavi claimed this task.