Puppet has been failing on these servers:
If these are one-off test servers, please let us know so we can delete them.
Otherwise, could you take a few minutes to fix that? Thank you!
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | • GTirloni | T208856 Instances with Puppet failures | |||
| Resolved | bd808 | T208885 Puppet errors on test-twemproxy project |
On mc-clustera-1.test-twemproxy.eqiad.wmflabs:
$ puppet agent -tv Warning: Unable to fetch my node definition, but the agent run will continue: Warning: SSL_connect returned=1 errno=0 state=unknown state: sslv3 alert certificate revoked Info: Retrieving pluginfacts Error: /File[/var/lib/puppet/facts.d]: Failed to generate additional resources using 'eval_generate': SSL_connect returned=1 errno=0 state=unknown state: sslv3 alert certificate revoked Error: /File[/var/lib/puppet/facts.d]: Could not evaluate: Could not retrieve file metadata for puppet:///pluginfacts: SSL_connect returned=1 errno=0 state=unknown state: sslv3 alert certificate revoked Info: Retrieving plugin Error: /File[/var/lib/puppet/lib]: Failed to generate additional resources using 'eval_generate': SSL_connect returned=1 errno=0 state=unknown state: sslv3 alert certificate revoked Error: /File[/var/lib/puppet/lib]: Could not evaluate: Could not retrieve file metadata for puppet:///plugins: SSL_connect returned=1 errno=0 state=unknown state: sslv3 alert certificate revoked Info: Loading facts Error: Could not retrieve catalog from remote server: SSL_connect returned=1 errno=0 state=unknown state: sslv3 alert certificate revoked Warning: Not using cache on failed catalog Error: Could not retrieve catalog; skipping run Error: Could not send report: SSL_connect returned=1 errno=0 state=unknown state: sslv3 alert certificate revoked
Fixed with:
$ rm -rf /var/lib/puppet/ssl $ puppet agent -tv
Another victim of eqiad1-r migration strangeness?
Same problem and fix on mc-clusterA-2.test-twemproxy.eqiad.wmflabs, mc-clusterB-1.test-twemproxy.eqiad.wmflabs, and mc-clusterB-2.test-twemproxy.eqiad.wmflabs
@Andrew, does this seem like a problem that could be caused by the region migration script? Is there a way that a "delete" event could have been sent out that made labs-puppetmaster.wikimedia.org revoke the client certs?
It's unlikely that moving the clients would break that. It's /possible/ that moving the master itself broke things but I haven't seen that happen before.