Per T303459 , these hosts are slated to be refreshed.
Creating this ticket to:
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | RKemper | T344198 Decommission wdqs100[3-5] | |||
| Resolved | Request | Jclark-ctr | T345081 hw troubleshooting: ipmi down for wdqs1005.eqiad.wmnet | ||
| Resolved | Request | Jclark-ctr | T345391 decommission wdqs1005 | ||
| Resolved | Request | Jclark-ctr | T346699 decommission wdqs100[3,4].eqiad.wmnet |
Mentioned in SAL (#wikimedia-operations) [2023-08-25T21:03:47Z] <inflatador> bking@cumin1001 shutting off wdqs1005 in preparation for decommission T344198
wdqs1005 is alerting for ipmi, and based on this search result it seems like it might have hardware issues.
Since it's already scheduled for refresh, I've shut if off and we'll continue the decom process on Monday.
Change 952864 had a related patch set uploaded (by Bking; author: Bking):
[operations/puppet@production] wdqs1005: Disable notifications
Mentioned in SAL (#wikimedia-operations) [2023-08-28T17:00:35Z] <inflatador> bking@cumin1001 depool wdqs1005 for decom T344198
Change 952864 merged by Ryan Kemper:
[operations/puppet@production] wdqs1005: Disable notifications and remove from lvs
Mentioned in SAL (#wikimedia-operations) [2023-08-31T19:03:21Z] <ryankemper> T344198 Temporarily disabling puppet on all wdqs* hosts in preparation for wdqs.discovery.wmnet certificate revocation
Mentioned in SAL (#wikimedia-operations) [2023-08-31T19:03:41Z] <ryankemper> T344198 on ryankemper@cumin1001: sudo -E cumin 'A:wdqs-all' 'sudo disable-puppet "revoking old cert and generating new one with new alt_names - T344198"'
Change 954123 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):
[operations/puppet@production] wdqs: new wqds.discovery cert
Change 954123 merged by Ryan Kemper:
[operations/puppet@production] wdqs: new wqds.discovery cert
Generated new cergen certs for wdqs.discovery.wmnet that include wdqs1016 in the alt_names instead of wdqs1005. Followed the steps below:
INSTRUCTIONS (1) Edit /srv/private/modules/secret/secrets/certificates/certificate.manifests.d/wdqs.certs.yaml #stop and disable puppet on all wdqs* hosts here # !log T344198 Temporarily disabling puppet on all `wdqs*` hosts in preparation for `wdqs.discovery.wmnet` certificate revocation # !log T344198 on `ryankemper@cumin1001`: `sudo -E cumin 'A:wdqs-all' 'sudo disable-puppet "revoking old cert and generating new one with new alt_names - T344198"'` (2) on `ryankemper@cumin1001`: `sudo -E cumin 'A:wdqs-all' 'sudo disable-puppet "revoking old cert and generating new one with new alt_names - T344198"'` (3) sudo puppet cert clean wdqs.discovery.wmnet (4) sudo rm -fv /srv/private/modules/secret/secrets/certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.crt.pem /srv/private/modules/secret/secrets/certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.csr.pem /srv/private/modules/secret/secrets/certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.keystore.jks /srv/private/modules/secret/secrets/certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.keystore.p12 /srv/private/modules/secret/secrets/certificates/wdqs.discovery.wmnet/truststore.jks (5) sudo cergen -c 'wdqs.*' --generate --base-path /srv/private/modules/secret/secrets/certificates /srv/private/modules/secret/secrets/certificates/certificate.manifests.d (6) The new cert should be under `/srv/private/modules/secret/secrets/certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.crt.pem`, add this to puppet under `files/ssl/wdqs.discovery.wmnet.crt` (7) <commit+merge the public and private updates> sudo chown -R gitpuppet:gitpuppet modules/secret/secrets/certificates/wdqs.discovery.wmnet/ sudo chown gitpuppet:gitpuppet /srv/private/modules/secret/secrets/ssl/wdqs.discovery.wmnet.key (8) <run puppet on one public wdqs server and verify queries work by tunneling in> # <re-enable puppet on all wdqs* and `sudo run-puppet-agent`>
cookbooks.sre.hosts.decommission executed by ryankemper@cumin1001 for hosts: wdqs1005.eqiad.wmnet
Made the decom ticket for wdqs1005 and ran the cookbook. We will decom 1003/1004 later. Moving this to blocked/waiting for now.
Change 955858 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):
[wikidata/query/deploy@master] wdqs1016 is taking over as canary
Change 955858 merged by Bking:
[wikidata/query/deploy@master] wdqs1016 is taking over as canary
Change 958572 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):
[operations/puppet@production] wdqs: decom old canary wdqs1003
Running decom cookbook for wdqs100[3,4]. Dc-ops ticket up here: https://phabricator.wikimedia.org/T346699
Change 958572 merged by Ryan Kemper:
[operations/puppet@production] wdqs: decom wdqs100[3,4]
cookbooks.sre.hosts.decommission executed by ryankemper@cumin1001 for hosts: wdqs1003.eqiad.wmnet
cookbooks.sre.hosts.decommission executed by ryankemper@cumin1001 for hosts: wdqs1004.eqiad.wmnet