The debug proxies for X-Wikimedia-Debug are currently running jessie:
- hassaleh.codfw.wmnet
- hassium.eqiad.wmnet
Description
Description
Details
Details
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| Remove debug proxy roles/classes | operations/puppet | production | +0 -133 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | MoritzMuehlenhoff | T224549 Track remaining jessie systems in production | |||
| Resolved | MoritzMuehlenhoff | T224567 decom debug proxies (was: Migrate debug proxies to Stretch/Buster) | |||
| Resolved | Request | MoritzMuehlenhoff | T242456 decommission hassium.eqiad.wmnet | ||
| Resolved | Request | MoritzMuehlenhoff | T242457 decommission hassaleh.codfw.wmnet |
Event Timeline
Comment Actions
@ema, @Vgutierrez, @BBlack With ATS handling the X-Wikimedia-Header and the ATS backend work completed these should be good to go. I've checked the NGINX access log on hassium/eqiad and the last entry is from 19th Dec. Can you confirm there's no other further pending work/tests which would make debug proxies needed again? Then I'd drop them from our environment.
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2020-01-10T11:48:49Z] <moritzm> stop/mask nginx on hassium/hassaleh T224567
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2020-01-13T14:55:30Z] <moritzm> remove hassaleh in Ganeti T224567
Comment Actions
cookbooks.sre.hosts.decommission executed by jmm@cumin2001 for hosts: hassaleh.codfw.wmnet
- hassaleh.codfw.wmnet (FAIL)
- Downtimed host on Icinga
- No management interface found (likely a VM)
- Unable to connect to the host, wipe of bootloaders will not be performed: Cumin execution failed (exit_code=2)
- Failed to shutdown, manual intervention required: Cumin execution failed (exit_code=2)
- Set Netbox status on VM not yet supported: manual intervention required
- Removed from DebMonitor
- Removed from Puppet master and PuppetDB
ERROR: some step on some host failed, check the bolded items above
Comment Actions
cookbooks.sre.hosts.decommission executed by jmm@cumin2001 for hosts: hassium.eqiad.wmnet
- hassium.eqiad.wmnet (FAIL)
- Downtimed host on Icinga
- No management interface found (likely a VM)
- Wiped bootloaders
- Shutdown issued. Verify it manually, verification not yet supported
- Set Netbox status on VM not yet supported: manual intervention required
- Removed from DebMonitor
- Removed from Puppet master and PuppetDB
ERROR: some step on some host failed, check the bolded items above
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2020-01-13T15:19:37Z] <moritzm> remove hassium in Ganeti T224567
Comment Actions
Change 564044 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Remove debug proxy roles/classes
Comment Actions
Change 564044 merged by Muehlenhoff:
[operations/puppet@production] Remove debug proxy roles/classes