Page MenuHomePhabricator

decom debug proxies (was: Migrate debug proxies to Stretch/Buster)
Closed, ResolvedPublic

Description

The debug proxies for X-Wikimedia-Debug are currently running jessie:

  • hassaleh.codfw.wmnet
  • hassium.eqiad.wmnet

Event Timeline

ArielGlenn triaged this task as Medium priority.Jun 11 2019, 7:57 AM
MoritzMuehlenhoff added subscribers: Vgutierrez, ema, BBlack.

@ema, @Vgutierrez, @BBlack With ATS handling the X-Wikimedia-Header and the ATS backend work completed these should be good to go. I've checked the NGINX access log on hassium/eqiad and the last entry is from 19th Dec. Can you confirm there's no other further pending work/tests which would make debug proxies needed again? Then I'd drop them from our environment.

jijiki added a subscriber: jijiki.Jan 10 2020, 9:50 AM
ema added a comment.Jan 10 2020, 9:52 AM

Can you confirm there's no other further pending work/tests which would make debug proxies needed again? Then I'd drop them from our environment.

Confirmed, hassaleh and hassium can retire.

Mentioned in SAL (#wikimedia-operations) [2020-01-10T11:48:49Z] <moritzm> stop/mask nginx on hassium/hassaleh T224567

Dzahn renamed this task from Migrate debug proxies to Stretch/Buster to decom debug proxies (was: Migrate debug proxies to Stretch/Buster).Jan 10 2020, 8:02 PM
Dzahn added a subscriber: Dzahn.

created decom subtasks via Lifecycle decom form

Mentioned in SAL (#wikimedia-operations) [2020-01-13T14:55:30Z] <moritzm> remove hassaleh in Ganeti T224567

cookbooks.sre.hosts.decommission executed by jmm@cumin2001 for hosts: hassaleh.codfw.wmnet

  • hassaleh.codfw.wmnet (FAIL)
    • Downtimed host on Icinga
    • No management interface found (likely a VM)
    • Unable to connect to the host, wipe of bootloaders will not be performed: Cumin execution failed (exit_code=2)
    • Failed to shutdown, manual intervention required: Cumin execution failed (exit_code=2)
    • Set Netbox status on VM not yet supported: manual intervention required
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB

ERROR: some step on some host failed, check the bolded items above

cookbooks.sre.hosts.decommission executed by jmm@cumin2001 for hosts: hassium.eqiad.wmnet

  • hassium.eqiad.wmnet (FAIL)
    • Downtimed host on Icinga
    • No management interface found (likely a VM)
    • Wiped bootloaders
    • Shutdown issued. Verify it manually, verification not yet supported
    • Set Netbox status on VM not yet supported: manual intervention required
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB

ERROR: some step on some host failed, check the bolded items above

Mentioned in SAL (#wikimedia-operations) [2020-01-13T15:19:37Z] <moritzm> remove hassium in Ganeti T224567

Change 564044 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Remove debug proxy roles/classes

https://gerrit.wikimedia.org/r/564044

MoritzMuehlenhoff closed this task as Resolved.Jan 13 2020, 3:45 PM

hassium/hassaleh have been retired.

Change 564044 merged by Muehlenhoff:
[operations/puppet@production] Remove debug proxy roles/classes

https://gerrit.wikimedia.org/r/564044

Krinkle updated the task description. (Show Details)
Krinkle added a subscriber: Krinkle.

Thanks! Less is more :)