Page MenuHomePhabricator

decom ununpentium
Closed, ResolvedPublic

Details

Related Gerrit Patches:

Event Timeline

Dzahn created this task.Mon, Oct 28, 10:18 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMon, Oct 28, 10:18 PM

Change 544080 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] site/install_server: decom ununpentium

https://gerrit.wikimedia.org/r/544080

Change 544080 merged by Dzahn:
[operations/puppet@production] site/install_server: decom ununpentium

https://gerrit.wikimedia.org/r/544080

Change 547045 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] mariadb::ferm_misc: remove ununpentium from allowed hosts

https://gerrit.wikimedia.org/r/547045

Change 547045 merged by Dzahn:
[operations/puppet@production] mariadb::ferm_misc: remove ununpentium from allowed hosts for RT

https://gerrit.wikimedia.org/r/547045

Change 547047 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/dns@master] remove ununpentium.wikimedia.org

https://gerrit.wikimedia.org/r/547047

Mentioned in SAL (#wikimedia-operations) [2019-10-29T22:17:46Z] <mutante> ununpentium - shutdown Ganeti VM - running decom script, schedule icinga downtime (T236748)

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: ununpentium.wikimedia.org

  • ununpentium.wikimedia.org (FAIL)
    • Downtimed host on Icinga
    • No management interface found (likely a VM)
    • Unable to connect to the host, wipe of bootloaders will not be performed: Cumin execution failed (exit_code=2)
    • Failed to shutdown, manual intervention required: Cumin execution failed (exit_code=2)
    • Set Netbox status on VM not yet supported: manual intervention required
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB

ERROR: some step on some host failed, check the bolded items above

^ Gone from Icinga. Status in Netbox was "offline" without manual intervention and there is no other status that would fit decom more. It was already shutdown in Ganeti hence the shutdown error.

Change 547047 merged by Dzahn:
[operations/dns@master] remove ununpentium.wikimedia.org

https://gerrit.wikimedia.org/r/547047

Dzahn closed this task as Resolved.Tue, Oct 29, 10:54 PM

Mentioned in SAL (#wikimedia-operations) [2019-10-29T23:09:04Z] <mutante> ganeti1003 - gnt-instance remove ununpentium.wikimedia.org (T236748)