Page MenuHomePhabricator

Rebuild ping* hosts with 10G disks
Open, Needs TriagePublic

Description

The ping* hosts only have 3G disks, which fill up very easily with software updates. When they get rebuilt for bullseye, let's re-create them with 10G disks.

Event Timeline

Mentioned in SAL (#wikimedia-operations) [2021-11-16T13:18:43Z] <moritzm> prune unused packages from ping1001/ping2001 T295767

Mentioned in SAL (#wikimedia-operations) [2021-11-16T13:21:42Z] <moritzm> prune unused packages from ping3001 T295767

Change 741912 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/puppet@production] Prepare site.pp for new ping VMs

https://gerrit.wikimedia.org/r/741912

Change 741912 merged by Ayounsi:

[operations/puppet@production] Prepare site.pp for new ping VMs

https://gerrit.wikimedia.org/r/741912

Change 741916 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/puppet@production] Add new ping VMs to DHCP

https://gerrit.wikimedia.org/r/741916

Change 741916 merged by Ayounsi:

[operations/puppet@production] Add new ping VMs to DHCP

https://gerrit.wikimedia.org/r/741916

Change 741918 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/puppet@production] Set flat partman receipe for all ping hosts

https://gerrit.wikimedia.org/r/741918

Change 741918 merged by Ayounsi:

[operations/puppet@production] Set flat partman receipe for all ping hosts

https://gerrit.wikimedia.org/r/741918

Change 741923 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/homer/public@master] Move ping offload to new ping VMs

https://gerrit.wikimedia.org/r/741923

Mentioned in SAL (#wikimedia-operations) [2021-11-25T14:42:53Z] <XioNoX> Update ping redirect to point to new ping VMs - T295767

Change 741923 merged by jenkins-bot:

[operations/homer/public@master] Move ping offload to new ping VMs

https://gerrit.wikimedia.org/r/741923

Change 741926 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/puppet@production] Remove old ping servers from DHCP

https://gerrit.wikimedia.org/r/741926

Change 741926 merged by Ayounsi:

[operations/puppet@production] Remove old ping servers from DHCP

https://gerrit.wikimedia.org/r/741926

cookbooks.sre.hosts.decommission executed by ayounsi@cumin1001 for hosts: ping2001.codfw.wmnet

  • ping2001.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.codfw.wmnet to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.codfw.wmnet to Netbox

cookbooks.sre.hosts.decommission executed by ayounsi@cumin1001 for hosts: ping3001.esams.wmnet

  • ping3001.esams.wmnet (PASS)
    • Downtimed host on Icinga
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.esams.wmnet to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.esams.wmnet to Netbox

cookbooks.sre.hosts.decommission executed by ayounsi@cumin1001 for hosts: ping1001.eqiad.wmnet

  • ping1001.eqiad.wmnet (PASS)
    • Downtimed host on Icinga
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.eqiad.wmnet to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.eqiad.wmnet to Netbox
ayounsi added a subscriber: ayounsi.

All 3 VMs got rebuilt with larger disks, but with the default Debian Buster.

@MoritzMuehlenhoff let me know if they need to be re-rebuild :)

All 3 VMs got rebuilt with larger disks, but with the default Debian Buster.

@MoritzMuehlenhoff let me know if they need to be re-rebuild :)

We'll need to rebuild with Bullseye at some point, but for now we can simply keep them as-is :-)