Page MenuHomePhabricator
Create Task
Maniphest T295767

Rebuild ping* hosts with 10G disks
Closed, ResolvedPublic
Actions

Description

The ping* hosts only have 3G disks, which fill up very easily with software updates. When they get rebuilt for bullseye, let's re-create them with 10G disks.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Remove old ping servers from DHCPoperations/puppetproduction+0 -15
Move ping offload to new ping VMsoperations/homer/publicmaster+3 -3
Set flat partman receipe for all ping hostsoperations/puppetproduction+1 -1
Add new ping VMs to DHCPoperations/puppetproduction+15 -0
Prepare site.pp for new ping VMsoperations/puppetproduction+1 -1
Customize query in gerrit

Related Objects

Event Timeline

MoritzMuehlenhoff created this task.Nov 16 2021, 1:18 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 16 2021, 1:18 PM
Stashbot added a comment.Nov 16 2021, 1:18 PM

Mentioned in SAL (#wikimedia-operations) [2021-11-16T13:18:43Z] <moritzm> prune unused packages from ping1001/ping2001 T295767

Stashbot added a comment.Nov 16 2021, 1:21 PM

Mentioned in SAL (#wikimedia-operations) [2021-11-16T13:21:42Z] <moritzm> prune unused packages from ping3001 T295767

gerritbot added a comment.Nov 25 2021, 12:57 PM

Change 741912 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/puppet@production] Prepare site.pp for new ping VMs

https://gerrit.wikimedia.org/r/741912

gerritbot added a project: Patch-For-Review.Nov 25 2021, 12:57 PM
gerritbot added a comment.Nov 25 2021, 1:06 PM

Change 741912 merged by Ayounsi:

[operations/puppet@production] Prepare site.pp for new ping VMs

https://gerrit.wikimedia.org/r/741912

Maintenance_bot removed a project: Patch-For-Review.Nov 25 2021, 1:10 PM
gerritbot added a comment.Nov 25 2021, 1:43 PM

Change 741916 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/puppet@production] Add new ping VMs to DHCP

https://gerrit.wikimedia.org/r/741916

gerritbot added a project: Patch-For-Review.Nov 25 2021, 1:43 PM

Change 741916 merged by Ayounsi:

[operations/puppet@production] Add new ping VMs to DHCP

https://gerrit.wikimedia.org/r/741916

gerritbot added a comment.Nov 25 2021, 1:57 PM

Change 741918 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/puppet@production] Set flat partman receipe for all ping hosts

https://gerrit.wikimedia.org/r/741918

gerritbot added a comment.Nov 25 2021, 1:59 PM

Change 741918 merged by Ayounsi:

[operations/puppet@production] Set flat partman receipe for all ping hosts

https://gerrit.wikimedia.org/r/741918

Maintenance_bot removed a project: Patch-For-Review.Nov 25 2021, 2:10 PM
gerritbot added a comment.Nov 25 2021, 2:38 PM

Change 741923 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/homer/public@master] Move ping offload to new ping VMs

https://gerrit.wikimedia.org/r/741923

gerritbot added a project: Patch-For-Review.Nov 25 2021, 2:38 PM
Stashbot added a comment.Nov 25 2021, 2:42 PM

Mentioned in SAL (#wikimedia-operations) [2021-11-25T14:42:53Z] <XioNoX> Update ping redirect to point to new ping VMs - T295767

gerritbot added a comment.Nov 25 2021, 2:42 PM

Change 741923 merged by jenkins-bot:

[operations/homer/public@master] Move ping offload to new ping VMs

https://gerrit.wikimedia.org/r/741923

ayounsi mentioned this in rOHPU7cf456a25440: Move ping offload to new ping VMs.Nov 25 2021, 2:43 PM
gerritbot added a comment.Nov 25 2021, 2:57 PM

Change 741926 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/puppet@production] Remove old ping servers from DHCP

https://gerrit.wikimedia.org/r/741926

gerritbot added a comment.Nov 25 2021, 2:58 PM

Change 741926 merged by Ayounsi:

[operations/puppet@production] Remove old ping servers from DHCP

https://gerrit.wikimedia.org/r/741926

ops-monitoring-bot added a comment.Nov 25 2021, 3:04 PM

cookbooks.sre.hosts.decommission executed by ayounsi@cumin1001 for hosts: ping2001.codfw.wmnet

  • ping2001.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.codfw.wmnet to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.codfw.wmnet to Netbox
ops-monitoring-bot added a comment.Nov 25 2021, 3:12 PM

cookbooks.sre.hosts.decommission executed by ayounsi@cumin1001 for hosts: ping3001.esams.wmnet

  • ping3001.esams.wmnet (PASS)
    • Downtimed host on Icinga
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.esams.wmnet to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.esams.wmnet to Netbox
ops-monitoring-bot added a comment.Nov 25 2021, 3:22 PM

cookbooks.sre.hosts.decommission executed by ayounsi@cumin1001 for hosts: ping1001.eqiad.wmnet

  • ping1001.eqiad.wmnet (PASS)
    • Downtimed host on Icinga
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.eqiad.wmnet to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.eqiad.wmnet to Netbox
ayounsi claimed this task.Nov 25 2021, 3:27 PM
ayounsi subscribed.

All 3 VMs got rebuilt with larger disks, but with the default Debian Buster.

@MoritzMuehlenhoff let me know if they need to be re-rebuild :)

MoritzMuehlenhoff added a comment.Nov 29 2021, 8:35 AM
In T295767#7529602, @ayounsi wrote:

All 3 VMs got rebuilt with larger disks, but with the default Debian Buster.

@MoritzMuehlenhoff let me know if they need to be re-rebuild :)

We'll need to rebuild with Bullseye at some point, but for now we can simply keep them as-is :-)

ayounsi closed this task as Resolved.Dec 6 2021, 4:59 PM

Alright, closing this for now then :)

LSobanski mentioned this in T273509: upgrade ping offload servers to bullseye (was: ping servers running out of disk).Dec 13 2022, 3:32 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits