Page MenuHomePhabricator
Create Task
Maniphest T273509

upgrade ping offload servers to bullseye (was: ping servers running out of disk)
Closed, ResolvedPublic
Actions

Description

All three ping* servers, ping1001, ping2001 and ping3001 ran out of disk space today, shortly after one another.

icinga alerted about it with minutes in between and they all had 0 bytes left.

We fixed them for now by running "apt-get clean" and "apt-get autoremove" (which finished kernel upgrades and removed old kernel and packages) and that got them all back to just 77% usage.

This was not a case of logs filling up /var/log, instead /usr/lib is the largest thing here and also / in general is just tiny on these. Just 3 GB on /dev/vda1.

Assuming it was caused by ongoing kernel upgrades pushing them over the edge.

This ticket is to decide that either nothing needs to be done, or that the disks are too small or that some other automated action is needed to prevent this in the future.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Remove old ping hostsoperations/puppetproduction+1 -20
Move ping offload from ping1002 to ping1003 in eqiadoperations/homer/publicmaster+1 -1
Move ping offload from ping2002 to ping2003 in codfwoperations/homer/publicmaster+1 -1
Fix MACoperations/puppetproduction+1 -1
Add ping[123]003operations/puppetproduction+19 -1
Customize query in gerrit

Related Objects

Event Timeline

Dzahn created this task.Feb 1 2021, 6:09 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 1 2021, 6:09 PM
MoritzMuehlenhoff subscribed.Feb 2 2021, 7:00 AM

We certainly should automate the removal of obsolete kernels in a better way, but with only 3G on the root partition that would happen again anyway (there will always be two kernels installed in any case), 3G is simply too small these days. The VMs should simply be bumped to 5 or 10G for the disks, but given that resizing the root disk is a little fiddly in Ganeti, we can simply wait until bullseye is out and recreate these as ping[123]002 on bullseye with bigger disks.

Dzahn added a comment.Feb 2 2021, 5:01 PM

Thank you. This solution seems good to me. Should we just close this again then? Or we can recycle/rename it to "upgrade ping servers to bullseye" :p

jcrespo added a comment.Feb 2 2021, 5:06 PM

+1 to rename and stall on bullseye being ready.

Dzahn renamed this task from ping servers running out of disk to upgrade ping offload servers to bullseye (was: ping servers running out of disk).Feb 6 2021, 12:06 AM
Dzahn changed the task status from Open to Stalled.
Dzahn triaged this task as Low priority.
taavi changed the task status from Stalled to Open.Jan 9 2022, 2:53 PM
LSobanski added a project: Infrastructure-Foundations.Dec 13 2022, 3:32 PM
LSobanski subscribed.

Looks like the disk space problem was addressed in T295767 but the upgrade to Bullseye is still pending.

MoritzMuehlenhoff claimed this task.Dec 13 2022, 3:47 PM
gerritbot added a comment.Jan 12 2023, 10:18 AM

Change 879284 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Add ping[123]003

https://gerrit.wikimedia.org/r/879284

gerritbot added a project: Patch-For-Review.Jan 12 2023, 10:18 AM
gerritbot added a comment.Jan 12 2023, 10:24 AM

Change 879284 merged by Muehlenhoff:

[operations/puppet@production] Add ping[123]003

https://gerrit.wikimedia.org/r/879284

Maintenance_bot removed a project: Patch-For-Review.Jan 12 2023, 10:31 AM
Stashbot added a comment.Jan 19 2023, 9:55 AM

Mentioned in SAL (#wikimedia-operations) [2023-01-19T09:55:14Z] <moritzm> installing ping3003 T273509

Stashbot added a comment.Jan 20 2023, 8:59 AM

Mentioned in SAL (#wikimedia-operations) [2023-01-20T08:59:06Z] <moritzm> installing ping2003 T273509

gerritbot added a comment.Jan 20 2023, 10:00 AM

Change 881829 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Fix MAC

https://gerrit.wikimedia.org/r/881829

gerritbot added a project: Patch-For-Review.Jan 20 2023, 10:00 AM
gerritbot added a comment.Jan 20 2023, 10:10 AM

Change 881829 merged by Muehlenhoff:

[operations/puppet@production] Fix MAC

https://gerrit.wikimedia.org/r/881829

Maintenance_bot removed a project: Patch-For-Review.Jan 20 2023, 10:30 AM
gerritbot added a comment.Jan 20 2023, 10:54 AM

Change 881837 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/homer/public@master] Move ping offload from ping2002 to ping2003 in codfw

https://gerrit.wikimedia.org/r/881837

gerritbot added a project: Patch-For-Review.Jan 20 2023, 10:54 AM
Stashbot added a comment.Jan 20 2023, 12:17 PM

Mentioned in SAL (#wikimedia-operations) [2023-01-20T12:17:10Z] <moritzm> installing ping1003 T273509

gerritbot added a comment.Jan 20 2023, 12:47 PM

Change 881869 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/homer/public@master] Move ping offload from ping1002 to ping1003 in eqiad

https://gerrit.wikimedia.org/r/881869

gerritbot added a comment.Jan 23 2023, 11:01 AM

Change 881837 merged by Muehlenhoff:

[operations/homer/public@master] Move ping offload from ping2002 to ping2003 in codfw

https://gerrit.wikimedia.org/r/881837

gerritbot added a comment.Jan 23 2023, 2:24 PM

Change 881869 merged by Muehlenhoff:

[operations/homer/public@master] Move ping offload from ping1002 to ping1003 in eqiad

https://gerrit.wikimedia.org/r/881869

Maintenance_bot removed a project: Patch-For-Review.Jan 23 2023, 2:32 PM
gerritbot added a comment.Jan 24 2023, 11:17 AM

Change 883137 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove old ping hosts

https://gerrit.wikimedia.org/r/883137

gerritbot added a project: Patch-For-Review.Jan 24 2023, 11:17 AM
ops-monitoring-bot added a comment.Jan 24 2023, 11:34 AM

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: ping3002.esams.wmnet

  • ping3002.esams.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster esams to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster esams to Netbox
ops-monitoring-bot added a comment.Jan 24 2023, 1:18 PM

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: ping2002.codfw.wmnet

  • ping2002.codfw.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster codfw to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster codfw to Netbox
ops-monitoring-bot added a comment.Jan 24 2023, 1:30 PM

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: ping1002.eqiad.wmnet

  • ping1002.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster eqiad to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster eqiad to Netbox
gerritbot added a comment.Jan 24 2023, 1:32 PM

Change 883137 merged by Muehlenhoff:

[operations/puppet@production] Remove old ping hosts

https://gerrit.wikimedia.org/r/883137

MoritzMuehlenhoff closed this task as Resolved.Jan 24 2023, 1:36 PM

New ping1003/ping2003/ping3003 Bullseye VMs with 10G disk space have been created and swapped into service. The old hosts have been decommissioned.

Maintenance_bot removed a project: Patch-For-Review.Jan 24 2023, 2:31 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits