Page MenuHomePhabricator
Create Task
Maniphest T297180

Revert 5.10.70 from bullseye hosts
Closed, ResolvedPublic
Actions

Description

The update to 5.10.70 (shipped as part of the Bullseye 11.1 kernel update and buster-backports) caused an outage on cloudgw servers when upgraded to it and we also saw an issue with conntrack on mx2001 which caused firewall issues. Revert it for now by

  • uninstalling the 5.10.70 kernel packages on bullseye systems
  • rebooting affected systems back into 5.10.46
  • cloudmetrics1003.eqiad.wmnet (ignored, to be upgraded to 5.10.84 when it's out)
  • cloudmetrics1004.eqiad.wmnet (ignored, to be upgraded to 5.10.84 when it's out)
  • copernicium.wikimedia.org
  • db1124.eqiad.wmnet (ignored, these are only test hosts, to be upgraded to 5.10.84 when it's out)
  • db1125.eqiad.wmnet (ignored, these are only test hosts, to be upgraded to 5.10.84 when it's out)
  • db1128.eqiad.wmnet (ignored, these are only test hosts, to be upgraded to 5.10.84 when it's out)
  • failoid2002.codfw.wmnet
  • ganeti6001.drmrs.wmnet (not yet fully set up, to be upgraded to 5.10.84 when it's out)
  • ganeti6002.drmrs.wmnet (not yet fully set up, to be upgraded to 5.10.84 when it's out)
  • ganeti6003.drmrs.wmnet (not yet fully set up, to be upgraded to 5.10.84 when it's out)
  • ganeti6004.drmrs.wmnet (not yet fully set up, to be upgraded to 5.10.84 when it's out)
  • graphite1004.eqiad.wmnet
  • graphite2003.codfw.wmnet
  • ldap-replica1003.wikimedia.org
  • ldap-replica2005.wikimedia.org
  • ldap-replica2006.wikimedia.org
  • pc2014.codfw.wmnet (ignored, these are only test hosts, to be upgraded to 5.10.84 when it's out)
  • people2002.codfw.wmnet (inactive, to be upgraded to 5.10.84 when it's out)
  • prometheus2005.codfw.wmnet
  • prometheus2006.codfw.wmnet
  • puppetboard2002.codfw.wmnet (inactive, to be upgraded to 5.10.84 when it's out)
  • rpki1001.eqiad.wmnet
  • rpki2002.codfw.wmnet
  • sretest1002.eqiad.wmnet (ignored, just a test host, to be upgraded to 5.10.84 when it's out)

Related Objects

Event Timeline

MoritzMuehlenhoff created this task.Dec 7 2021, 8:14 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 7 2021, 8:14 AM
MoritzMuehlenhoff updated the task description. (Show Details)Dec 7 2021, 8:20 AM
MoritzMuehlenhoff updated the task description. (Show Details)Dec 7 2021, 10:52 AM
MoritzMuehlenhoff mentioned this in T297128: Bringing mx2001 back into service.Dec 7 2021, 11:13 AM
fgiunchedi updated the task description. (Show Details)Dec 7 2021, 11:33 AM
fgiunchedi subscribed.Dec 7 2021, 11:35 AM

I chatted with @MoritzMuehlenhoff re: the rollback, apt won't let you remove a running kernel though there's a way to ask grub to reboot into another menu entry (the second entry of the first submenu in this case). Therefore the procedure can look like this:

# grub-reboot '1>2'
# reboot
...
# apt remove linux-image-5.10.0-9-amd64
# grub-editenv /boot/grub/grubenv unset next_entry

Note linux-image-amd64 will also get removed, which is correct since it depends on -9- not -8-

Stashbot added a comment.Dec 7 2021, 1:40 PM

Mentioned in SAL (#wikimedia-operations) [2021-12-07T13:40:54Z] <godog> reboot graphite2003 - T297180

Stashbot added a comment.Dec 7 2021, 2:28 PM

Mentioned in SAL (#wikimedia-operations) [2021-12-07T14:28:30Z] <godog> reboot graphite1004 - T297180

fgiunchedi updated the task description. (Show Details)Dec 7 2021, 2:41 PM
MoritzMuehlenhoff updated the task description. (Show Details)Dec 7 2021, 2:43 PM
MoritzMuehlenhoff updated the task description. (Show Details)
MoritzMuehlenhoff updated the task description. (Show Details)
MoritzMuehlenhoff updated the task description. (Show Details)Dec 7 2021, 3:56 PM
MoritzMuehlenhoff updated the task description. (Show Details)Dec 7 2021, 4:03 PM
MoritzMuehlenhoff updated the task description. (Show Details)Dec 7 2021, 4:13 PM
MoritzMuehlenhoff updated the task description. (Show Details)Dec 7 2021, 5:27 PM
MoritzMuehlenhoff updated the task description. (Show Details)Dec 7 2021, 5:52 PM
bd808 subscribed.Dec 7 2021, 6:16 PM
Dzahn subscribed.Dec 7 2021, 9:14 PM
Legoktm mentioned this in T297265: graphite1004 freezing.Dec 8 2021, 2:47 AM
MoritzMuehlenhoff updated the task description. (Show Details)Dec 8 2021, 8:32 AM
MoritzMuehlenhoff closed this task as Resolved.Dec 13 2021, 7:52 AM
MoritzMuehlenhoff claimed this task.
MoritzMuehlenhoff triaged this task as High priority.
MoritzMuehlenhoff updated the task description. (Show Details)

This is complete

Platonides mentioned this in T297127: Incident: 2021-12-03 mx2001->Gmail delivery issues.Jan 13 2022, 12:01 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits