Page MenuHomePhabricator
Create Task
Maniphest T393366

Regression in RAID10 software RAID with 6.1.135
Closed, ResolvedPublic
Actions

Description

There is a regression in the handling discard/TRIM on RAID 10 software RAID, which leads to soft lockups: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1104460

These hosts are running 6.1.135 with bookworm and RAID10. They needs to be rebooted into linux-image-6.1.0-33-amd64 (6.1.133): Once -33- is running, we can uninstall -34-

  • centrallog2002.codfw.wmnet
  • centrallog1002.eqiad.wmnet
  • vrts2002.codfw.wmnet
  • vrts1003.eqiad.wmnet
  • prometheus2005.codfw.wmnet
  • prometheus2006.codfw.wmnet
  • prometheus2007.codfw.wmnet
  • prometheus2008.codfw.wmnet
  • prometheus1005.eqiad.wmnet
  • prometheus1006.eqiad.wmnet
  • prometheus1007.eqiad.wmnet
  • prometheus1008.eqiad.wmnet

Once a fixed kernel is out, these can be reverted to the latest Bookworm kernel again.

These Bookworm hosts use software RAID10, but have not yet rebooted into 6.1.135, on these we only need to uninstall the 6.1.135 kernel.

  • cloudnet2005-dev.codfw.wmnet
  • cloudnet2006-dev.codfw.wmnet
  • cloudnet2007-dev.codfw.wmnet
  • cloudnet2008-dev.codfw.wmnet
  • cloudnet1005.eqiad.wmnet
  • cloudnet1006.eqiad.wmnet
  • cloudrabbit1001.eqiad.wmnet
  • cloudrabbit1002.eqiad.wmnet
  • cloudrabbit1003.eqiad.wmnet
  • cloudservices2004-dev.codfw.wmnet
  • cloudservices2005-dev.codfw.wmnet
  • cloudservices1005.eqiad.wmnet
  • cloudservices1006.eqiad.wmnet
  • puppetserver2001.codfw.wmnet
  • puppetserver2002.codfw.wmnet
  • puppetserver2003.codfw.wmnet
  • puppetserver2004.codfw.wmnet
  • puppetserver1001.eqiad.wmnet
  • puppetserver1002.eqiad.wmnet
  • puppetserver1003.eqiad.wmnet

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
benthos/mw-accesslog-metrics: undo consumer group renameoperations/puppetproduction+1 -1
benthos/mw-accesslog-metrics: start_from_oldest: falseoperations/puppetproduction+2 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

MoritzMuehlenhoff created this task.May 5 2025, 10:06 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 5 2025, 10:06 AM
MoritzMuehlenhoff added projects: collaboration-services, observability.May 5 2025, 10:11 AM
ops-monitoring-bot added a comment.May 5 2025, 10:17 AM

Host rebooted by jelto@cumin1002 with reason: revert kernel

MoritzMuehlenhoff updated the task description. (Show Details)May 5 2025, 10:31 AM
ops-monitoring-bot added a comment.May 5 2025, 10:32 AM

Host rebooted by jelto@cumin1002 with reason: revert kernel

LSobanski moved this task from Incoming to Work in Progress on the collaboration-services board.May 5 2025, 10:38 AM
MoritzMuehlenhoff updated the task description. (Show Details)May 5 2025, 10:49 AM
ops-monitoring-bot added a comment.May 5 2025, 10:58 AM

Host rebooted by jelto@cumin1002 with reason: revert kernel

MoritzMuehlenhoff updated the task description. (Show Details)May 5 2025, 11:05 AM
ops-monitoring-bot added a comment.May 5 2025, 11:05 AM

Host rebooted by jelto@cumin1002 with reason: revert kernel

Jelto updated the task description. (Show Details)May 5 2025, 11:11 AM
MoritzMuehlenhoff updated the task description. (Show Details)May 5 2025, 11:14 AM
MoritzMuehlenhoff added a parent task: T393357: soft lockup on prometheus, centrallog, vrts hosts with the new kernel.
MoritzMuehlenhoff triaged this task as High priority.May 5 2025, 11:34 AM
MoritzMuehlenhoff updated the task description. (Show Details)
fgiunchedi mentioned this in T393357: soft lockup on prometheus, centrallog, vrts hosts with the new kernel.May 5 2025, 11:55 AM
fgiunchedi updated the task description. (Show Details)May 5 2025, 11:59 AM
LSobanski mentioned this in T393345: SystemdUnitFailed - vrts1003 - user-runtime-dir@499.May 5 2025, 12:26 PM
LSobanski mentioned this in T393362: ProbeDown - vrts.
tappof updated the task description. (Show Details)May 5 2025, 1:04 PM
MoritzMuehlenhoff closed this task as Resolved.May 5 2025, 1:33 PM
MoritzMuehlenhoff claimed this task.

All RAID10 servers which were upgraded to 6.1.135, are reverted to 6.1.133. In addition linux-image-amd64 has been downgraded to 6.1.133 on all RAID10 hosts and the Linux 6.1.135 has been purged. I've also updated the underlying Bookworm reboot task (https://phabricator.wikimedia.org/T392804) so that these servers are going to be rebooted to a fixed 6.1 kernel once fixed upstream.

As such, marking this task which provided the immediate remediation as fixed.

gerritbot added a comment.May 5 2025, 2:14 PM

Change #1141905 had a related patch set uploaded (by Kamila Součková; author: Kamila Součková):

[operations/puppet@production] benthos/mw-accesslog-metrics: set start_offset to latest

https://gerrit.wikimedia.org/r/1141905

gerritbot added a project: Patch-For-Review.May 5 2025, 2:14 PM
jhathaway subscribed.May 5 2025, 3:47 PM
gerritbot added a comment.May 5 2025, 4:22 PM

Change #1141905 merged by Kamila Součková:

[operations/puppet@production] benthos/mw-accesslog-metrics: start_from_oldest: false

https://gerrit.wikimedia.org/r/1141905

Maintenance_bot removed a project: Patch-For-Review.May 5 2025, 4:30 PM
Dwisehaupt mentioned this in T393352: Host lockups for frban2002 and frmon2002 after fstrim operation.May 5 2025, 8:19 PM
gerritbot added a comment.May 6 2025, 11:47 AM

Change #1142576 had a related patch set uploaded (by Kamila Součková; author: Kamila Součková):

[operations/puppet@production] benthos/mw-accesslog-metrics: undo consumer group rename

https://gerrit.wikimedia.org/r/1142576

gerritbot added a project: Patch-For-Review.May 6 2025, 11:47 AM
gerritbot added a comment.May 6 2025, 1:02 PM

Change #1142576 merged by Kamila Součková:

[operations/puppet@production] benthos/mw-accesslog-metrics: undo consumer group rename

https://gerrit.wikimedia.org/r/1142576

Maintenance_bot removed a project: Patch-For-Review.May 6 2025, 1:31 PM
MoritzMuehlenhoff added a comment.May 7 2025, 5:46 AM

The cause of the regression is now identified; the backport to 6.1. missed an depending patch: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1104460#76

Once that additional patch has landed in the next 6.1.x update (likely for the Bookworm 12.11 update which will be released next week), we can reboot the affected systems into the fixed kernels.

MoritzMuehlenhoff added a comment.May 8 2025, 7:10 AM

A fixed package is now in bookworm-proposed-updates and will be part of the Bookworm 12.11 point relese which will be released in nine days:
https://tracker.debian.org/news/1644528/accepted-linux-61137-1-source-into-proposed-updates/

taavi mentioned this in T394883: cloudservices2005-dev backups are clogging all backups.May 21 2025, 7:53 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits