Page MenuHomePhabricator
Create Task
Maniphest T434299

Noisy alerts: investigate fstrim.service failure on dse-k8s-wdqs2001 (new Supermicro host)
Closed, ResolvedPublic
Actions

Description

I noticed a SystemdUnitFailed alert for fstrim.service on dse-k8s-wdqs2001 , an EFI host.

Looking in the systemd journal, I see that fstrim is failing against the EFI partition:

fstrim: /boot/efi: FITRIM ioctl failed: Input/output error

EFI partitions are required to be VFAT-formatted, which is incompatible with fstrim.
Upon further review, VFAT does in fact work with fstrim. The problem is localized to dse-k8s-wdqs2001, so it's either a hardware fault or a drive that doesn't support FITRIM

Creating this ticket to:

  • Investigate why the above operation failed
  • Hopefully find a fix

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
base: don't run fstrim against EFI system partitionBkingoperations/puppetproduction+13 -0
Customize query in gerrit

Event Timeline

bking created this task.Aug 7 2026, 1:58 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 7 2026, 1:58 PM
bking updated the task description. (Show Details)Aug 7 2026, 3:24 PM
bking added a comment.Aug 7 2026, 3:26 PM

I remounted boot/efi on dse-k8s-wdqs2001 with the X-fstrim.notrim mount option. Note that this option doesn't show up in the output of mount:

 mount | grep efi
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
/dev/sda2 on /boot/efi type vfat (rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro)

But I did place it in /etc/fstab:

# /boot/efi was on /dev/nvme0n1p1 during installation
UUID=DA13-823F  /boot/efi       vfat    umask=0077,X-fstrim.notrim      0       1

After remounting with this option, I can confirm that fstrim.service runs without erring:

Aug 07 15:22:46 dse-k8s-wdqs2001 systemd[1]: Starting fstrim.service - Discard unused blocks on filesystems from /etc/fstab...
Aug 07 15:23:09 dse-k8s-wdqs2001 fstrim[1509734]: /var/lib/kubelet: 244.2 MiB (256053248 bytes) trimmed on /dev/mapper/vg0-kubelet
Aug 07 15:23:09 dse-k8s-wdqs2001 fstrim[1509734]: /var/lib/containerd: 520.7 MiB (546004992 bytes) trimmed on /dev/mapper/vg0-containerd
Aug 07 15:23:09 dse-k8s-wdqs2001 fstrim[1509734]: /: 3.9 GiB (4202504192 bytes) trimmed on /dev/mapper/vg0-root
Aug 07 15:23:09 dse-k8s-wdqs2001 systemd[1]: fstrim.service: Deactivated successfully.
Aug 07 15:23:09 dse-k8s-wdqs2001 systemd[1]: Finished fstrim.service - Discard unused blocks on filesystems from /etc/fstab.

The next step is to add these options to Puppet.

gerritbot added a comment.Aug 7 2026, 4:00 PM

Change #1322791 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] base: don't run fstrim against EFI system partition

https://gerrit.wikimedia.org/r/1322791

gerritbot added a project: Patch-For-Review.Aug 7 2026, 4:00 PM
bking updated the task description. (Show Details)Aug 7 2026, 9:36 PM
bking renamed this task from Noisy alerts: stop running fstrim against EFI partition to Noisy alerts: investigate fstrim.service failure on dse-k8s-wdqs2001 (new Supermicro host).Aug 7 2026, 9:39 PM
gerritbot added a comment.Aug 7 2026, 9:42 PM

Change #1322791 abandoned by Bking:

[operations/puppet@production] base: don't run fstrim against EFI system partition

Reason:

fstrim actually does work with vfat; this is a different problem.

https://gerrit.wikimedia.org/r/1322791

bking added a comment.EditedAug 7 2026, 9:49 PM

Correction: the problem is localized to dse-k8s-wdqs2001, a brand-new SuperMicro host. None of the other wdqs hosts, including those with the same chassis as dse-k8s-wdqs2001, are exhibiting this problem:

 sudo cumin dse-k8s-wdqs* "journalctl -u fstrim.service | grep error"
9 hosts will be targeted:
dse-k8s-wdqs[2001-2004].codfw.wmnet,dse-k8s-wdqs[1001-1003].eqiad.wmnet,dse-k8s-wdqs-test2001.codfw.wmnet,dse-k8s-wdqs-test1001.eqiad.wmnet
OK to proceed on 9 hosts? Enter the number of affected hosts to confirm or "q" to quit: 9
===== NODE GROUP =====
(1) dse-k8s-wdqs2001.codfw.wmnet
----- OUTPUT for command #1: 'journalctl -u fs...ice | grep error' -----
Jun 22 00:45:29 dse-k8s-wdqs2001 fstrim[1885754]: fstrim: /boot/efi: FITRIM ioctl failed: Input/output error
Jun 29 01:19:49 dse-k8s-wdqs2001 fstrim[3113435]: fstrim: /boot/efi: FITRIM ioctl failed: Input/output error
Jul 06 00:38:15 dse-k8s-wdqs2001 fstrim[144012]: fstrim: /boot/efi: FITRIM ioctl failed: Input/output error
Jul 13 01:02:26 dse-k8s-wdqs2001 fstrim[1473089]: fstrim: /boot/efi: FITRIM ioctl failed: Input/output error
Jul 20 00:56:27 dse-k8s-wdqs2001 fstrim[3422099]: fstrim: /boot/efi: FITRIM ioctl failed: Input/output error
Jul 27 00:21:45 dse-k8s-wdqs2001 fstrim[497929]: fstrim: /boot/efi: FITRIM ioctl failed: Input/output error
Aug 03 01:07:14 dse-k8s-wdqs2001 fstrim[1791809]: fstrim: /boot/efi: FITRIM ioctl failed: Input/output error
================

So this becomes a question for DC Ops.

DC Ops, have y'all seen this error before? It only seems to affect the one host, and only on its EFI partition. The host is a pretty new SuperMicro SYS-111E-WR if that helps. Just wondering if we need to upgrade firmware or anything like that. This host is not running production workloads, but please do ping us in IRC (inflatador) or Data-Platform-SRE Slack if you need to reboot.

bking added a project: ops-codfw.Aug 7 2026, 9:49 PM
Restricted Application added a project: DC-Ops. · View Herald TranscriptAug 7 2026, 9:49 PM
bking moved this task from Backlog - project to Tracking on the Data-Platform-SRE (2026-07-03 - 2026-07-31) board.Aug 7 2026, 9:49 PM
bking updated the task description. (Show Details)
Maintenance_bot added a project: SRE.Aug 7 2026, 10:29 PM
Maintenance_bot removed a project: Patch-For-Review.
Gehel edited projects, added: Data-Platform-SRE (2026-08-07 - 2026-08-28); removed: Data-Platform-SRE (2026-07-03 - 2026-07-31).Tue, Aug 11, 6:56 AM
Gehel moved this task from Backlog - project to Tracking on the Data-Platform-SRE (2026-08-07 - 2026-08-28) board.
Jhancock.wm moved this task from Backlog to Hardware Failure / Troubleshoot on the ops-codfw board.Tue, Aug 11, 1:39 PM
Jhancock.wm subscribed.Tue, Aug 11, 5:09 PM

I've never seen it before. I did check the version of the bios and the bmc. there are new versions available, but other dse-k8s-worker servers have the same firmware version on them right now. I do not think it would be a firmware issue. further I found some complaints online that newer firmware versions in general are not helping with fstrim issues.

ops-monitoring-bot added a comment.Thu, Aug 13, 7:58 PM

Icinga downtime and Alertmanager silence (ID=0525ce6b-a5e4-405d-a19f-e9dafc4c055f) set by robh@cumin2002 for 2:00:00 on 1 host(s) and their services with reason: updating firmware

dse-k8s-wdqs2001.codfw.wmnet
ops-monitoring-bot added a comment.Thu, Aug 13, 10:05 PM

Icinga downtime and Alertmanager silence (ID=811db3f2-efb2-4b2b-a3f3-29808791321d) set by robh@cumin2002 for 2:00:00 on 1 host(s) and their services with reason: updating firmware

dse-k8s-wdqs2001.codfw.wmnet
Jhancock.wm added a comment.Thu, Aug 13, 11:43 PM

firmware updates are completed. please test out the fitrim when you can to see if it fixed it.

Gehel moved this task from Tracking to In Progress on the Data-Platform-SRE (2026-08-07 - 2026-08-28) board.Fri, Aug 14, 4:02 PM
bking changed the task status from Open to In Progress.Thu, Aug 27, 6:49 PM
bking claimed this task.
bking triaged this task as Low priority.
bking closed this task as Resolved.Thu, Aug 27, 7:05 PM
bking moved this task from In Progress to Done on the Data-Platform-SRE (2026-08-07 - 2026-08-28) board.

Per IRC conversation with @Jhancock.wm , I tried running the fstrim.service and it is still failing. To prevent noisy alerts, I've exempted the EFI partition from fstrim.service via /etc/fstab:

UUID=DA13-823F /boot/efi vfat umask=0077,X-fstrim.notrim 0 1

Since fstrim seems to be working against other partitions, and this problem isn't happening on any other hosts that have the same hardware, I'll go ahead and close the ticket for now. We can always re-open if this starts happening on other hosts.

Gehel moved this task from Done to Reported on the Data-Platform-SRE (2026-08-07 - 2026-08-28) board.Fri, Aug 28, 7:40 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits