Page MenuHomePhabricator
Create Task
Maniphest T253374

Stale nic firmware files on some hosts
Closed, ResolvedPublic
Actions

Description

Looks like some hosts have stale files for nic firmware, I've poked on thanos-be2001 and indeed the timer for prometheus-nic-firmware-textfile showed n/a as if the timer never started, starting/stopping the related service seems to have fixed the issue (?) I have left the other hosts untouched

WARNING	2020-05-22 08:29:40	0d 15h 59m 22s	3/3	cluster={misc,thanos} file=nic_firmware.prom instance={thanos-be2001:9100,thanos-be2002:9100,thanos-be2003:9100,thanos-be2004:9100,thanos-fe2001:9100,thanos-fe2002:9100,thanos-fe2003:9100} job=node site=codfw	
Stale file for node-exporter textfile in eqiad

WARNING	2020-05-22 08:14:46	0d 11h 25m 22s	3/3	cluster=mysql file=nic_firmware.prom instance=db1140:9100 job=node site=eqiad

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+22 -25systemd::timer::job: unkludge OnUnitInactiveSec/OnUnitActiveSec
operations/puppetproduction+12 -4node_nic_firmware: don't brick the timer on reboot
Customize query in gerrit

Related Objects
Search...

Event Timeline

fgiunchedi created this task.May 22 2020, 8:36 AM
ayounsi removed a subscriber: ayounsi.May 22 2020, 8:41 AM
CDanis added a comment.May 22 2020, 12:07 PM

Thanks for the report, I'll have to dig through Puppet logs I guess.

Here's all eqiad hosts that have an mtime more than 10 minutes in the past:
https://w.wiki/RYU

In codfw it's just the thanos-{fe,be} hosts.

ulsfo and esams are fine, eqsin has one host (cp5012).

CDanis added a subscriber: Joe.May 22 2020, 12:51 PM

P11283 contains syslog from one of the hosts where it didn't start properly.

We see the service unit being installed, a systemctl daemon-reload, the timer unit being installed, then three systemctl daemon-reloads interleaved with the timer unit being started (the timer's description is Periodic execution of prometheus-nic-firmware-textfile.service), then we see the Puppet exec block that @Joe and I added invoking a systemctl start on the service unit.

I compared logs from thanos-be2002 (where it didn't work) and dns1001 (where it did) and they're practically identical; the only difference that shows up on thanos-be2002 is these two lines in the first systemd daemon-reload:

May 19 15:05:18 thanos-be2002 systemd[1]: serial-getty@ttyS1.service: Current command vanished from the unit file, execution of the command list won't be resumed.
May 19 15:05:18 thanos-be2002 systemd[1]: getty@tty1.service: Current command vanished from the unit file, execution of the command list won't be resumed.

Hard to imagine that being related, though...

gerritbot added a comment.May 22 2020, 1:10 PM

Change 598042 had a related patch set uploaded (by CDanis; owner: CDanis):
[operations/puppet@production] node_nic_firmware: add timer schedules

https://gerrit.wikimedia.org/r/598042

gerritbot added a project: Patch-For-Review.May 22 2020, 1:10 PM
gerritbot added a comment.May 22 2020, 2:07 PM

Change 598050 had a related patch set uploaded (by CDanis; owner: CDanis):
[operations/puppet@production] systemd::timer::job: unkludge OnUnitInactiveSec/OnUnitActiveSec

https://gerrit.wikimedia.org/r/598050

gerritbot added a comment.May 22 2020, 2:08 PM

Change 598042 abandoned by CDanis:
node_nic_firmware: don't brick the timer on reboot

Reason:
superseded by I3d37ea50

https://gerrit.wikimedia.org/r/598042

gerritbot added a comment.May 22 2020, 3:12 PM

Change 598050 merged by CDanis:
[operations/puppet@production] systemd::timer::job: unkludge OnUnitInactiveSec/OnUnitActiveSec

https://gerrit.wikimedia.org/r/598050

Stashbot added a comment.May 22 2020, 3:25 PM

Mentioned in SAL (#wikimedia-operations) [2020-05-22T15:25:47Z] <cdanis> fixing prometheus-nic-firmware-textfile.service wherever it is broken T253374

CDanis closed this task as Resolved.EditedMay 22 2020, 3:36 PM

Fixed, and shouldn't happen again for this or other similar systemd::timer::jobs. See 598050's patch description for deets

Maintenance_bot removed a project: Patch-For-Review.May 22 2020, 4:10 PM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL