Page MenuHomePhabricator
Create Task
Maniphest T274751

Upgrade firmware on wdqs1009
Closed, ResolvedPublic
Actions

Description

wdqs1009 crashed around Feb 13 15:41:01. Syslog and System Event Log don't show anything obviously suspicious. @MoritzMuehlenhoff suggested updating the firmware, which might resolve the issue and would probably be the first step if we need to report this to the vendor.

Related Objects

Event Timeline

Gehel created this task.Feb 15 2021, 8:50 AM
Restricted Application added a project: Wikidata. · View Herald TranscriptFeb 15 2021, 8:50 AM
Maintenance_bot added a project: SRE.Feb 15 2021, 9:45 AM
Restricted Application added a project: [DEPRECATED] wdwb-tech. · View Herald TranscriptFeb 15 2021, 9:45 AM
MoritzMuehlenhoff triaged this task as Medium priority.Feb 15 2021, 12:37 PM
Gehel moved this task from Incoming to Current work on the Wikidata-Query-Service board.Feb 15 2021, 4:04 PM
Gehel added a project: Discovery-Search (Current work).
Gehel moved this task from Incoming to Blocked/Waiting on the Discovery-Search (Current work) board.Feb 15 2021, 4:33 PM
wiki_willy assigned this task to Jclark-ctr.Feb 16 2021, 8:09 PM
• Cmjohnson subscribed.Feb 23 2021, 4:27 PM

@Gehel updating firmware requires rebooting a few times. Let me know if this will be okay, if it needs to be a scheduled downtime, Let's plan for tomorrow Wednesday at 1400UTC or Thursday same time

Gehel added a subscriber: RKemper.Feb 23 2021, 4:30 PM
In T274751#6853205, @Cmjohnson wrote:

@Gehel updating firmware requires rebooting a few times. Let me know if this will be okay, if it needs to be a scheduled downtime, Let's plan for tomorrow Wednesday at 1400UTC or Thursday same time

We have a data reload in progress at the moment, I'd prefer not to interrupt it (T267927). I'll ping you (or @RKemper will) once it's done. This is a test server, so once the data reload is done, we can shutdown that server whenever, with no user impact.

• Cmjohnson claimed this task.Feb 23 2021, 5:04 PM
• Cmjohnson added a subscriber: Jclark-ctr.

Perfect! thanks

• Cmjohnson moved this task from Backlog to Hardware Failure / Troubleshoot on the ops-eqiad board.Feb 24 2021, 1:54 PM
RKemper added a comment.Feb 26 2021, 3:47 AM

@Cmjohnson The data reload is complete on wdqs1009, so the host can now have its firmware upgraded and be rebooted at its convenience. Note this is an internal wdqs test host, so there is no public-facing service for us to worry about.

Feel free to proceed when convenient for you - just shoot me a ping or drop a line in #wikimedia-discovery, but no need to schedule a formal window.

Gehel added a comment.Feb 27 2021, 6:38 PM

Note that wdqs1011 had a similar issue today (might not be related at all)

Addshore moved this task from Inbox to External Realm on the [DEPRECATED] wdwb-tech board.Mar 5 2021, 12:19 PM
Stashbot added a comment.Mar 11 2021, 3:53 PM

Mentioned in SAL (#wikimedia-operations) [2021-03-11T15:53:13Z] <cmjohnson1> updating firmware wdqs1009 T274751

• Cmjohnson closed this task as Resolved.Mar 11 2021, 4:35 PM

updated BIOS, IDRAC and NIC firmware. Resolving the task, if an issue persists please open a new task with the error.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits