Page MenuHomePhabricator
Create Task
Maniphest T243963

es1019: reseat IPMI
Closed, ResolvedPublic
Actions

Description

Looks like es1019 has an issue with the IPMI.
Per icinga:

ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-es1019.localhost: internal IPMI error

I wanted to soft restart the IPMI, but I cannot access it, it looks down.

ssh es1019.mgmt.eqiad.wmnet -lroot
channel 0: open failed: connect failed: Connection timed out


root@cumin1001:~# ping es1019.mgmt.eqiad.wmnet
PING es1019.mgmt.eqiad.wmnet (10.65.4.44) 56(84) bytes of data.
^C
--- es1019.mgmt.eqiad.wmnet ping statistics ---
128 packets transmitted, 0 received, 100% packet loss, time 130024ms

I am not sure if the IPMI on-site reseat needs the host to go fully down, if so, please coordinate with me, as we'd need to depool it.

This is not the first occurrence on this host:
T201132
T187530
T213422
T155691

Details

SubjectRepoBranchLines +/-
es1019: Disable notificationsoperations/puppetproduction+1 -0
Customize query in gerrit

Related Objects

Event Timeline

• Marostegui triaged this task as Medium priority.Jan 30 2020, 5:35 PM
• Marostegui created this task.
• Marostegui moved this task from Triage to Blocked external/Not db team on the DBA board.
jcrespo subscribed.Jan 30 2020, 5:37 PM

Strange, not as if it has happened before! T120689 T155691 T187530 T201132 T213422 T233698

• Marostegui updated the task description. (Show Details)Jan 30 2020, 5:38 PM
wiki_willy assigned this task to • Cmjohnson.Jan 30 2020, 11:19 PM
wiki_willy added a project: DC-Ops.
• Cmjohnson added a comment.Feb 4 2020, 8:00 PM

@jcrespo yes, this seems to be an issue and when it's down I will see if there are any f/w updates for it. Is this something you need right away? I can put it on the schedule for Thursday but it will not be until my afternoon.

• Marostegui added a comment.Feb 4 2020, 8:46 PM

@Cmjohnson - Thursday sounds good. I can leave the host depooled, downtimed and off, so you can tackle it in your afternoon.
Just leave it powered on once you are done and I will take care of it in Friday when I get online.

Thank you!

jcrespo added a comment.Feb 5 2020, 1:40 AM

yes, this seems to be an issue

I hope you understood this was a rant directed towards the machine/vendor
only and for background info. I don't think we will get rid of it until we
renew the hw, but ofc no problem on trying an upgrade.

Stashbot added a comment.Feb 6 2020, 1:51 PM

Mentioned in SAL (#wikimedia-operations) [2020-02-06T13:51:58Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool es1019 for onsite maintenance T243963', diff saved to https://phabricator.wikimedia.org/P10321 and previous config saved to /var/cache/conftool/dbconfig/20200206-135157-marostegui.json

gerritbot added a comment.Feb 6 2020, 1:53 PM

Change 570643 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] es1019: Disable notifications

https://gerrit.wikimedia.org/r/570643

gerritbot added a project: Patch-For-Review.Feb 6 2020, 1:53 PM

Change 570643 merged by Marostegui:
[operations/puppet@production] es1019: Disable notifications

https://gerrit.wikimedia.org/r/570643

Stashbot added a comment.Feb 6 2020, 1:55 PM

Mentioned in SAL (#wikimedia-operations) [2020-02-06T13:55:11Z] <marostegui> Stop MySQL on es1019, upgrade and poweroff for on-site maintenance - T243963

• Marostegui added a comment.Feb 6 2020, 1:58 PM

@Cmjohnson es1019 is off.
Once you are done, just start it back and I will it from there.

Thank you!

Maintenance_bot removed a project: Patch-For-Review.Feb 6 2020, 2:11 PM
• Marostegui added a comment.Feb 7 2020, 6:11 AM

From what I can see this maintenance did not happen yesterday in the end - as the host is still off and its IPMI is unreachable. And as the IPMI is involved, I cannot power this host back on for the weekend.
Even if we don't troubleshoot the IPMI today, can someone power it back on today so we don't go thru the weekend without it - and also provide me a new date for the IPMI reseat?

Thanks

Jclark-ctr added a comment.Feb 7 2020, 5:12 PM

performed flea power drain. powered on host

• Marostegui added a comment.Feb 7 2020, 5:13 PM

Thanks - IPMI is back. I will take it from here.
Thank you!

Stashbot added a comment.Feb 7 2020, 5:19 PM

Mentioned in SAL (#wikimedia-operations) [2020-02-07T17:19:04Z] <marostegui> Start MySQL on es1019 after onsite maintenance T243963

Stashbot added a comment.Feb 7 2020, 5:25 PM

Mentioned in SAL (#wikimedia-operations) [2020-02-07T17:25:42Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool es1019 after on-site maintenance T243963', diff saved to https://phabricator.wikimedia.org/P10349 and previous config saved to /var/cache/conftool/dbconfig/20200207-172541-marostegui.json

• Marostegui added a comment.Feb 7 2020, 5:36 PM

John found this: https://www.dell.com/support/article/es/es/esbsdt1/sln316859/idrac7-idrac8-idrac-unresponsive-or-sluggish-performance?lang=en which is an update from May 2019, so maybe we should try it.
Should we schedule a maintenance window to get it update?

Stashbot added a comment.Feb 7 2020, 5:38 PM

Mentioned in SAL (#wikimedia-operations) [2020-02-07T17:38:50Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool es1019 after on-site maintenance T243963', diff saved to https://phabricator.wikimedia.org/P10350 and previous config saved to /var/cache/conftool/dbconfig/20200207-173850-marostegui.json

• Cmjohnson added a comment.Feb 12 2020, 4:05 PM

@Marostegui We can upgrade the f/w. That can be anytime, please pick a convenient date for you.

• Marostegui added a comment.Feb 12 2020, 4:06 PM
In T243963#5877162, @Cmjohnson wrote:

@Marostegui We can upgrade the f/w. That can be anytime, please pick a convenient date for you.

Can we do it tomorrow at the most convenient time for you?

• Marostegui added a comment.Feb 12 2020, 4:08 PM

@Cmjohnson I can have the host depooled and off tomorrow in the UTC morning so you can do it whenever you can tomorrow, and once done, just power it back on. Would that work?

wiki_willy added a comment.Feb 13 2020, 4:47 PM

@Cmjohnson - looks like it's too late to do this one today, since they need 24hrs to depool. Chatted with @Marostegui briefly, so just let know them know when's a good date for you to upgrade the firmware, and they can prep for it then. Thanks, Willy

• Cmjohnson moved this task from Backlog to Hardware Failure / Troubleshoot on the ops-eqiad board.Feb 19 2020, 4:24 PM
• Marostegui added a comment.Feb 24 2020, 3:49 PM
In T243963#5881470, @wiki_willy wrote:

@Cmjohnson - looks like it's too late to do this one today, since they need 24hrs to depool. Chatted with @Marostegui briefly, so just let know them know when's a good date for you to upgrade the firmware, and they can prep for it then. Thanks, Willy

Any ETA?
Thanks!

Stashbot added a comment.Feb 25 2020, 12:46 PM

Mentioned in SAL (#wikimedia-operations) [2020-02-25T12:46:51Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool es1019 for on-site maintenance - T243963', diff saved to https://phabricator.wikimedia.org/P10512 and previous config saved to /var/cache/conftool/dbconfig/20200225-124650-marostegui.json

Stashbot added a comment.Feb 25 2020, 12:51 PM

Mentioned in SAL (#wikimedia-operations) [2020-02-25T12:51:47Z] <marostegui> Stop mysql on es1019 - T243963

• Cmjohnson added a comment.Feb 25 2020, 2:37 PM

I attempted to update the idrac f/w for es1019 but the update failed several times for not being able to verify package signature. The update was downloaded directly from dell's portal and I did not see any way of verifying the signature. I did remove and download it again and still failed. I also attempted to load directly from a USB stick but the file was not seen by the lifecycle controller and failed.

JID_826621862819 Firmware Update: iDRAC Failed

Start Time: Not Applicable
Expiration Time: Not Applicable
Message: RED007: Unable to verify Update Package signature.

• Marostegui added a comment.Feb 25 2020, 3:10 PM

Thanks Chris for tackling this.
Let's not spend more time on this host, it has a big history of failing idrac :(
So let's just make sure it is available and if it fails in a few months again, let's reseat it again.
This host is meant to be refreshed next FY, so we will be able to get rid of it in a few months time hopefully.

Thanks!

jcrespo added a comment.Feb 25 2020, 3:13 PM

Will close this, then, once the host is fully back into production.

• Cmjohnson closed this task as Resolved.Feb 25 2020, 3:56 PM

The reseat was completed but idrac f/w updated failed. resolving the task and will do flea power drains if or when idrac freezes again.

Stashbot added a comment.Feb 25 2020, 4:02 PM

Mentioned in SAL (#wikimedia-operations) [2020-02-25T16:02:15Z] <jynus@cumin1001> dbctl commit (dc=all): 'repool es1019 with low load after maintenance T243963', diff saved to https://phabricator.wikimedia.org/P10516 and previous config saved to /var/cache/conftool/dbconfig/20200225-160215-jynus.json

Stashbot added a comment.Feb 25 2020, 5:21 PM

Mentioned in SAL (#wikimedia-operations) [2020-02-25T17:21:34Z] <jynus@cumin1001> dbctl commit (dc=all): 'increase es1019 load to 50% T243963', diff saved to https://phabricator.wikimedia.org/P10519 and previous config saved to /var/cache/conftool/dbconfig/20200225-172133-jynus.json

jcrespo added a comment.Feb 25 2020, 5:51 PM

es1019 is just pending the last config push back to normal traffic weights (and reducing the master's).

Stashbot added a comment.Feb 26 2020, 6:16 AM

Mentioned in SAL (#wikimedia-operations) [2020-02-26T06:16:41Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Restore es1017 (master) original weight (0) T243963', diff saved to https://phabricator.wikimedia.org/P10529 and previous config saved to /var/cache/conftool/dbconfig/20200226-061640-marostegui.json

Stashbot added a comment.Feb 26 2020, 6:17 AM

Mentioned in SAL (#wikimedia-operations) [2020-02-26T06:17:11Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool es1019 after on-site maintenance T243963', diff saved to https://phabricator.wikimedia.org/P10530 and previous config saved to /var/cache/conftool/dbconfig/20200226-061710-marostegui.json

• Marostegui added a comment.Feb 26 2020, 6:18 AM
In T243963#5916919, @jcrespo wrote:

es1019 is just pending the last config push back to normal traffic weights (and reducing the master's).

Done!
Thank you for handling this

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL