Page MenuHomePhabricator
Create Task
Maniphest T273803

mw2220 - broken IPMI / mgmt
Closed, ResolvedPublic
Actions

Description

mw2220.codfw.wmnet has broken IPMI / mgmt / DRAC.

It showed up when using the reimaging script failed with failed remote IPMI.

I started the debugging steps from https://wikitech.wikimedia.org/wiki/Management_Interfaces#Does_IPMI_work_locally?

and it fails right away locally.. and not with the "typical" error but an unusual one

[mw2220:~] $ sudo ipmi-chassis --get-chassis-status
ipmi_cmd_get_chassis_status: bad completion code

Also the other test commands, fail in unusual ways:

[mw2220:~] $ sudo ipmi-config --section=Lan_Channel --key-pair="Lan_Channel:Volatile_Access_Mode=Always_Available" --key-pair="Lan_Channel:Non_Volatile_Access_Mode=Always_Available" --diff
Unable to get Number of Users

And finally I also can't ssh to mgmt to reset DRAC:

ssh root@mw2220.mgmt.codfw.wmnet

channel 0: open failed: connect failed: Connection timed out
stdio forwarding failed
ssh_exchange_identification: Connection closed by remote host

Could you please check on it locally?

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedNoneT248925 Make MediaWiki release tarball compatible with PHP 8.0
 ResolvedJdforrester-WMFT300463 Make PHP 8.0 voting on MW master
 ResolvedJdforrester-WMFT313563 Bump lcobucci/jwt & league/uri-components for php8
 ResolvedJdforrester-WMFT313564 Bump onoi/message-reporter in vendor.git to 1.4.2 for php 8 support
 ResolvedJdforrester-WMFT247658 Make Wikimedia Production MediaWiki compatible with PHP 7.4
 Resolved• toanT243590 Fix WikibaseDataModel CI for php 7.4
 ResolvedLucas_Werkmeister_WMDET316923 Restore skipped test in ReferenceListTest.php
 ResolvedJoeT318918 Undeploy patch to use old PHP serialization in PHP 7.4
 ResolvedJdforrester-WMFT264168 Drop PHP 7.2 support from Wikibase master branch, once Wikimedia production is on 7.4
 ResolvedLadsgroupT270740 Drop hacky support for doctrine/dbal class renaming
 InvalidNoneT303505 [S] Remove Deprecated methods "serialize" and "unserialize" after php production upgrade to PHP 7.4
 ResolvedReedyT251043 Cleanup css-sanitizer when it only requires PHP >= 7.4
 OpenNoneT166010 The Great Namespaceization Effort
 ResolvedtstarlingT277618 var_dump() on various objects writes gigabytes of data and takes minutes to run
 ResolvedJdforrester-WMFT261872 Drop PHP 7.2 & 7.3 support from MediaWiki master branch, once Wikimedia production is on 7.4
 StalledNoneT302086 Set scap minimum python version to 3.7
 ResolvedNoneT247045 Migrate all of production metal and VMs to Buster or later
 ResolvedPRODUCTION ERRORLegoktmT293568 PHP Notice: Undefined offset in wikimedia/remex-html when rendering rest.php error page
 ResolvedtstarlingT297667 mysqli/mysqlnd memory leak
 ResolvedJoeT271736 Migrate WMF production from PHP 7.2 to PHP 7.4
 ResolvedKrinkleT248191 Can't reopen table in wikidb-unittest_ (from SpecialPageFatalTest)
 ResolvedhasharT278203 Migrate all CI jobs from stretch to buster or later and drop stretch testing support
 ResolvedJdforrester-WMFT252432 Drop MediaWiki testing in stretch and instead test only in buster
 ResolvedDzahnT245757 Upgrade MediaWiki clusters to Debian Buster (debian 10)
 ResolvedPapaulT273803 mw2220 - broken IPMI / mgmt

Event Timeline

Dzahn created this task.Feb 3 2021, 6:38 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 3 2021, 6:38 PM
Dzahn triaged this task as Medium priority.Feb 3 2021, 6:38 PM
Dzahn added projects: serviceops-radar, ops-codfw.
Dzahn added a parent task: T245757: Upgrade MediaWiki clusters to Debian Buster (debian 10).
Maintenance_bot added a project: SRE.Feb 3 2021, 6:45 PM
wiki_willy assigned this task to Papaul.Feb 5 2021, 10:27 PM
wiki_willy added a project: DC-Ops.
Papaul added a comment.Feb 8 2021, 11:46 PM

@Dzahn if the server is still in service can it please be de-pool so i can work on it tomorrow while on site.

Thanks

Stashbot added a comment.Feb 8 2021, 11:52 PM

Mentioned in SAL (#wikimedia-operations) [2021-02-08T23:52:46Z] <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on mw2220.codfw.wmnet with reason: T273803

Stashbot added a comment.Feb 8 2021, 11:52 PM

Mentioned in SAL (#wikimedia-operations) [2021-02-08T23:52:51Z] <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on mw2220.codfw.wmnet with reason: T273803

Dzahn added a comment.Feb 8 2021, 11:54 PM

@Papaul Server is set to pooled=inactive and downtime for 2 days. Go ahead and thank you!

Papaul added a comment.Feb 8 2021, 11:54 PM

Thank you will work on it tomorrow.

Papaul moved this task from Backlog to Hardware Failure / Troubleshoot on the ops-codfw board.Feb 9 2021, 3:29 PM
Papaul closed this task as Resolved.Feb 9 2021, 4:05 PM

@Dzahn
Drained power and upgrade IDRAC firmware from 2.30.30.30 to 2.63.

All looks good now

Dzahn added a comment.Feb 9 2021, 5:04 PM

Thank you @Papaul. Reimaging it now and things look normal.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL