Page MenuHomePhabricator
Create Task
Maniphest T274403

mw1379 - down after reboot attempt and DRAC can't powercycle
Closed, ResolvedPublic
Actions

Description

mw1379 was reimaged and everything went normal, just at the very end after the final reboot triggered by the reimaging script it never came back and stayed down.

I connected to mgmt in an attempt to powercycle it and found this unusual DRAC behaviour:

racadm serveraction powerstatus - ON
racadm serveraction powerup - already up
racadm serveraction powerdown - can't execute command
racadm serveraction powercycle - can't execute command

Then i did a "racadm racreset" and was able to send those commands again.

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedNoneT248925 Make MediaWiki release tarball compatible with PHP 8.0
 ResolvedJdforrester-WMFT300463 Make PHP 8.0 voting on MW master
 ResolvedJdforrester-WMFT313563 Bump lcobucci/jwt & league/uri-components for php8
 ResolvedJdforrester-WMFT313564 Bump onoi/message-reporter in vendor.git to 1.4.2 for php 8 support
 ResolvedJdforrester-WMFT247658 Make Wikimedia Production MediaWiki compatible with PHP 7.4
 Resolved• toanT243590 Fix WikibaseDataModel CI for php 7.4
 ResolvedLucas_Werkmeister_WMDET316923 Restore skipped test in ReferenceListTest.php
 ResolvedJoeT318918 Undeploy patch to use old PHP serialization in PHP 7.4
 ResolvedJdforrester-WMFT264168 Drop PHP 7.2 support from Wikibase master branch, once Wikimedia production is on 7.4
 ResolvedLadsgroupT270740 Drop hacky support for doctrine/dbal class renaming
 InvalidNoneT303505 [S] Remove Deprecated methods "serialize" and "unserialize" after php production upgrade to PHP 7.4
 ResolvedReedyT251043 Cleanup css-sanitizer when it only requires PHP >= 7.4
 OpenNoneT166010 The Great Namespaceization Effort
 ResolvedtstarlingT277618 var_dump() on various objects writes gigabytes of data and takes minutes to run
 ResolvedJdforrester-WMFT261872 Drop PHP 7.2 & 7.3 support from MediaWiki master branch, once Wikimedia production is on 7.4
 StalledNoneT302086 Set scap minimum python version to 3.7
 ResolvedNoneT247045 Migrate all of production metal and VMs to Buster or later
 ResolvedPRODUCTION ERRORLegoktmT293568 PHP Notice: Undefined offset in wikimedia/remex-html when rendering rest.php error page
 ResolvedtstarlingT297667 mysqli/mysqlnd memory leak
 ResolvedJoeT271736 Migrate WMF production from PHP 7.2 to PHP 7.4
 ResolvedKrinkleT248191 Can't reopen table in wikidb-unittest_ (from SpecialPageFatalTest)
 ResolvedhasharT278203 Migrate all CI jobs from stretch to buster or later and drop stretch testing support
 ResolvedJdforrester-WMFT252432 Drop MediaWiki testing in stretch and instead test only in buster
 ResolvedDzahnT245757 Upgrade MediaWiki clusters to Debian Buster (debian 10)
 ResolvedDzahnT274403 mw1379 - down after reboot attempt and DRAC can't powercycle

Event Timeline

Dzahn created this task.Feb 10 2021, 7:11 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 10 2021, 7:11 PM
Dzahn added a project: SRE.Feb 10 2021, 7:16 PM
Dzahn added a parent task: T245757: Upgrade MediaWiki clusters to Debian Buster (debian 10).
Dzahn closed this task as Resolved.Feb 10 2021, 7:33 PM
Dzahn claimed this task.
Papaul subscribed.Feb 10 2021, 11:50 PM

I looked at 3 hosts wmf-auto-reimage .out log, there were no indication of this issue then i looked at the IDRAC log
of 3 of the hosts that are having this issue (mw1377,mw1378 and mw1379) all have an error in the IDRAC log . see below

Log Sequence Number:
118
Detailed Description:
The operating system or an application failed to communicate to the baseboard management controller (BMC) within the timeout period. The system was reset per the configured setting.
Recommended Action:
Check the operating system, application, hardware, and system event log for exception events.
Log Sequence Number:
119
Detailed Description:
The system was reset due to a timeout from the watchdog timer.
Recommended Action:
Check the System Event Log (SEL) or crash dumps from Operating System to identify the source that caused the watchdog timer reset. Update the firmware or driver for the identified device.

mw1381 did complete the auto re-image process with out any issue and don;t have the error.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL