Some of these may have problems rebooting by getting stuck at Loading ramdisk... T214840
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| db-codfw.php: Repool db2080 | operations/mediawiki-config | master | +2 -2 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Declined | None | T216240 Reboot, upgrade firmware and kernel of db1096-db1106, db2071-db2092 | |||
| Resolved | • Cmjohnson | T228732 Upgrade db1100 firmware and BIOS | |||
| Resolved | • Cmjohnson | T235877 db1105 rebooted itself | |||
| Resolved | Marostegui | T237905 Upgrade db2072 firmware and bios | |||
| Resolved | Papaul | T241103 Upgrade BIOS and firmware on db2084 | |||
| Resolved | Papaul | T241647 Upgrade BIOS and firmware on db2076 | |||
| Resolved | Papaul | T254139 db2075 failed to boot kernel 2/3 tries, please upgrade firmware/BIOS to mitigate | |||
| Resolved | • Kormat | T267220 db2077 hung on reboot | |||
| Resolved | Papaul | T276909 Upgrade firmware on db2073 | |||
| Resolved | • Cmjohnson | T277007 Upgrade firmware on db1136 | |||
| Resolved | Jclark-ctr | T286226 Upgrade db1104 firmware | |||
| Resolved | Papaul | T293740 Upgrade db2112 firmware/BIOS |
@Papaul from next week, please also upgrade firmware/BIOS of db2089 (only that one for now). I will put it back to production for now.
I have rebooted db2085 without debug option on kernel as part of (T216273) and I have taken the opportunity to upgrade its kernel too.
Mentioned in SAL (#wikimedia-operations) [2019-02-19T07:46:32Z] <marostegui> Reboot db1106 for kernel upgrade (and remove debug from kernel) T216240 T216273
Mentioned in SAL (#wikimedia-operations) [2019-02-19T14:16:04Z] <jynus> stop db2090 for reboot testing T216240
Rebooting db2090:
PowerEdge R630 BIOS Version: 2.4.3
1st reboot: OK 2nd reboot: FAIL 3rd reboot: OK 4th reboot: OK 5th reboot: OK 6th reboot: 7th reboot: 8th reboot: 9th reboot: 10th reboot: 11th reboot: 12th reboot: 13th reboot: 14th reboot:
Mentioned in SAL (#wikimedia-operations) [2019-02-19T14:53:11Z] <jynus> stopping db2089 for hw maintenance T216240
Rebooting db2090:
PowerEdge R630 BIOS Version: 2.4.3
1st reboot: OK 2nd reboot: FAIL 3rd reboot: OK 4th reboot: OK 5th reboot: OK 6th reboot: OK 7th reboot: OK 8th reboot: OK 9th reboot: OK 10th reboot: OK 11th reboot: OK 12th reboot: OK 13th reboot: OK 14th reboot: OK
Rebooting db2089:
PowerEdge R630 BIOS Version: 2.9.1
1st reboot: OK 2nd reboot: OK 3rd reboot: OK 4th reboot: OK 5th reboot: OK 6th reboot: OK 7th reboot: OK 8th reboot: OK 9th reboot: OK 10th reboot: OK 11th reboot: OK
So I believe this is still an ongoing issue, but the remaining hosts may have a lower probability of failing (less than 1 out of 10), so I will stall this and do it only when other hw or sw maintenance is due.
Mentioned in SAL (#wikimedia-operations) [2019-04-24T13:37:44Z] <marostegui> Poweroff db2080 for onsite maintenance - T216240
Before:
BIOS Version
2.4.3
Firmware Version
2.40.40.40
IP Address(es)
10.193.1.75
iDRAC MAC Address
84:7B:EB:F6:99:B2
DNS Domain Name
Lifecycle Controller Firmware
2.40.40.40
After
Service Tag
JCBSDH2
Express Service Code
42104258102
BIOS Version
2.9.1
Firmware Version
2.61.60.60
IP Address(es)
10.193.1.75
iDRAC MAC Address
84:7B:EB:F6:99:B2
DNS Domain Name
Lifecycle Controller Firmware
2.61.60.60
Complete.
Change 506349 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-codfw.php: Repool db2080
Change 506349 merged by jenkins-bot:
[operations/mediawiki-config@master] db-codfw.php: Repool db2080
Mentioned in SAL (#wikimedia-operations) [2019-04-25T06:14:05Z] <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Repool db2080 after onsite maintenance to upgrade BIOS and firmware - T216240 (duration: 00m 54s)
Mentioned in SAL (#wikimedia-operations) [2019-07-08T15:15:32Z] <jynus> shutting down db2097 T225378 T216240
When doing a reboot for T273280 I just ran into this issue with db2073.
db2072 (which as a newer firmware rebooted just fine).
I am going to close this as declined as we're going to refresh all these hosts in Q3 via T297646 and T297648, so no more of the already hosts will be upgraded (unless something punctual happens between now and the date of the refresh).