Maniphest T427388

db2212 failed to reboot
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	ops-monitoring-bot
	Wed, May 27, 12:30 PM

Project Tags

Referenced Files

None

Subscribers

ops-monitoring-bot

Description

db2212 failed to reboot as part of T426633

The host hasn't come back yet after rebooting at:

2026-05-27 12:07:22.661071 cumin db2212.codfw.wmnet "reboot-host" --force --no-progress

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	db2212: Enable notifications	operations/puppet	production	+0 -1
	db2212: Disable notifications	operations/puppet	production	+2 -1

Customize query in gerrit

Event Timeline

ops-monitoring-bot created this task.Wed, May 27, 12:30 PM

ops-monitoring-bot triaged this task as High priority.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptWed, May 27, 12:30 PM

FCeratto-WMF added projects: DC-Ops, ops-codfw.Wed, May 27, 12:57 PM

FCeratto-WMF updated the task description. (Show Details)

There are no events in getsel after 06/13/2025 14:24:15

Mentioned in SAL (#wikimedia-operations) [2026-05-27T13:05:57Z] <fceratto@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 99 days, 0:00:00 on db2212.codfw.wmnet with reason: failed to reboot T427388 T426633

Maintenance_bot added a project: SRE.Wed, May 27, 1:29 PM

Jhancock.wm moved this task from Backlog to Hardware Failure / Troubleshoot on the ops-codfw board.Wed, May 27, 2:22 PM

it halted in the boot and i had to pull the power entirely to get it to reboot and make it past post. There still isn't anything new in the event logs. Can I update the firmware for this one? or do you need it back quick since it's high priority

In T427388#11960531, @Jhancock.wm wrote:

it halted in the boot and i had to pull the power entirely to get it to reboot and make it past post. There still isn't anything new in the event logs. Can I update the firmware for this one? or do you need it back quick since it's high priority

Yes please, update everything that can be updated :)
Thanks!

Change #1294807 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db2212: Disable notifications

https://gerrit.wikimedia.org/r/1294807

Change #1294807 merged by Marostegui:

[operations/puppet@production] db2212: Disable notifications

https://gerrit.wikimedia.org/r/1294807

Maintenance_bot removed a project: Patch-For-Review.Thu, May 28, 6:31 AM

Marostegui moved this task from Triage to In progress on the DBA board.Thu, May 28, 7:18 AM

@Marostegui all done.

The host was shut down cleanly so I can check and repool it.

No error in the logs, replication is catching up.

FCeratto-WMF claimed this task.Thu, May 28, 2:46 PM

Change #1295389 had a related patch set uploaded (by Federico Ceratto; author: Federico Ceratto):

[operations/puppet@production] Enable notifications for db2212

https://gerrit.wikimedia.org/r/1295389

gerritbot added a project: Patch-For-Review.Fri, May 29, 8:07 AM

Change #1295389 merged by Federico Ceratto:

[operations/puppet@production] db2212: Enable notifications

https://gerrit.wikimedia.org/r/1295389

Maintenance_bot removed a project: Patch-For-Review.Fri, May 29, 9:31 AM

Starting pool of db2212 by fceratto@cumin1003: Pooling

Completed pooling of db2212 by fceratto@cumin1003: Pooling

FCeratto-WMF closed this task as Resolved.Fri, May 29, 11:44 AM

Maintenance_bot moved this task from In progress to Done on the DBA board.Fri, May 29, 12:29 PM