Page MenuHomePhabricator
Create Task
Maniphest T440033

an-druid1003 Bootlooping: able to reseat/power drain?
Closed, ResolvedPublic
Actions

Description

Hello DC Ops,

Per parent ticket, an-druid1003 was rebooted to apply OS updates. Unfortunately, the server has been bootlooping since it was rebooted.

The DRAC and BIOS firmware is up-to-date and I just updated the NIC firmware, but it doesn't seem to have made a difference. I've powered down the host so it doesn't bootloop endlessly.

Are you able to power drain the host and reseat the drives? We are in danger of running out of disk space on the an-druid cluster, recovering this could help mitigate that.

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
install_server: Reuse the 8 disk recipe for an-druid100[67]Btullisoperations/puppetproduction+3 -3
install_server: Rebuild an-druid100[67] RAID10 across all 8 disksBtullisoperations/puppetproduction+4 -1
install_server: Install grub on all disks in druid reuse recipesBtullisoperations/puppetproduction+14 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

bking created this task.Fri, Oct 2, 8:02 PM
Restricted Application added a project: DC-Ops. · View Herald TranscriptFri, Oct 2, 8:02 PM
bking removed bking as the assignee of this task.Fri, Oct 2, 8:02 PM
bking triaged this task as High priority.
Maintenance_bot added a project: SRE.Fri, Oct 2, 8:29 PM
Stashbot added a comment.Fri, Oct 2, 8:54 PM

Mentioned in SAL (#wikimedia-operations) [2026-10-02T20:54:34Z] <inflatador> bking@an-druid100[4-7] tune2fs -m0 /dev/mapper/vg0-srv T440033

dr0ptp4kt mentioned this in T387839: Update `netflow` retention strategy in Druid (too much data).Fri, Oct 2, 9:34 PM
bking assigned this task to Jclark-ctr.Fri, Oct 2, 9:54 PM

Assiging to @Jclark-ctr per conversation in #wikimedia-dcops IRC.

gerritbot added a comment.Mon, Oct 5, 9:06 AM

Change #1351177 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] install_server: Install grub on all disks in druid reuse recipes

https://gerrit.wikimedia.org/r/1351177

gerritbot added a project: Patch-For-Review.Mon, Oct 5, 9:06 AM
gerritbot added a comment.Mon, Oct 5, 9:24 AM

Change #1351177 merged by Btullis:

[operations/puppet@production] install_server: Install grub on all disks in druid reuse recipes

https://gerrit.wikimedia.org/r/1351177

Maintenance_bot removed a project: Patch-For-Review.Mon, Oct 5, 9:30 AM
BTullis claimed this task.Mon, Oct 5, 9:50 AM
BTullis added a subscriber: Jclark-ctr.
BTullis moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2026-09-18 - 2026-10-09) board.Mon, Oct 5, 9:53 AM

I think that this is a preseeding issue, so something that we should be able to sort out.
My reckoning at the moment it that it is because grub is not installed to all of the hard drives. I'm trying out a patch now to see if I can fix it.

BTullis closed this task as Resolved.Mon, Oct 5, 11:27 AM
BTullis moved this task from In Progress to Done on the Data-Platform-SRE (2026-09-18 - 2026-10-09) board.

Yes, this is fixed now. an-druid1003 is reimaged and back in the cluster.
I'll reimage all of the other druid servers in the cluster, to make sure that they also have grub installed to all of the SSDs in the array.

gerritbot added a comment.Mon, Oct 5, 1:43 PM

Change #1351321 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] install_server: Rebuild an-druid100[67] RAID10 across all 8 disks

https://gerrit.wikimedia.org/r/1351321

gerritbot added a project: Patch-For-Review.Mon, Oct 5, 1:43 PM

Change #1351322 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] install_server: Reuse the 8 disk recipe for an-druid100[67]

https://gerrit.wikimedia.org/r/1351322

gerritbot added a comment.Mon, Oct 5, 1:48 PM

Change #1351321 merged by Btullis:

[operations/puppet@production] install_server: Rebuild an-druid100[67] RAID10 across all 8 disks

https://gerrit.wikimedia.org/r/1351321

gerritbot added a comment.Tue, Oct 6, 7:52 PM

Change #1351322 merged by Btullis:

[operations/puppet@production] install_server: Reuse the 8 disk recipe for an-druid100[67]

https://gerrit.wikimedia.org/r/1351322

Maintenance_bot removed a project: Patch-For-Review.Tue, Oct 6, 8:30 PM
Gehel moved this task from Done to Reported on the Data-Platform-SRE (2026-09-18 - 2026-10-09) board.Fri, Oct 9, 7:13 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits