Page MenuHomePhabricator
Create Task
Maniphest T132387

wdqs1002 does not reboot, stops at "Scanning for devices"
Closed, ResolvedPublic
Actions

Description

Just as the title says... wdqs1002.eqiad.wmnet does not come up after a routine reboot.

@Joe and @Volans did the investigation. What we know sofar:

  • server was booting in pxe, not on HDD. Luckily pxe installation did not complete due to missing swap partition.
  • LV were checked and look good (no partition erased during failed install, data is still there) (check was done by dropping to shell during the pxe installation)
  • RAM, disk and proc look good when checking in BIOS
  • post serial redirection was on the whole time, nothing strange is logged
  • there was something like no supported device found at reboot but went away to quikly

Details

SubjectRepoBranchLines +/-
Revert "remove wdqs1002 from varnish during reinstall / fix"operations/puppetproduction+1 -0
remove wdqs1002 from varnish during reinstall / fixoperations/puppetproduction+0 -1
Customize query in gerrit

Related Objects

Event Timeline

Gehel created this task.Apr 11 2016, 7:42 PM
Restricted Application added projects: SRE, Wikidata. · View Herald TranscriptApr 11 2016, 7:42 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Smalyshev moved this task from Incoming to Watching / Waiting on the Wikidata-Query-Service board.Apr 11 2016, 7:44 PM
Smalyshev subscribed.
Gehel updated the task description. (Show Details)Apr 11 2016, 7:44 PM
RobH subscribed.EditedApr 11 2016, 9:12 PM

I've disabled the lifecycle controller completely, so its not the issue. After more testing, I think the initial load into the installer here: "server was booting in pxe, not on HDD. Luckily pxe installation did not complete due to missing swap partition." When it hit that screen, I think it wiped the MBR, so while data is in place, there is no master boot record.

The system will likely require reinstall.

When it asks about no swap, you simply confirm you do not want a swap and the installer will continue.

Smalyshev added a comment.Apr 11 2016, 9:51 PM

If it's getting reinstalled, can it be also linked to T120714 - i.e. extending diskspace and making correct LVM etc.?

RobH added a comment.Apr 11 2016, 9:51 PM

Please note: T120714

This should have an entirely new partman recipe designated to accomodate the new disks installed into this system previously.

Joe added a comment.Apr 12 2016, 6:34 AM

So, a few points:

  • If the MBR was wiped, it was not wiped by the installation process (the MBR stays untouched until the very end of the install process, thanks god)
  • If the MBR was wiped, we can simply boot from pxe, get to the installer stops, get a shell and reinstall the bootoloader in the mbr

Since these machines hold tons of data I guess it would be *really* preferrable not to wipe them out if possible.

MoritzMuehlenhoff subscribed.Apr 12 2016, 6:48 AM

It's really strange that it initiated a PXE install, both wdqs* servers were rebooted w/o problems for the keyctl security bug (and no additional BIOS config were needed).

Joe added a comment.Apr 12 2016, 6:51 AM

Why were we rebooting this system to start with?

Smalyshev added a comment.Apr 12 2016, 6:55 AM

AFAIK something to do with kernel upgrade, @Gehel should know the details.

MoritzMuehlenhoff added a comment.Apr 12 2016, 6:57 AM

The reboot was for the upgrade to Linux 4.4

Volans added a comment.Apr 12 2016, 7:05 AM

@Joe FYI: I was able yesterday to get a shell in the installer and mount /dev/sda1 and there was a boot partition with 4 kernels there, so the data was not wiped out IMHO.
I was also able to see the other 2 partitions and lvdisplay was showing 3 logical volumes.

Joe added a comment.Apr 12 2016, 9:00 AM

so, it could be possible that the package upgrade failed to correctly install grub to the mbr?

Gehel claimed this task.Apr 12 2016, 1:26 PM
Gehel removed a project: ops-eqiad.
gerritbot subscribed.Apr 12 2016, 2:44 PM

Change 282946 had a related patch set uploaded (by Gehel):
remove wdqs1002 from varnish during reinstall / fix

https://gerrit.wikimedia.org/r/282946

gerritbot added a project: Patch-For-Review.Apr 12 2016, 2:44 PM
gerritbot added a comment.Apr 12 2016, 3:45 PM

Change 282946 merged by Gehel:
remove wdqs1002 from varnish during reinstall / fix

https://gerrit.wikimedia.org/r/282946

Gehel added a comment.Apr 12 2016, 5:06 PM

Full reinstall is in progress as we needed to account for new disks anyway.

JanZerebecki subscribed.Apr 12 2016, 7:59 PM
gerritbot added a comment.Apr 14 2016, 5:31 PM

Change 283485 had a related patch set uploaded (by Gehel):
Revert "remove wdqs1002 from varnish during reinstall / fix"

https://gerrit.wikimedia.org/r/283485

gerritbot added a comment.Apr 15 2016, 9:00 AM

Change 283485 merged by Gehel:
Revert "remove wdqs1002 from varnish during reinstall / fix"

https://gerrit.wikimedia.org/r/283485

Stashbot subscribed.Apr 15 2016, 9:01 AM

Mentioned in SAL [2016-04-15T09:01:04Z] <gehel> reenabling wdqs1002 in varnish rotation after reinstall (T132387)

Gehel added a project: Discovery-Wikidata-Query-Service-Sprint.Apr 15 2016, 4:16 PM
Gehel moved this task from Backlog to Done on the Discovery-Wikidata-Query-Service-Sprint board.
Smalyshev closed this task as Resolved.EditedApr 15 2016, 5:46 PM

I think this is done? Thanks @Gehel for all the work.

Gehel mentioned this in T132952: Failed to deploy WDQS.Apr 18 2016, 6:35 PM
Smalyshev removed a project: Discovery-Wikidata-Query-Service-Sprint.Jul 14 2017, 10:31 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL