Page MenuHomePhabricator

maps-wma1.maps.eqiad.wmflabs is down (
Closed, ResolvedPublic

Description is failing with 502 bad gateway

Suspect to be the cause of meta:WikiMiniAtlas failures

User reports at

Related Objects

Event Timeline

Xaosflux created this task.Jan 27 2018, 4:59 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 27 2018, 4:59 AM
Xaosflux triaged this task as High priority.Jan 27 2018, 5:00 AM
Andrew added a subscriber: Andrew.Jan 27 2018, 7:30 AM

That url is backed by maps-wma1.maps.eqiad.wmflabs. That VM is completely broken -- unreachable, no logs even after a reboot. I don't yet know what's wrong with it.

Xaosflux renamed this task from is failing with 502 bad gateway to maps-wma1.maps.eqiad.wmflabs is down ( 27 2018, 3:21 PM

This instance appears to have suffered a mishap during the meltdown fixes -- it appears that it had broken packages that prevented a proper kernel install. That means it has probably been down for a few weeks.

Anyway, here's how I fixed it:

  • mounted the instances drive, following the instructions here:

(Note the old version, as this old VM can not be handled via the up-to-date instructions).

  • reverted to an older boot/grub/menu.lst which pre-dated the Meltdown fix, unmonted the drive, and booted the VM. This allowed me to log in.
  • re-applied the meltdown fix kernel and rebooted yet again. (apt-get install -y linux-image-generic)
  • forced a fresh grub build (moved /boot/grub/menu.lst to /boot/grub/, and update-grub -y)
  • rebooted
  • (and then all of the above a second time because apt issues prevented the kernel from properly installing the first time)
  • purge-old-kernels
  • rebooted again
Andrew closed this task as Resolved.Jan 27 2018, 4:54 PM
Andrew claimed this task.

This VM is back up. Nevertheless, I suggest moving the on-wiki services to if possible.