Page MenuHomePhabricator

maps-wma1.maps.eqiad.wmflabs is down (wma.wmflabs.org)
Closed, ResolvedPublic

Description

wma.wmflabs.org is failing with 502 bad gateway

Suspect to be the cause of meta:WikiMiniAtlas failures

User reports at https://en.wikipedia.org/w/index.php?title=Wikipedia:Village_pump_(technical)&oldid=822573988#Globe_link_to_the_left_of_title_coordinates_gives_502_Bad_Gateway_error

Related Objects

Event Timeline

Xaosflux created this task.Jan 27 2018, 4:59 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 27 2018, 4:59 AM
Xaosflux triaged this task as High priority.Jan 27 2018, 5:00 AM
Andrew added a subscriber: Andrew.Jan 27 2018, 7:30 AM

That url is backed by maps-wma1.maps.eqiad.wmflabs. That VM is completely broken -- unreachable, no logs even after a reboot. I don't yet know what's wrong with it.

Xaosflux renamed this task from wma.wmflabs.org is failing with 502 bad gateway to maps-wma1.maps.eqiad.wmflabs is down (wma.wmflabs.org).Jan 27 2018, 3:21 PM

This instance appears to have suffered a mishap during the meltdown fixes -- it appears that it had broken packages that prevented a proper kernel install. That means it has probably been down for a few weeks.

Anyway, here's how I fixed it:

  • mounted the instances drive, following the instructions here:

https://wikitech.wikimedia.org/w/index.php?title=Portal:Cloud_VPS/Admin/Troubleshooting&oldid=1764875

(Note the old version, as this old VM can not be handled via the up-to-date instructions).

  • reverted to an older boot/grub/menu.lst which pre-dated the Meltdown fix, unmonted the drive, and booted the VM. This allowed me to log in.
  • re-applied the meltdown fix kernel and rebooted yet again. (apt-get install -y linux-image-generic)
  • forced a fresh grub build (moved /boot/grub/menu.lst to /boot/grub/menu.lst.works, and update-grub -y)
  • rebooted
  • (and then all of the above a second time because apt issues prevented the kernel from properly installing the first time)
  • purge-old-kernels
  • rebooted again
Andrew closed this task as Resolved.Jan 27 2018, 4:54 PM
Andrew claimed this task.

This VM is back up. Nevertheless, I suggest moving the on-wiki services to https://www.mediawiki.org/wiki/Help:Extension:Kartographer if possible.