Page MenuHomePhabricator

https://phab-01.wmflabs.org returns a core exception
Closed, ResolvedPublic

Description

When I try to access https://phab-01.wmflabs.org, it returns a 502 error. I tried a hard refresh, but no success. I was using Google Chrome (49.0.2623.112 (Official Build) m (32-bit)) in a Microsoft Windows XP 5.1.2600 (Service Pack 3) computer.

phab-01.wmflabs.org 502 error.PNG (800×1 px, 65 KB)

Event Timeline

Poyekhali raised the priority of this task from Medium to High.Jun 8 2016, 5:06 AM

Without phab-01.wmflabs.org, we cannot test Phabricator.

phab-03.wmflabs.org should work

Yep, phab-03 is working now. phab-01, phab-02, and phab-04 are still not working.

Those broken phab-* hosts won't even let me log in as root, only phab-03 does.

Last time the disc was full. Maybe happend again?

Difficult to tell when you don't have salt access.

I can't log into phab-01, -02 nor -04, like I could in T125666, anymore.

@mmodell It looks like from that error that apache has been stopped.

I can't log in to any of the phab hosts so I don't know what to do to help.

I just rebooted phab-01 and can log in to it. Try doing the same for the rest?

I rebooted -02 as well, but that doesn't let me in - not user key nor root key. Same for -04. I pronounce those two as dead, and -01 and -03 as alive.

I actually looked a bit deeper into phab-02, it turns out the problems were manyfold:

  1. It had a puppetmaster set to 'deploy.phabricator.eqiad.wmflabs' which is a host that does not exist
  2. It now fails puppet due to something scap related

I think if you remove the roles it has from it, it should work?

(Salt was able to get access to it)

@mmodell would you be able to upgrade phabricator on phab-01 and phab-02 and phab-03 and phab-04 please.

I fixed up phab-03 and upgraded it. I'll work on phab-01 next

Is phab-04 repairable via salt/changing roles in the same way phab-02 was?

phab-02 to phab-04 are working, but not phab-01. The error message was:

[Core Exception/PhutilBootloaderException] Include of extension file '/srv/deployment/phabricator/deployment/phabricator/src/extensions/LDAPUserpageCustomField.php' failed!

@mmodell I think the phabricator wikimedia extension needs updating in phab-01 please.

Poyekhali renamed this task from https://phab-01.wmflabs.org (and other installs, like phab-02) returns a 502 error to https://phab-01.wmflabs.org returns a core exception.Jun 19 2016, 1:36 AM
mmodell claimed this task.

Re opening this.

I tried ssh into it but failed I looked at the website and it showed http 500 error so I rebooted it but now it failed to reboot.

This instance needs upgrading to a medium please.

13:24 < paladox> mutante https://phab-01.wmflabs.org/
13:24 < paladox> I fixed phab-01 and recreated it on a large image

Reassining to me. Since I'm currently fixing the instance.

Resolving this now.

I had help with setting some things by @Luke081515

Please reopen if the problems continue.

You will notice a performance increase plus it will be more stable since we chose a large instance since daemons can cause it to stop working on a small instance.

Uptime will also improve now.

Mentioned in SAL [2016-06-29T21:16:01Z] <paladox> phab-01 all setup now closing T137270 as resolved, uptime has improved now too :).

@Krenair ive deleted phab-02 and recreated as phab-05. The url is still the same but just running under a different instance.

Reason is because phab-02 is using port 222 and should be using port 22. Recreating didn't help either just kept setting it at port 222. So I will wait a see if I can recreate phab-02 or not.

@Paladox Phab-02 doesn't have the same puppet class assigned as all the other instances. It is using role::phabricator::labs::diffusion instead of role::phabricator::labs to test potential Diffusion changes that would include hosting a git server on port 22. So, yes, SSH is supposed to be hosted on port 222. Whether it is still topical is another issue.

You shouldn't be removing or modifying machines you don't fully understand the purpose or setup of.