Page MenuHomePhabricator

HTTP Error 502 / 503 when logging into https://paws.wmflabs.org
Closed, ResolvedPublic

Description

I cannot log into https://paws.wmflabs.org anymore.
Initially I got HTTP error 502 Bad Gateway Error.
After reading related issues I deleted all browser cookies.
Now I get HTTP error 503 and still cannot login.

Background:
I was executing a Pywikibot script to extract data from Wikidata.
Then I opened a second Python3 Jupyter Notebook to start another script in parallel.
Maybe I was blocked out because of this?

Event Timeline

Chicocvenancio added a subscriber: Chicocvenancio.

While I can get my own server started, trying to access the server of another user is giving me a 503.

I'm not sure, but all similar Wikimedia tools are currently not working properly and throwing errors (perhaps some server issue), please try again later (perhaps tomorrow?)

Mentioned in SAL (#wikimedia-cloud) [2019-02-25T12:21:07Z] <chicocvenancio> PAWS: killed proxy and hub pods to attempt to get it to see routes to open notebooks servers to no avail. Restarted BernhardHumm's notebook pod T217010

BernhardHumm claimed this task.

Thank you very much for your quick help!

P.S. was it my fault when starting two Pywikibot script in two different Python3 Jupyter Notebooks?
Is there another way of running long scripts in the background?

Thank you very much for your quick help!

P.S. was it my fault when starting two Pywikibot script in two different Python3 Jupyter Notebooks?
Is there another way of running long scripts in the background?

I don't think it is your fault. You should be able to run multiple Pywikibot scripts at once on one Notebook or multiple Notebooks, as you wish. Wikimedia tools servers have got multiple problems today. And these days the whole Wikimedia tools server background is updated and multiple thing are getting broken, so I would wait with long running scripts until april.

Anyway the another way (also currently getting some issues from that update) is to create a tool for your bot on Toolforge: https://tools.wmflabs.org/

Mentioned in SAL (#wikimedia-cloud) [2019-02-25T12:54:25Z] <chicocvenancio> PAWS: Restarted Criscod notebook pod T217010

Chicocvenancio claimed this task.

@BernhardHumm certainly not your fault. It seems this was an NFS outage that affected PAWS and it restarted all user servers but failed to update the routes to the new servers. Our jupyterhub deployment has some peculiarities that probably make it unique, but I was surprised that restarting the proxy pod and the hub pod was insufficient to make it update the routes. At any rate, restarting the user servers was sufficient. This is something any admin can do for all users, and is also self-serviceable by going to https://paws.wmflabs.org/paws/hub/home and clicking Stop My Server. In about one hour all idle servers are killed, so waiting would also have done the trick.

Chicocvenancio lowered the priority of this task from Medium to Low.Feb 25 2019, 1:00 PM

Keeping it open for a while but lowering priority since we have an easy user-serviceable workaround.

Mentioned in SAL (#wikimedia-cloud) [2019-02-25T13:11:50Z] <chicocvenancio> PAWS: Stopped AABot notebook pod T217010

With that all affected user can access PAWS.