Page MenuHomePhabricator

HTTP Error 502 / 503 when logging into https://paws.wmflabs.org
Closed, ResolvedPublic

Description

I cannot log into https://paws.wmflabs.org anymore.
Initially I got HTTP error 502 Bad Gateway Error.
After reading related issues I deleted all browser cookies.
Now I get HTTP error 503 and still cannot login.

Background:
I was executing a Pywikibot script to extract data from Wikidata.
Then I opened a second Python3 Jupyter Notebook to start another script in parallel.
Maybe I was blocked out because of this?

Event Timeline

Restricted Application added a subscriber: pywikibot-bugs-list. · View Herald TranscriptFeb 25 2019, 11:37 AM
Chicocvenancio triaged this task as Normal priority.Feb 25 2019, 11:42 AM
Chicocvenancio added a subscriber: Chicocvenancio.

While I can get my own server started, trying to access the server of another user is giving me a 503.

Dvorapa added a subscriber: Dvorapa.EditedFeb 25 2019, 11:43 AM

I'm not sure, but all similar Wikimedia tools are currently not working properly and throwing errors (perhaps some server issue), please try again later (perhaps tomorrow?)

Mentioned in SAL (#wikimedia-cloud) [2019-02-25T12:21:07Z] <chicocvenancio> PAWS: killed proxy and hub pods to attempt to get it to see routes to open notebooks servers to no avail. Restarted BernhardHumm's notebook pod T217010

@BernhardHumm PAWS should be accessible to you again.

BernhardHumm closed this task as Resolved.Feb 25 2019, 12:27 PM
BernhardHumm claimed this task.

Thank you very much for your quick help!

P.S. was it my fault when starting two Pywikibot script in two different Python3 Jupyter Notebooks?
Is there another way of running long scripts in the background?

Dvorapa added a comment.EditedFeb 25 2019, 12:38 PM

Thank you very much for your quick help!
P.S. was it my fault when starting two Pywikibot script in two different Python3 Jupyter Notebooks?
Is there another way of running long scripts in the background?

I don't think it is your fault. You should be able to run multiple Pywikibot scripts at once on one Notebook or multiple Notebooks, as you wish. Wikimedia tools servers have got multiple problems today. And these days the whole Wikimedia tools server background is updated and multiple thing are getting broken, so I would wait with long running scripts until april.

Anyway the another way (also currently getting some issues from that update) is to create a tool for your bot on Toolforge: https://tools.wmflabs.org/

Mentioned in SAL (#wikimedia-cloud) [2019-02-25T12:54:25Z] <chicocvenancio> PAWS: Restarted Criscod notebook pod T217010

Chicocvenancio reopened this task as Open.Feb 25 2019, 12:59 PM
Chicocvenancio claimed this task.

@BernhardHumm certainly not your fault. It seems this was an NFS outage that affected PAWS and it restarted all user servers but failed to update the routes to the new servers. Our jupyterhub deployment has some peculiarities that probably make it unique, but I was surprised that restarting the proxy pod and the hub pod was insufficient to make it update the routes. At any rate, restarting the user servers was sufficient. This is something any admin can do for all users, and is also self-serviceable by going to https://paws.wmflabs.org/paws/hub/home and clicking Stop My Server. In about one hour all idle servers are killed, so waiting would also have done the trick.

Chicocvenancio lowered the priority of this task from Normal to Low.Feb 25 2019, 1:00 PM

Keeping it open for a while but lowering priority since we have an easy user-serviceable workaround.

Mentioned in SAL (#wikimedia-cloud) [2019-02-25T13:11:50Z] <chicocvenancio> PAWS: Stopped AABot notebook pod T217010

Chicocvenancio closed this task as Resolved.Feb 25 2019, 1:12 PM

With that all affected user can access PAWS.

Again, thank you very much!