Page MenuHomePhabricator

Public-Link of Notebooks of users with non-ASCII characters return 500 error
Closed, ResolvedPublic

Description

Since some time the publik links to paws notebooks won't work.
Example: https://paws-public.wmflabs.org/paws-public/User:Michael_Sch%C3%B6nitzer_(WMDE)/TWL19-Statistics.ipynb

Event Timeline

Restricted Application added a subscriber: Sadads. · View Herald TranscriptMay 8 2020, 3:49 PM
2020/05/21 18:18:18 [error] 7#7: *325234 lua entry thread aborted: runtime error: rewrite_by_lua(nginx.conf:140):15: Expected value but found invalid token at character 1
stack traceback:
coroutine 0:
        [C]: in function 'decode'
        rewrite_by_lua(nginx.conf:140):15: in function <rewrite_by_lua(nginx.conf:140):1>, client: XXX.XXX.XXX.XXX server: , request: "GET /paws-public/User:Michael_Sch%C3%B6nitzer_(WMDE)/ HTTP/1.1", host: "paws-public.wmflabs.org"
XXX.XXX.XXX.XXX - - [21/May/2020:18:18:18 +0000] "GET /paws-public/User:Michael_Sch%C3%B6nitzer_(WMDE)/ HTTP/1.1" 500 595 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36"

PAWS-public seems to be erroring out with all usernames that go outside ascii.

Chicocvenancio renamed this task from Public-Link of Notebooks leads to Error 500 to Public-Link of Notebooks of users with non-ASCII characters return 500 error.May 21 2020, 6:53 PM
Chicocvenancio triaged this task as Medium priority.
Chicocvenancio lowered the priority of this task from Medium to Lowest.EditedMay 28 2020, 2:05 AM

I've hacked a fix for this into the live pod, but a rebuild of the image is necessary for persistence.

PAWS-public links with non-ascii characters work now. (@MichaelSchoenitzer_WMDE )

I don't expect the pod to restart before a rebuild of the image is done, but to better document the issue:

The problem is the username is being sent without being url-encoded to the meta wiki api, that triggers a 400 error that does not come in JSON format and that breaks the error handling in the nginx lua script.

By adding url-escaping we can get a json response and all is well.

To live hack the pod we must first log in to toolforge and become paws-public.
Then exec into the pod:`
kubectl exec -it deploy/nbserve bash
Add the url-escaping with sed:
sed -i 's/\.\. m\[1\];/.. ngx.escape_uri\(m\[1\]\);/' /tmp/nginx.conf
Reload nginx configuration:
kill -s HUP 6 (perhaps check nginx still has PID 6 with ps aux).

Bstorm added a subscriber: Bstorm.May 28 2020, 2:51 PM

Turns out that we are planning on restarting that pod today. I'll rebuild the image.

Bad news. The image cannot build.

E: The value 'jessie-backports' is invalid for APT::Default-Release as such a release is not available in the sources
The command '/bin/sh -c apt-get install --yes --no-install-recommends -t jessie-backports             nginx-extras             luarocks             unzip             build-essential' returned a non-zero code: 100

I'll need to fiddle with it. There may be a workaround. It needs nginx-extras and luarocks, though. I can also use my new image with this fix applied, but it is a very different image, so I'd prefer to test it first.

The backports repo is gone. It needs to change to the debian archive. I'll get that fixed shortly.

Mentioned in SAL (#wikimedia-cloud) [2020-05-29T18:12:42Z] <bstorm_> applied in-place fix for non-ASCII usernames and applied this to my own version of the image T252217

Ok so they work again for now, and I don't plan on restarting the pod for a while. I may try building my newer version of the image (not Debian Jessie-based) and testing it with a tag that will allow quick rollback if it doesn't work.

That way depooling a Kubernetes node doesn't roll this back again.

Mentioned in SAL (#wikimedia-cloud) [2020-05-29T19:37:13Z] <bstorm_> adding docker image for paws-public docker-registry.tools.wmflabs.org/paws-public-nginx:openresty T252217

Mentioned in SAL (#wikimedia-cloud) [2020-05-29T19:39:00Z] <bstorm_> switch deployment to the openresty version to try it out T252217

Bstorm closed this task as Resolved.May 29 2020, 7:40 PM
Bstorm claimed this task.

Apparently my version works! That makes this stable and I'll close the ticket.

If we end up not merging all of my coming pull request into the paws repo, I'll make a separate PR with just the new image.