Since some time the publik links to paws notebooks won't work.
Example: https://paws-public.wmflabs.org/paws-public/User:Michael_Sch%C3%B6nitzer_(WMDE)/TWL19-Statistics.ipynb
Description
Description
Event Timeline
Comment Actions
https://paws-public.wmflabs.org/paws-public/User:Michael_Sch%C3%B6nitzer_(WMDE)/ Triggers a 500 by itself.
Comment Actions
2020/05/21 18:18:18 [error] 7#7: *325234 lua entry thread aborted: runtime error: rewrite_by_lua(nginx.conf:140):15: Expected value but found invalid token at character 1
stack traceback:
coroutine 0:
[C]: in function 'decode'
rewrite_by_lua(nginx.conf:140):15: in function <rewrite_by_lua(nginx.conf:140):1>, client: XXX.XXX.XXX.XXX server: , request: "GET /paws-public/User:Michael_Sch%C3%B6nitzer_(WMDE)/ HTTP/1.1", host: "paws-public.wmflabs.org"
XXX.XXX.XXX.XXX - - [21/May/2020:18:18:18 +0000] "GET /paws-public/User:Michael_Sch%C3%B6nitzer_(WMDE)/ HTTP/1.1" 500 595 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36"Comment Actions
@MichaelSchoenitzer_WMDE In the meantime you can go to https://paws-public.wmflabs.org/paws-public/47694471/ to see your public notebooks.
Comment Actions
I've hacked a fix for this into the live pod, but a rebuild of the image is necessary for persistence.
PAWS-public links with non-ascii characters work now. (@MichaelSchoenitzer_WMDE )
I don't expect the pod to restart before a rebuild of the image is done, but to better document the issue:
The problem is the username is being sent without being url-encoded to the meta wiki api, that triggers a 400 error that does not come in JSON format and that breaks the error handling in the nginx lua script.
By adding url-escaping we can get a json response and all is well.
To live hack the pod we must first log in to toolforge and become paws-public.
Then exec into the pod:`
kubectl exec -it deploy/nbserve bash
Add the url-escaping with sed:
sed -i 's/\.\. m\[1\];/.. ngx.escape_uri\(m\[1\]\);/' /tmp/nginx.conf
Reload nginx configuration:
kill -s HUP 6 (perhaps check nginx still has PID 6 with ps aux).
Comment Actions
Bad news. The image cannot build.
E: The value 'jessie-backports' is invalid for APT::Default-Release as such a release is not available in the sources The command '/bin/sh -c apt-get install --yes --no-install-recommends -t jessie-backports nginx-extras luarocks unzip build-essential' returned a non-zero code: 100
I'll need to fiddle with it. There may be a workaround. It needs nginx-extras and luarocks, though. I can also use my new image with this fix applied, but it is a very different image, so I'd prefer to test it first.
Comment Actions
The backports repo is gone. It needs to change to the debian archive. I'll get that fixed shortly.
Comment Actions
Mentioned in SAL (#wikimedia-cloud) [2020-05-29T18:12:42Z] <bstorm_> applied in-place fix for non-ASCII usernames and applied this to my own version of the image T252217
Comment Actions
Ok so they work again for now, and I don't plan on restarting the pod for a while. I may try building my newer version of the image (not Debian Jessie-based) and testing it with a tag that will allow quick rollback if it doesn't work.
That way depooling a Kubernetes node doesn't roll this back again.
Comment Actions
Mentioned in SAL (#wikimedia-cloud) [2020-05-29T19:37:13Z] <bstorm_> adding docker image for paws-public docker-registry.tools.wmflabs.org/paws-public-nginx:openresty T252217
Comment Actions
Mentioned in SAL (#wikimedia-cloud) [2020-05-29T19:39:00Z] <bstorm_> switch deployment to the openresty version to try it out T252217
Comment Actions
If we end up not merging all of my coming pull request into the paws repo, I'll make a separate PR with just the new image.