2020/05/21 18:18:18 [error] 7#7: *325234 lua entry thread aborted: runtime error: rewrite_by_lua(nginx.conf:140):15: Expected value but found invalid token at character 1 stack traceback: coroutine 0: [C]: in function 'decode' rewrite_by_lua(nginx.conf:140):15: in function <rewrite_by_lua(nginx.conf:140):1>, client: XXX.XXX.XXX.XXX server: , request: "GET /paws-public/User:Michael_Sch%C3%B6nitzer_(WMDE)/ HTTP/1.1", host: "paws-public.wmflabs.org" XXX.XXX.XXX.XXX - - [21/May/2020:18:18:18 +0000] "GET /paws-public/User:Michael_Sch%C3%B6nitzer_(WMDE)/ HTTP/1.1" 500 595 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36"
I've hacked a fix for this into the live pod, but a rebuild of the image is necessary for persistence.
PAWS-public links with non-ascii characters work now. (@MichaelSchoenitzer_WMDE )
I don't expect the pod to restart before a rebuild of the image is done, but to better document the issue:
The problem is the username is being sent without being url-encoded to the meta wiki api, that triggers a 400 error that does not come in JSON format and that breaks the error handling in the nginx lua script.
By adding url-escaping we can get a json response and all is well.
To live hack the pod we must first log in to toolforge and become paws-public.
Then exec into the pod:`
kubectl exec -it deploy/nbserve bash
Add the url-escaping with sed:
sed -i 's/\.\. m\[1\];/.. ngx.escape_uri\(m\[1\]\);/' /tmp/nginx.conf
Reload nginx configuration:
kill -s HUP 6 (perhaps check nginx still has PID 6 with ps aux).
Bad news. The image cannot build.
E: The value 'jessie-backports' is invalid for APT::Default-Release as such a release is not available in the sources The command '/bin/sh -c apt-get install --yes --no-install-recommends -t jessie-backports nginx-extras luarocks unzip build-essential' returned a non-zero code: 100
I'll need to fiddle with it. There may be a workaround. It needs nginx-extras and luarocks, though. I can also use my new image with this fix applied, but it is a very different image, so I'd prefer to test it first.
Ok so they work again for now, and I don't plan on restarting the pod for a while. I may try building my newer version of the image (not Debian Jessie-based) and testing it with a tag that will allow quick rollback if it doesn't work.
That way depooling a Kubernetes node doesn't roll this back again.