Page MenuHomePhabricator

swift wmf/rewrite.py middleware broken on bullseye (and its test suite doesn't work either)
Closed, ResolvedPublic

Description

ms-fe1012 has been upgraded to bullseye, and the wmf/rewrite middleware doesn't work. 2 examples:

#working host ms-fe1011 (stretch)
mvernon@ms-fe1011:~$ curl -o /dev/null -v -H "Host: upload.wikimedia.org" http://localhost/wikipedia/commons/thumb/2/2d/%E5%8F%B0%E7%81%A3%E7%BE%A4%E5%B1%B1%E6%97%97.svg/120px-%E5%8F%B0%E7%81%A3%E7%BE%A4%E5%B1%B1%E6%97%97.svg.png 2>&1 | grep '< HTTP'
< HTTP/1.1 200 OK
mvernon@ms-fe1011:~$ curl -o /tmp/foo -v -H "Host: upload.wikimedia.org" http://localhost//wikipedia/commons/thumb/2/23/Drapeau_fr_d%C3%A9partement_Morbihan.svg/188px-Drapeau_fr_d%C3%A9partement_Morbihan.svg.png 2>&1 | grep '< HTTP'
< HTTP/1.1 200 OK
#
# Broken host ms-fe1012 (bullseye)
mvernon@ms-fe1012:~$ curl -o /dev/null -v -H "Host: upload.wikimedia.org" http://localhost/wikipedia/commons/thumb/2/2d/%E5%8F%B0%E7%81%A3%E7%BE%A4%E5%B1%B1%E6%97%97.svg/120px-%E5%8F%B0%E7%81%A3%E7%BE%A4%E5%B1%B1%E6%97%97.svg.png 2>&1 | grep '< HTTP'
< HTTP/1.1 500 Internal Error
mvernon@ms-fe1012:~$ curl -o /tmp/foo -v -H "Host: upload.wikimedia.org" http://localhost//wikipedia/commons/thumb/2/23/Drapeau_fr_d%C3%A9partement_Morbihan.svg/188px-Drapeau_fr_d%C3%A9partement_Morbihan.svg.png 2>&1 | grep '< HTTP'
< HTTP/1.1 400 Bad Request

Further, the test suite for the rewrite middleware fails in two ways - it doesn't detect this problem, and has two now-spurious failures that predate changing purging policies. Example from stretch host (behaviour on bullseye is the same):

mvernon@ms-fe1011:~$ python /usr/local/lib/python2.7/dist-packages/wmf/rewrite_integration_test.py 
...FF..
======================================================================
FAIL: test_purge_recreate_thumb (__main__.WMFRewriteTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/wmf/rewrite_integration_test.py", line 50, in test_purge_recreate_thumb
    self.assertEquals(r.status_code, 302)
AssertionError: 403 != 302

======================================================================
FAIL: test_purge_thumb (__main__.WMFRewriteTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/wmf/rewrite_integration_test.py", line 44, in test_purge_thumb
    self.assertEquals(r.status_code, 302)
AssertionError: 403 != 302

----------------------------------------------------------------------
Ran 7 tests in 0.124s

FAILED (failures=2)

Event Timeline

Change 779900 had a related patch set uploaded (by MVernon; author: MVernon):

[operations/puppet@production] swift: correct handling of non-ASCII paths in rewrite.py & test suite

https://gerrit.wikimedia.org/r/779900

Change 779900 merged by MVernon:

[operations/puppet@production] swift: correct handling of non-ASCII paths in rewrite.py & test suite

https://gerrit.wikimedia.org/r/779900

New failures:

Apr 26 09:25:15 ms-fe1012 proxy-server: Error: An error occurred: #012Traceback (most recent call last):#012  File "/usr/lib/python3/dist-packages/swift/common/middleware/catch_errors.py", line 75, in handle_request#012    resp = self._app_call(env)#012  File "/usr/lib/python3/dist-packages/swift/common/wsgi.py", line 1327, in _app_call#012    resp = self.app(env, self._start_response)#012  File "/usr/lib/python3/dist-packages/swift/common/middleware/gatekeeper.py", line 129, in __call__#012    return self.app(env, gatekeeper_response)#012  File "/usr/lib/python3/dist-packages/swift/common/middleware/proxy_logging.py", line 427, in __call__#012    iterable = self.app(env, my_start_response)#012  File "/usr/lib/python3/dist-packages/swift/common/middleware/listing_formats.py", line 157, in __call__#012    return self.app(env, start_response)#012  File "/usr/local/lib/python3.9/dist-packages/wmf/ensure_max_age.py", line 28, in __call__#012    return self.app(env, _start_response)#012  File "/usr/local/lib/python3.9/dist-packages/wmf/rewrite.py", line 446, in __call__#012    return context.handle_request(env, start_response)#012  File "/usr/local/lib/python3.9/dist-packages/wmf/rewrite.py", line 210, in handle_request#012    return self._handle_request(env, start_response)#012  File "/usr/local/lib/python3.9/dist-packages/wmf/rewrite.py", line 407, in _handle_request#012    resp = self.handle404(reqorig, url, container, obj)#012  File "/usr/local/lib/python3.9/dist-packages/wmf/rewrite.py", line 117, in handle404#012    c_t = uinfo.gettype()#012AttributeError: 'HTTPMessage' object has no attribute 'gettype' (txn: tx608e8794d8e34aa1abaf9-006267ba62) (client_ip: 66.249.64.159)

Matches this request:

Apr 26 09:25:15 ms-fe1012 proxy-server: 66.249.64.159 10.64.130.2 26/Apr/2022/09/25/15 GET /v1/AUTH_mw/wikipedia-commons-local-thumb.55/5/55/All_Saints_Church%252C_Church_Lane%252C_East_Meon_%2528NHLE_Code_1351116%2529_%2528July_2019%2529_%2528Interior%2529_%25283%2529.JPG/512px-All_Saints_Church%252C_Church_Lane%252C_East_Meon_%2528NHLE_Code_1351116%2529_%2528July_2019%2529_%2528Interior%2529_%25283%2529.JPG HTTP/1.0 500 - Googlebot-Image/1.0 - - - - tx608e8794d8e34aa1abaf9-006267ba62 - 25.6604 - - 1650965090.304027319 1650965115.964461088 0

Change 786290 had a related patch set uploaded (by MVernon; author: MVernon):

[operations/puppet@production] swift: wmf/rewrite.py py2->3 HTTPMessage changes

https://gerrit.wikimedia.org/r/786290

That patch fixes the above error; I've found another...

Change 786290 merged by MVernon:

[operations/puppet@production] swift: wmf/rewrite.py py2->3 HTTPMessage changes

https://gerrit.wikimedia.org/r/786290

Change 786311 had a related patch set uploaded (by MVernon; author: MVernon):

[operations/puppet@production] swift: wmf/rewrite.py say 400 earlier if passed bad UTF-8

https://gerrit.wikimedia.org/r/786311

Change 786311 merged by MVernon:

[operations/puppet@production] swift: wmf/rewrite.py say 400 earlier if passed bad UTF-8

https://gerrit.wikimedia.org/r/786311

MatthewVernon claimed this task.

I think this is now all working satisfactorily (ms-fe1012 is now pooled in prod.)