The root cause of the user facing issues described in T162035 is a bug in swift-object-server 1.13.1. The bug is not reproducible on 2.2.0.
Steps to reproduce
The following conditional request results in the wrong Content-Type being returned by swift-object-server on ms-be1016 (running swift 1.13.1):
curl -v -H 'If-Modified-Since: Thu, 24 Oct 2013 15:17:41 GMT' http://localhost:6000/sdd1/16240/AUTH_mw/wikipedia-commons-local-thumb.22/2/22/Tarin_des_aulnes_m%C3%A2le2.JPG/1200px-Tarin_des_aulnes_m%C3%A2le2.JPG > /dev/null [...] < HTTP/1.1 304 Not Modified < Last-Modified: Thu, 24 Oct 2013 15:17:41 GMT < Content-Type: text/html; charset=UTF-8
Setting IMS to a date earlier than LM, instead, the right CT is returned:
curl -v -H 'If-Modified-Since: Thu, 24 Oct 2012 15:17:41 GMT' http://localhost:6000/sdd1/16240/AUTH_mw/wikipedia-commons-local-thumb.22/2/22/Tarin_des_aulnes_m%C3%A2le2.JPG/1200px-Tarin_des_aulnes_m%C3%A2le2.JPG > /dev/null [...] < HTTP/1.1 200 OK < Last-Modified: Thu, 24 Oct 2013 15:17:41 GMT < Content-Type: image/jpeg
The same behavior cannot be reproduced on ms-be1027 (running swift 2.2.0):
curl -v -H 'If-Modified-Since: Sat, 09 Nov 2013 11:25:18 GMT' http://localhost:6000/sdd1/29423/AUTH_mw/wikipedia-commons-local-thumb.5f/5/5f/Stereogram_%28three-dimensional_population_pyramid%29_modeled_on_actual_data_%28Swedish_census%2C_1750-1875%29.jpg/220px-Stereogram_%28three-dimensional_population_pyramid%29_modeled_on_actual_data_%28Swedish_census%2C_1750-1875%29.jpg > /dev/null [...] < HTTP/1.1 304 Not Modified < Last-Modified: Sat, 09 Nov 2013 10:25:18 GMT < Content-Type: image/jpeg
200 responses are also fine on 2.2.0:
curl -v -H 'If-Modified-Since: Sat, 09 Nov 2012 11:25:18 GMT' http://localhost:6000/sdd1/29423/AUTH_mw/wikipedia-commons-local-thumb.5f/5/5f/Stereogram_%28three-dimensional_population_pyramid%29_modeled_on_actual_data_%28Swedish_census%2C_1750-1875%29.jpg/220px-Stereogram_%28three-dimensional_population_pyramid%29_modeled_on_actual_data_%28Swedish_census%2C_1750-1875%29.jpg > /dev/null [...] < HTTP/1.1 200 OK < Last-Modified: Sat, 09 Nov 2013 10:25:18 GMT < Content-Type: image/jpeg
Proposed solutions
Any of the following actions would solve the problem to the best of my knowledge;
- Upgrade all hosts running swift-object-server to 2.2.0 (result: right CT is returned on 304 responses)
- Patch swift-object-server 1.13.1 and make it drop CT on 304 responses
- Patch swift-proxy 2.2.0 and make it drop CT on 304 responses
According to RFC 7232 section 4.1, 304 Not Modified responses shouldn't include representation metadata other than Cache-Control, Content-Location, Date, ETag, Expires, Vary and possibly Last-Modified. IMHO, thus, swift should stop returning CT on 304s.
My proposed patch is https://phabricator.wikimedia.org/P5212, which I've deployed on swift-proxy in beta (deployment-ms-fe01, which confusingly runs 1.13.1) and works fine there.
The patch applies cleanly on a fresh apt-get source swift=2.2.0-1+deb8u1 and all tests pass.