Purging for all sites but eqiad is broken
Closed, ResolvedPublic

Description

+++ This bug was initially created as a clone of Bug #44508 comment 13 +++

  • Comment 13 by Marco 2013-03-19 23:54:38 UTC ---

New instance:
https://upload.wikimedia.org/wikipedia/commons/thumb/a/a5/PlacaenmonumentoaMarconi.jpg/120px-PlacaenmonumentoaMarconi.jpg

"Broken" response from esams:

HTTP request sent, awaiting response...

HTTP/1.1 200 OK
Server: nginx/1.1.19
Date: Tue, 19 Mar 2013 23:35:35 GMT
Content-Type: image/jpeg
Content-Length: 3637
Connection: keep-alive
X-Object-Meta-Sha1base36: q311dfj910y0loxw51q2fm79rw8j32o
Content-Disposition: inline;filename*=UTF-8''PlacaenmonumentoaMarconi.jpg
X-Timestamp: 1363609261.89105
Accept-Ranges: bytes
Access-Control-Allow-Origin: *
Last-Modified: Mon, 18 Mar 2013 12:21:01 GMT
ETag: b737ec5a44b9f7975b69facfde655da0
X-Cache: HIT from sq82.wikimedia.org
X-Cache-Lookup: HIT from sq82.wikimedia.org:3128
Age: 41471
X-Cache: HIT from amssq59.esams.wikimedia.org
X-Cache-Lookup: HIT from amssq59.esams.wikimedia.org:3128
X-Cache: MISS from knsq19.esams.wikimedia.org
X-Cache-Lookup: MISS from knsq19.esams.wikimedia.org:80
Via: 1.0 sq82.wikimedia.org:3128 (squid/2.7.STABLE9), 1.0

amssq59.esams.wikimedia.org:3128 (squid/2.7.STABLE9), 1.0
knsq19.esams.wikimedia.org:80 (squid/2.7.STABLE9)
Length: 3637 (3.6K) [image/jpeg]

"Valid" response via Varnish:

HTTP request sent, awaiting response...

HTTP/1.1 200 OK
Server: nginx/1.1.19
Date: Tue, 19 Mar 2013 23:35:24 GMT
Content-Type: image/jpeg
Content-Length: 3994
Connection: keep-alive
X-Object-Meta-Sha1base36: qtiacup2waiwupgx7sma7puadwpqeig
Content-Disposition: inline;filename*=UTF-8''PlacaenmonumentoaMarconi.jpg
Last-Modified: Tue, 19 Mar 2013 23:24:09 GMT
Etag: 92f154cb562f20cf648199f1bd88c464
X-Timestamp: 1363735449.88825
X-Varnish: 2316312654, 2904564637, 670145209
Via: 1.1 varnish, 1.1 varnish, 1.1 varnish
Accept-Ranges: bytes
Age: 0
X-Cache: cp1028 miss (0), cp3009 miss (0), cp3009 frontend miss (0)
Access-Control-Allow-Origin: *

Length: 3994 (3.9K) [image/jpeg]

Please confirm. (I already purged the file page)


Version: wmf-deployment
Severity: critical

bzimport set Reference to bz46350.
Aklapper created this task.Via LegacyMar 20 2013, 9:15 AM
Aklapper added a comment.Via ConduitMar 20 2013, 9:16 AM

$:andre\> wget -S -U Malyacko --header 'host: upload.wikimedia.org' 'http://upload-lb.eqiad.wikimedia.org/wikipedia/commons/a/a5/PlacaenmonumentoaMarconi.jpg'
--2013-03-20 10:09:14-- http://upload-lb.eqiad.wikimedia.org/wikipedia/commons/a/a5/PlacaenmonumentoaMarconi.jpg
Resolving upload-lb.eqiad.wikimedia.org... 208.80.154.235, 2620:0:861:ed1a::b
Connecting to upload-lb.eqiad.wikimedia.org|208.80.154.235|:80... connected.
HTTP request sent, awaiting response...

HTTP/1.1 200 OK
X-Object-Meta-Sha1base36: quwkq14ur1ogdvm9amhhkky19and4wy
Last-Modified: Tue, 19 Mar 2013 14:55:49 GMT
Etag: 5627db7e9874a4cf2ff47acddd548c8a
X-Timestamp: 1363704949.43621
Content-Type: image/jpeg
X-Varnish: 684751357, 3232235220
Via: 1.1 varnish, 1.1 varnish
Content-Length: 2112147
Accept-Ranges: bytes
Date: Wed, 20 Mar 2013 09:09:16 GMT
Age: 1
Connection: keep-alive
X-Cache: cp1027 miss (0), cp1035 frontend miss (0)
Access-Control-Allow-Origin: *

Length: 2112147 (2.0M) [image/jpeg]

2013-03-20 10:09:21 (449 KB/s) - “PlacaenmonumentoaMarconi.jpg” saved [2112147/2112147]

$:andre\> wget -S -U Malyacko --header 'host: upload.wikimedia.org' 'http://upload-lb.esams.wikimedia.org/wikipedia/commons/a/a5/PlacaenmonumentoaMarconi.jpg'
--2013-03-20 10:09:27-- http://upload-lb.esams.wikimedia.org/wikipedia/commons/a/a5/PlacaenmonumentoaMarconi.jpg
Resolving upload-lb.esams.wikimedia.org... 91.198.174.234, 2620:0:862:ed1a::b
Connecting to upload-lb.esams.wikimedia.org|91.198.174.234|:80... connected.
HTTP request sent, awaiting response...

HTTP/1.0 200 OK
X-Object-Meta-Sha1base36: 8xysnw9wchzvrlb71wizuqw9c86snkp
Last-Modified: Mon, 18 Mar 2013 12:20:53 GMT
ETag: 56d70ec74402692887b1ca918fff4bd9
X-Timestamp: 1363609253.81912
Accept-Ranges: bytes
Content-Length: 1333860
Content-Type: image/jpeg
Access-Control-Allow-Origin: *
Date: Tue, 19 Mar 2013 11:22:20 GMT
Age: 78427
X-Cache: HIT from sq53.wikimedia.org
X-Cache-Lookup: HIT from sq53.wikimedia.org:3128
X-Cache: MISS from knsq21.esams.wikimedia.org
X-Cache-Lookup: MISS from knsq21.esams.wikimedia.org:3128
X-Cache: MISS from amssq49.esams.wikimedia.org
X-Cache-Lookup: MISS from amssq49.esams.wikimedia.org:80
Connection: keep-alive

Length: 1333860 (1.3M) [image/jpeg]

2013-03-20 10:09:30 (594 KB/s) - “PlacaenmonumentoaMarconi.jpg.1” saved [1333860/1333860]

$:andre\>

Nemo_bis added a comment.Via ConduitMar 20 2013, 3:01 PM

Same for https://upload.wikimedia.org/wikipedia/commons/0/04/Wikipedia-logo-v2-uz.png (just reuploaded):

Last-Modified: Thu, 12 Jul 2012 20:11:39 GMT
[...]
X-Cache-Lookup: HIT from sq56.wikimedia.org:3128
Age: 9332151

But varnish is equally wrong:

Last-Modified: Thu, 12 Jul 2012 20:11:39 GMT
Etag: dc298c819efca185c0200435e68428a6
X-Timestamp: 1342123899.97366
X-Varnish: 2769044483 1786526384, 2956910772 864530478, 824447606 824386074
Via: 1.1 varnish, 1.1 varnish, 1.1 varnish
Accept-Ranges: bytes
Age: 2665367
X-Cache: cp1024 hit (8978), cp3009 hit (3621), cp3009 frontend hit (2)

Eqiad is correct:

$ wget -S --header 'host: upload.wikimedia.org' 'http://upload-lb.eqiad.wikimedia.org/wikipedia/commons/0/04/Wikipedia-logo-v2-uz.png'
[...]

Last-Modified: Wed, 20 Mar 2013 01:12:41 GMT
Etag: 66bb874273a5120bb99c83c7827151d8
X-Timestamp: 1363741961.71944
Content-Type: image/png
X-Varnish: 1993846029, 2883426698
Via: 1.1 varnish, 1.1 varnish
Content-Length: 18240
Accept-Ranges: bytes
Date: Wed, 20 Mar 2013 14:59:35 GMT
Age: 0
Connection: keep-alive
X-Cache: cp1024 miss (0), cp1021 frontend miss (0)
Nemo_bis added a comment.Via ConduitMar 20 2013, 5:02 PM

15:12 mark: Fixed purging for all sites but eqiad (which wasn't broken)

Aklapper added a comment.Via ConduitMar 20 2013, 6:12 PM

Nemo was quicker (thanks!) and I had a mid-air collision when coming back. :)

https://wikitech.wikimedia.org/wiki/Server_admin_log says for March 20:

  • 15:12 mark: Fixed purging for all sites but eqiad (which wasn't broken)

Add Comment

Column Prototype
This is a very early prototype of a persistent column. It is not expected to work yet, and leaving it open will activate other new features which will break things. Press "\" (backslash) on your keyboard to close it now.