Page MenuHomePhabricator

220px of Fachada_e_lateral_da_Catedral_São_Sebastião_após_pintura,_Coronel_Fabriciano_MG.JPG not purging
Closed, ResolvedPublic

Description

Compare https://upload.wikimedia.org/wikipedia/commons/thumb/a/a0/Fachada_e_lateral_da_Catedral_S%C3%A3o_Sebasti%C3%A3o_ap%C3%B3s_pintura%2C_Coronel_Fabriciano_MG.JPG/220px-Fachada_e_lateral_da_Catedral_S%C3%A3o_Sebasti%C3%A3o_ap%C3%B3s_pintura%2C_Coronel_Fabriciano_MG.JPG and https://upload.wikimedia.org/wikipedia/commons/thumb/a/a0/Fachada_e_lateral_da_Catedral_S%C3%A3o_Sebasti%C3%A3o_ap%C3%B3s_pintura%2C_Coronel_Fabriciano_MG.JPG/221px-Fachada_e_lateral_da_Catedral_S%C3%A3o_Sebasti%C3%A3o_ap%C3%B3s_pintura%2C_Coronel_Fabriciano_MG.JPG

The 220px version is showing the old version. It continues to show the old version after purging. Purge on the image page does not reset the Age: header. Going to https://upload.wikimedia.org/wikipedia/commons/thumb/a/a0/Fachada_e_lateral_da_Catedral_S%C3%A3o_Sebasti%C3%A3o_ap%C3%B3s_pintura%2C_Coronel_Fabriciano_MG.JPG/220px-Fachada_e_lateral_da_Catedral_S%C3%A3o_Sebasti%C3%A3o_ap%C3%B3s_pintura%2C_Coronel_Fabriciano_MG.JPG?junkAtEndToBypassVarnish still returns the old version (with cache miss headers), which suggests that this is a swift issue not a varnish issue.

So from the outside it would appear that (This is guesswork):
*Swift has an (old) version of that file at 220px
*When MediaWiki asks swift for a list of all thumbnails I would presume that particular one is being omitted
**Thus no htcp purge is sent for that size, and no delete is issued for that thumbnail to swift
*As a result, thumbnail gets stuck

Note: This is not the first time an issue like this has happened, but reports of this type of issue have been very rare.

Perhaps someone with access could do some debugging on swift and just try and see if something is weird about this particular object. Allegedly you can get statistics on a specific object by doing something like: swift -A http://127.0.0.1/auth/v1.0 -U mw:thumb -K $pass stat wikipedia-commons-local-thumb.a0 a/a0/Fachada_e_lateral_da_Catedral_S%C3%A3o_Sebasti%C3%A3o_ap%C3%B3s_pintura%2C_Coronel_Fabriciano_MG.JPG/220px-Fachada_e_lateral_da_Catedral_S%C3%A3o_Sebasti%C3%A3o_ap%C3%B3s_pintura%2C_Coronel_Fabriciano_MG.JPG


hmm, how do I mark this blocking T43371

Event Timeline

Bawolff created this task.Apr 7 2015, 7:48 PM
Bawolff raised the priority of this task from to Needs Triage.
Bawolff updated the task description. (Show Details)
Bawolff added subscribers: Bawolff, Gilles.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 7 2015, 7:48 PM
Bawolff updated the task description. (Show Details)Apr 7 2015, 7:48 PM
Bawolff set Security to None.

not sure there's a way to block a task after the fact

anyway there doesn't seem to be a 220px file in swift

# swift list wikipedia-commons-local-thumb.a0 --prefix a/a0/Fachada_e_lateral_da_Catedral | sort -n
a/a0/Fachada_e_lateral_da_Catedral_São_Sebastião_após_pintura,_Coronel_Fabriciano_MG.JPG/1024px-Fachada_e_lateral_da_Catedral_São_Sebastião_após_pintura,_Coronel_Fabriciano_MG.JPG
a/a0/Fachada_e_lateral_da_Catedral_São_Sebastião_após_pintura,_Coronel_Fabriciano_MG.JPG/120px-Fachada_e_lateral_da_Catedral_São_Sebastião_após_pintura,_Coronel_Fabriciano_MG.JPG
a/a0/Fachada_e_lateral_da_Catedral_São_Sebastião_após_pintura,_Coronel_Fabriciano_MG.JPG/1280px-Fachada_e_lateral_da_Catedral_São_Sebastião_após_pintura,_Coronel_Fabriciano_MG.JPG
a/a0/Fachada_e_lateral_da_Catedral_São_Sebastião_após_pintura,_Coronel_Fabriciano_MG.JPG/130px-Fachada_e_lateral_da_Catedral_São_Sebastião_após_pintura,_Coronel_Fabriciano_MG.JPG
a/a0/Fachada_e_lateral_da_Catedral_São_Sebastião_após_pintura,_Coronel_Fabriciano_MG.JPG/1920px-Fachada_e_lateral_da_Catedral_São_Sebastião_após_pintura,_Coronel_Fabriciano_MG.JPG
a/a0/Fachada_e_lateral_da_Catedral_São_Sebastião_após_pintura,_Coronel_Fabriciano_MG.JPG/210px-Fachada_e_lateral_da_Catedral_São_Sebastião_após_pintura,_Coronel_Fabriciano_MG.JPG
a/a0/Fachada_e_lateral_da_Catedral_São_Sebastião_após_pintura,_Coronel_Fabriciano_MG.JPG/221px-Fachada_e_lateral_da_Catedral_São_Sebastião_após_pintura,_Coronel_Fabriciano_MG.JPG
a/a0/Fachada_e_lateral_da_Catedral_São_Sebastião_após_pintura,_Coronel_Fabriciano_MG.JPG/225px-Fachada_e_lateral_da_Catedral_São_Sebastião_após_pintura,_Coronel_Fabriciano_MG.JPG
a/a0/Fachada_e_lateral_da_Catedral_São_Sebastião_após_pintura,_Coronel_Fabriciano_MG.JPG/240px-Fachada_e_lateral_da_Catedral_São_Sebastião_após_pintura,_Coronel_Fabriciano_MG.JPG
a/a0/Fachada_e_lateral_da_Catedral_São_Sebastião_após_pintura,_Coronel_Fabriciano_MG.JPG/2560px-Fachada_e_lateral_da_Catedral_São_Sebastião_após_pintura,_Coronel_Fabriciano_MG.JPG
a/a0/Fachada_e_lateral_da_Catedral_São_Sebastião_após_pintura,_Coronel_Fabriciano_MG.JPG/260px-Fachada_e_lateral_da_Catedral_São_Sebastião_após_pintura,_Coronel_Fabriciano_MG.JPG
a/a0/Fachada_e_lateral_da_Catedral_São_Sebastião_após_pintura,_Coronel_Fabriciano_MG.JPG/320px-Fachada_e_lateral_da_Catedral_São_Sebastião_após_pintura,_Coronel_Fabriciano_MG.JPG
a/a0/Fachada_e_lateral_da_Catedral_São_Sebastião_após_pintura,_Coronel_Fabriciano_MG.JPG/330px-Fachada_e_lateral_da_Catedral_São_Sebastião_após_pintura,_Coronel_Fabriciano_MG.JPG
a/a0/Fachada_e_lateral_da_Catedral_São_Sebastião_após_pintura,_Coronel_Fabriciano_MG.JPG/440px-Fachada_e_lateral_da_Catedral_São_Sebastião_após_pintura,_Coronel_Fabriciano_MG.JPG
a/a0/Fachada_e_lateral_da_Catedral_São_Sebastião_após_pintura,_Coronel_Fabriciano_MG.JPG/640px-Fachada_e_lateral_da_Catedral_São_Sebastião_após_pintura,_Coronel_Fabriciano_MG.JPG
a/a0/Fachada_e_lateral_da_Catedral_São_Sebastião_após_pintura,_Coronel_Fabriciano_MG.JPG/72px-Fachada_e_lateral_da_Catedral_São_Sebastião_após_pintura,_Coronel_Fabriciano_MG.JPG
a/a0/Fachada_e_lateral_da_Catedral_São_Sebastião_após_pintura,_Coronel_Fabriciano_MG.JPG/800px-Fachada_e_lateral_da_Catedral_São_Sebastião_após_pintura,_Coronel_Fabriciano_MG.JPG

btw on the 220px I got x-cache:cp1063 hit (26), cp3004 hit (8), cp3009 frontend miss (0)
and this on the 221px x-cache:cp1051 hit (1), cp3007 hit (1), cp3009 frontend miss (0)

Aklapper triaged this task as High priority.Apr 8 2015, 11:28 AM

I don't think it's directly related to the esams depool, although perhaps the excess swift load has induced errors at the swift layer? As in the original report, if I put a cache-busting query string on the 220px URL, varnish gives a double-miss and seems to fetch the old image from Swift, with a Last-Modified of Sat, 11 Oct 2014 22:04:07 GMT. This seems to be in direct conflict with "swift list" claiming there isn't a 220px. Confirmed that 219px and 222px also show the new image, it's just 220 that seems to be stuck somehow...

Steinsplitter moved this task from Incoming to Backlog on the Commons board.Apr 8 2015, 1:04 PM

indeed Brandon is correct and I jumped the gun, the file is on one of the handoff nodes:

root@ms-fe1001:~# swift-get-nodes /etc/swift/object.ring.gz AUTH_mw wikipedia-commons-local-thumb.a0 a/a0/Fachada_e_lateral_da_Catedral_São_Sebastião_após_pintura,_Coronel_Fabriciano_MG.JPG/220px-Fachada_e_lateral_da_Catedral_São_Sebastião_após_pintura,_Coronel_Fabriciano_MG.JPG

Account 	AUTH_mw
Container	wikipedia-commons-local-thumb.a0
Object   	a/a0/Fachada_e_lateral_da_Catedral_São_Sebastião_após_pintura,_Coronel_Fabriciano_MG.JPG/220px-Fachada_e_lateral_da_Catedral_São_Sebastião_após_pintura,_Coronel_Fabriciano_MG.JPG


Partition	41724
Hash       	a2fc89e070be62a9832e4b0390a049cd

Server:Port Device	10.64.48.32:6000 sdi1
Server:Port Device	10.64.16.165:6000 sdm1
Server:Port Device	10.64.0.175:6000 sdg1
Server:Port Device	10.64.0.177:6000 sdl1	 [Handoff]
Server:Port Device	10.64.32.10:6000 sdi1	 [Handoff]
Server:Port Device	10.64.32.14:6000 sdj1	 [Handoff]


curl -I -XHEAD "http://10.64.48.32:6000/sdi1/41724/AUTH_mw/wikipedia-commons-local-thumb.a0/a/a0/Fachada_e_lateral_da_Catedral_S%C3%A3o_Sebasti%C3%A3o_ap%C3%B3s_pintura%2C_Coronel_Fabriciano_MG.JPG/220px-Fachada_e_lateral_da_Catedral_S%C3%A3o_Sebasti%C3%A3o_ap%C3%B3s_pintura%2C_Coronel_Fabriciano_MG.JPG"
curl -I -XHEAD "http://10.64.16.165:6000/sdm1/41724/AUTH_mw/wikipedia-commons-local-thumb.a0/a/a0/Fachada_e_lateral_da_Catedral_S%C3%A3o_Sebasti%C3%A3o_ap%C3%B3s_pintura%2C_Coronel_Fabriciano_MG.JPG/220px-Fachada_e_lateral_da_Catedral_S%C3%A3o_Sebasti%C3%A3o_ap%C3%B3s_pintura%2C_Coronel_Fabriciano_MG.JPG"
curl -I -XHEAD "http://10.64.0.175:6000/sdg1/41724/AUTH_mw/wikipedia-commons-local-thumb.a0/a/a0/Fachada_e_lateral_da_Catedral_S%C3%A3o_Sebasti%C3%A3o_ap%C3%B3s_pintura%2C_Coronel_Fabriciano_MG.JPG/220px-Fachada_e_lateral_da_Catedral_S%C3%A3o_Sebasti%C3%A3o_ap%C3%B3s_pintura%2C_Coronel_Fabriciano_MG.JPG"
curl -I -XHEAD "http://10.64.0.177:6000/sdl1/41724/AUTH_mw/wikipedia-commons-local-thumb.a0/a/a0/Fachada_e_lateral_da_Catedral_S%C3%A3o_Sebasti%C3%A3o_ap%C3%B3s_pintura%2C_Coronel_Fabriciano_MG.JPG/220px-Fachada_e_lateral_da_Catedral_S%C3%A3o_Sebasti%C3%A3o_ap%C3%B3s_pintura%2C_Coronel_Fabriciano_MG.JPG" # [Handoff]
curl -I -XHEAD "http://10.64.32.10:6000/sdi1/41724/AUTH_mw/wikipedia-commons-local-thumb.a0/a/a0/Fachada_e_lateral_da_Catedral_S%C3%A3o_Sebasti%C3%A3o_ap%C3%B3s_pintura%2C_Coronel_Fabriciano_MG.JPG/220px-Fachada_e_lateral_da_Catedral_S%C3%A3o_Sebasti%C3%A3o_ap%C3%B3s_pintura%2C_Coronel_Fabriciano_MG.JPG" # [Handoff]
curl -I -XHEAD "http://10.64.32.14:6000/sdj1/41724/AUTH_mw/wikipedia-commons-local-thumb.a0/a/a0/Fachada_e_lateral_da_Catedral_S%C3%A3o_Sebasti%C3%A3o_ap%C3%B3s_pintura%2C_Coronel_Fabriciano_MG.JPG/220px-Fachada_e_lateral_da_Catedral_S%C3%A3o_Sebasti%C3%A3o_ap%C3%B3s_pintura%2C_Coronel_Fabriciano_MG.JPG" # [Handoff]


Use your own device location of servers:
such as "export DEVICE=/srv/node"
ssh 10.64.48.32 "ls -lah ${DEVICE:-/srv/node}/sdi1/objects/41724/9cd/a2fc89e070be62a9832e4b0390a049cd/"
ssh 10.64.16.165 "ls -lah ${DEVICE:-/srv/node}/sdm1/objects/41724/9cd/a2fc89e070be62a9832e4b0390a049cd/"
ssh 10.64.0.175 "ls -lah ${DEVICE:-/srv/node}/sdg1/objects/41724/9cd/a2fc89e070be62a9832e4b0390a049cd/"
ssh 10.64.0.177 "ls -lah ${DEVICE:-/srv/node}/sdl1/objects/41724/9cd/a2fc89e070be62a9832e4b0390a049cd/" # [Handoff]
ssh 10.64.32.10 "ls -lah ${DEVICE:-/srv/node}/sdi1/objects/41724/9cd/a2fc89e070be62a9832e4b0390a049cd/" # [Handoff]
ssh 10.64.32.14 "ls -lah ${DEVICE:-/srv/node}/sdj1/objects/41724/9cd/a2fc89e070be62a9832e4b0390a049cd/" # [Handoff]
root@ms-fe1001:~# curl -I -XHEAD "http://10.64.48.32:6000/sdi1/41724/AUTH_mw/wikipedia-commons-local-thumb.a0/a/a0/Fachada_e_lateral_da_Catedral_S%C3%A3o_Sebasti%C3%A3o_ap%C3%B3s_pintura%2C_Coronel_Fabriciano_MG.JPG/220px-Fachada_e_lateral_da_Catedral_S%C3%A3o_Sebasti%C3%A3o_ap%C3%B3s_pintura%2C_Coronel_Fabriciano_MG.JPG"
HTTP/1.1 404 Not Found
Content-Type: text/html; charset=UTF-8
Content-Length: 0
Date: Wed, 08 Apr 2015 14:07:15 GMT

root@ms-fe1001:~# curl -I -XHEAD "http://10.64.48.32:6000/sdi1/41724/AUTH_mw/wikipedia-commons-local-thumb.a0/a/a0/Fachada_e_lateral_da_Catedral_S%C3%A3o_Sebasti%C3%A3o_ap%C3%B3s_pintura%2C_Coronel_Fabriciano_MG.JPG/220px-Fachada_e_lateral_da_Catedral_S%C3%A3o_Sebasti%C3%A3o_ap%C3%B3s_pintura%2C_Coronel_Fabriciano_MG.JPG"
HTTP/1.1 404 Not Found
Content-Type: text/html; charset=UTF-8
Content-Length: 0
Date: Wed, 08 Apr 2015 14:07:24 GMT

root@ms-fe1001:~# curl -I -XHEAD "http://10.64.16.165:6000/sdm1/41724/AUTH_mw/wikipedia-commons-local-thumb.a0/a/a0/Fachada_e_lateral_da_Catedral_S%C3%A3o_Sebasti%C3%A3o_ap%C3%B3s_pintura%2C_Coronel_Fabriciano_MG.JPG/220px-Fachada_e_lateral_da_Catedral_S%C3%A3o_Sebasti%C3%A3o_ap%C3%B3s_pintura%2C_Coronel_Fabriciano_MG.JPG"
HTTP/1.1 404 Not Found
Content-Type: text/html; charset=UTF-8
Content-Length: 0
Date: Wed, 08 Apr 2015 14:07:24 GMT

root@ms-fe1001:~# curl -I -XHEAD "http://10.64.0.175:6000/sdg1/41724/AUTH_mw/wikipedia-commons-local-thumb.a0/a/a0/Fachada_e_lateral_da_Catedral_S%C3%A3o_Sebasti%C3%A3o_ap%C3%B3s_pintura%2C_Coronel_Fabriciano_MG.JPG/220px-Fachada_e_lateral_da_Catedral_S%C3%A3o_Sebasti%C3%A3o_ap%C3%B3s_pintura%2C_Coronel_Fabriciano_MG.JPG"
HTTP/1.1 404 Not Found
Content-Type: text/html; charset=UTF-8
Content-Length: 0
Date: Wed, 08 Apr 2015 14:07:26 GMT

root@ms-fe1001:~# curl -I -XHEAD "http://10.64.0.177:6000/sdl1/41724/AUTH_mw/wikipedia-commons-local-thumb.a0/a/a0/Fachada_e_lateral_da_Catedral_S%C3%A3o_Sebasti%C3%A3o_ap%C3%B3s_pintura%2C_Coronel_Fabriciano_MG.JPG/220px-Fachada_e_lateral_da_Catedral_S%C3%A3o_Sebasti%C3%A3o_ap%C3%B3s_pintura%2C_Coronel_Fabriciano_MG.JPG" # [Handoff]
HTTP/1.1 404 Not Found
Content-Type: text/html; charset=UTF-8
Content-Length: 0
Date: Wed, 08 Apr 2015 14:08:00 GMT

root@ms-fe1001:~# curl -I -XHEAD "http://10.64.32.10:6000/sdi1/41724/AUTH_mw/wikipedia-commons-local-thumb.a0/a/a0/Fachada_e_lateral_da_Catedral_S%C3%A3o_Sebasti%C3%A3o_ap%C3%B3s_pintura%2C_Coronel_Fabriciano_MG.JPG/220px-Fachada_e_lateral_da_Catedral_S%C3%A3o_Sebasti%C3%A3o_ap%C3%B3s_pintura%2C_Coronel_Fabriciano_MG.JPG" # [Handoff]
HTTP/1.1 200 OK
Content-Length: 15072
X-Object-Meta-Sha1base36: 8fcugatnhqd5ttdjg7iq4gym5p2iymp
Content-Disposition: inline;filename*=UTF-8''Fachada_e_lateral_da_Catedral_S%C3%A3o_Sebasti%C3%A3o_ap%C3%B3s_pintura%2C_Coronel_Fabriciano_MG.JPG
Last-Modified: Sat, 11 Oct 2014 22:04:07 GMT
Etag: "f09454d282e53adba7cf144ea827cf46"
X-Timestamp: 1413065046.80895
Content-Type: image/jpeg
Date: Wed, 08 Apr 2015 14:08:00 GMT

root@ms-fe1001:~# curl -I -XHEAD "http://10.64.32.14:6000/sdj1/41724/AUTH_mw/wikipedia-commons-local-thumb.a0/a/a0/Fachada_e_lateral_da_Catedral_S%C3%A3o_Sebasti%C3%A3o_ap%C3%B3s_pintura%2C_Coronel_Fabriciano_MG.JPG/220px-Fachada_e_lateral_da_Catedral_S%C3%A3o_Sebasti%C3%A3o_ap%C3%B3s_pintura%2C_Coronel_Fabriciano_MG.JPG" # [Handoff]
HTTP/1.1 404 Not Found
Content-Type: text/html; charset=UTF-8
Content-Length: 0
Date: Wed, 08 Apr 2015 14:08:00 GMT

root@ms-fe1001:~#

so the object-replicator attempted to rsync the 41724 partition and failed, given that we are also rebalancing the swift cluster with new machines I suspect the object-replicator didn't have a chance to go back to this partition yet

Apr  4 20:52:48 ms-be1005 object-replicator: <f+++++++++ 9cd/a2fc89e070be62a9832e4b0390a049cd/1413065046.80895.data
Apr  4 20:53:45 ms-be1005 object-replicator: rsync: read error: Connection reset by peer (104)
Apr  4 20:53:45 ms-be1005 object-replicator: rsync error: error in rsync protocol data stream (code 12) at io.c(764) [sender=3.0.9]
Apr  4 20:53:45 ms-be1005 object-replicator: Bad rsync return code: 12 <- ['rsync', '--recursive', '--whole-file', '--human-readable', '--xattrs', '--itemize-changes', '--ignore-existing', '--timeout=30', '--contimeout=30', '--bwlimit=0', '/srv/swift-storage/sdi1/objects/41724/303', '/srv/swift-storage/sdi1/objects/41724/d8c', '/srv/swift-storage/sdi1/objects/41724/7ee', '/srv/swift-storage/sdi1/objects/41724/181', '/srv/swift-storage/sdi1/objects/41724/123', '/srv/swift-storage/sdi1/objects/41724/7ae', '/srv/swift-storage/sdi1/objects/41724/669', '/srv/swift-storage/sdi1/objects/41724/c78', '/srv/swift-storage/sdi1/objects/41724/9d4', '/srv/swift-storage/sdi1/objects/41724/9b1', '/srv/swift-storage/sdi1/objects/41724/ec6', '/srv/swift-storage/sdi1/objects/41724/e44', '/srv/swift-storage/sdi1/objects/41724/619', '/srv/swift-storage/sdi1/objects/41724/c36', '/srv/swift-storage/sdi1/objects/41724/3ce', '/srv/swift-storage/sdi1/objects/41724/08a', '/srv/swift-storage/sdi1/objects/41724/c02', '/srv/swift-storage/sdi1/objects/41724/ce3', '/srv/swift-storage/sdi1/objects/41724/d30', '/srv/swift-storage/sdi1/objects/41724/df3', '/srv/swift-storage/sdi1/objects/41724/0b3', '/srv/swift-storage/sdi1/objects/41724/c3c', '/srv/swift-storage/sdi1/objects/41724/409', '/srv/swift-storage/sdi1/objects/41724/ad9', '/srv/swift-storage/sdi1/objects/41724/77e', '/srv/swift-storage/sdi1/objects/41724/b10', '/srv/swift-storage/sdi1/objects/41724/759', '/srv/swift-storage/sdi1/objects/41724/fa9', '/srv/swift-storage/sdi1/objects/41724/6e5', '/srv/swift-storage/sdi1/objects/41724/a30', '/srv/swift-storage/sdi1/objects/41724/546', '/srv/swift-storage/sdi1/objects/41724/edc', '/srv/swift-storage/sdi1/objects/41724/67b', '/srv/swift-storage/sdi1/objects/41724/08c', '/srv/swift-storage/sdi1/objects/41724/abc', '/srv/swift-storage/sdi1/objects/41724/69a', '/srv/swift-storage/sdi1/objects/41724/bb3', '/srv/swift-storage/sdi1/objects/41724/7e2', '/srv/swift-storage/sdi1/objects/41724/b4e', '/srv/swift-storage/sdi1/objects/41724/0dd', '/srv/swift-storage/sdi1/objec
fgiunchedi closed this task as Resolved.Apr 8 2015, 3:26 PM
fgiunchedi claimed this task.

I've manually removed the file from the handoff and purged it, I'm seeing the right version now but please double check.

I'm following up on the rsync issue which I suspect is (one of) the root cause in T95429, also possibly a swift bug since the object is no longer listed in the container anyway