No... I still see the image using the link https://upload.wikimedia.org/wikipedia/commons/archive/2/2a/20121125171455!Columbia_City_Cinema_main_hall.jpg . No 404 for me...

Had they visited it already? I also have a 404, it still sounds like the cache at the moment...

Ah, esams still has it cached, equid gives a 404.

@BBlack, do you know who in ops could verify that purges are typically going to esams, or if this is an indication of a larger issue?

I've confirmed realtime flow of PURGE traffic on esams cache instances at the varnish level, with requests that look like:

16 ReqEnd       c 2734020121 1439942720.255194902 1439942720.255226612 0.000029087 0.000018358 0.000013351
16 ReqStart     c 127.0.0.1 42514 2734020122
16 RxRequest    c PURGE
16 RxURL        c /wikipedia/commons/8/82/David_Ispiryan_1.JPG
16 RxProtocol   c HTTP/1.1
16 RxHeader     c Host: upload.wikimedia.org
16 RxHeader     c User-Agent: vhtcpd

We also have ganglia stats on PURGE traffic flowing through vhtcpd on the nodes. The rate pattern is always spiky, but doesn't seem to show any major dropouts. This is the 1w view (scroll past the generic cpu/mem/net stats):

http://ganglia.wikimedia.org/latest/?r=week&cs=&ce=&c=Upload+caches+esams&h=&tab=m&vn=&hide-hf=false&m=vhtcpd_inpkts_recvd&sh=1&z=small&hc=4&host_regex=&max_graphs=0&s=by+name

In general, our purges are not 100% reliable, but they're usually fairly reliable.

I went ahead and manually hit the one image in question in esams, should be gone now, it case it was just a rare failure. If this is systemic, we can dig deeper on why.

So this particular instance should be dealt with now. @Josve05a are you seeing any more instances?

Not that I know of at least (have no way to find any either, unless I stalk images being delted). So, perhaps just one-time thing (although it seems odd)

Thanks @Josve05a. I'm going to close this for now, but please reopen if you see any further discrepencies

In T109331#1551496, @Josve05a wrote:

Not that I know of at least (have no way to find any either, unless I stalk images being delted). So, perhaps just one-time thing (although it seems odd)

This isn't the first time something like this has happened. HTCP is an unreliable protocol, and purges sometimes don't go through.

HTCP is an unreliable protocol, and purges sometimes don't go through.

Yep. A more correct status for this report seems to be "Declined" (or "Invalid" if we think reliable purging is nearly an unsolvable problem of CS).

In T109331#1566387, @Nemo_bis wrote:

HTCP is an unreliable protocol, and purges sometimes don't go through.

Yep. A more correct status for this report seems to be "Declined" (or "Invalid" if we think reliable purging is nearly an unsolvable problem of CS).

I vaguely remember Tim talking about alternatives to multicast udp somewhere. Such things certainly do exist now (But didn't really, or at least weren't mature enough at the time HTCP was chosen from what I understand). See things like ZeroMQ and PGM.

In T109331#1566632, @Bawolff wrote:

I vaguely remember Tim talking about alternatives to multicast udp somewhere. Such things certainly do exist now (But didn't really, or at least weren't mature enough at the time HTCP was chosen from what I understand). See things like ZeroMQ and PGM.

See also: T45449#471604

The caching issue is a known problem. If i remember correctly it was reported years ago, but can't find the bug.

Yeah, known issue. I didn't realize it was as bad as it is.

We could possibly have a bot watch esams and purge stuff again if it doesn't get removed.

I made a script at terbium:~krenair/T109331.php which goes through the list of the last 100 deleted file entries on commons and checks that they 404 from upload-lb.esams.

If the rate of missed purges is truly higher in esams than elsewhere, then there's likely a technical issue here that we can investigate and address (not to prevent all possible missed purges, but to bring the excessive rate at esams back into line).

Looks like some more of these surfaced at T111838#1785015?

T129845#2173254

See new report at https://commons.wikimedia.org/w/index.php?title=Commons:Administrators%27_noticeboard&oldid=192030706#Deleted_videos_still_accessible_at_Facebook

@csteipp I think this is pretty much common knowledge by now. Do you think it'd be okay to make this task public, so to make it clear that someone is aware of the issue?

For the record I attempted to use eraseArchivedFile.php to clear the cache but doesn't work. I imagine this is already known by those smarter then me but since I know it purges some caches (like thumbnails) I thought it might do something here. It could obviously but used to force delete it from the servers but that's not generally what we do.

T129845#2173254 and https://commons.wikimedia.org/w/index.php?title=Commons:Administrators%27_noticeboard&oldid=192030706#Deleted_videos_still_accessible_at_Facebook (both mentioned above) are about this file:

• Deleted file: https://commons.wikimedia.org/wiki/File:Pee_Jaun_(Official_Video_Song)_Ft._Farhan_Saeed_Uploded_by_Free_download_Links_BD.webm
• Still accessible: https://upload.wikimedia.org/wikipedia/commons/1/18/Pee_Jaun_(Official_Video_Song)_Ft._Farhan_Saeed_Uploded_by_Free_download_Links_BD.webm

The file seems to be still cached in Varnish.

$ curl -I https://upload.wikimedia.org/wikipedia/commons/1/18/Pee_Jaun_(Official_Video_Song)_Ft._Farhan_Saeed_Uploded_by_Free_download_Links_BD.webm
HTTP/1.1 200 OK
Date: Sat, 02 Apr 2016 23:55:47 GMT
Content-Type: video/webm
Content-Length: 28563766
Connection: keep-alive
X-Object-Meta-Sha1base36: a1r0ggxeg3zflgapjy04gutf9tyo5sq
Last-Modified: Sat, 02 Apr 2016 15:05:19 GMT
Etag: b6cf4ed823a37252ed4585033000bc88
X-Timestamp: 1459609518.07927
X-Trans-Id: txfe475a81178a4defb836c-0056ffe0fd
X-Varnish: 1021489314 1018498251, 546653600 463751166, 914281061 913860628
Via: 1.1 varnish, 1.1 varnish, 1.1 varnish
Accept-Ranges: bytes
Age: 31494
X-Cache: cp1050 hit(4), cp3046 hit(28), cp3035 frontend hit(1)
Strict-Transport-Security: max-age=31536000
Set-Cookie: WMF-Last-Access=02-Apr-2016;Path=/;HttpOnly;Expires=Wed, 04 May 2016 12:00:00 GMT
X-Analytics: https=1;nocookies=1
X-Client-IP: 91.218.200.230
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: Age, Date, Content-Length, Content-Range, X-Content-Duration, X-Cache, X-Varnish
Timing-Allow-Origin: *

I've purged it with purgeList.php and it appears gone. Now the question is why it didn't happen at the moment of deletion? Braindump:

• No purge was issued
• Not all purges go through
• Race condition: purge happens before the file is gone from Swift
• Edge case with percent encoding of parentheses, for example (for the reference, I didn't encode them in my purge command)

In T109331#2173552, @matmarex wrote:

I think this is pretty much common knowledge by now. Do you think it'd be okay to make this task public, so to make it clear that someone is aware of the issue?

+1 Moreover, I'm quite sure there's some other old, public bug there telling how deleted images are not always purged.

In T109331#2176611, @Platonides wrote:

In T109331#2173552, @matmarex wrote:

I think this is pretty much common knowledge by now. Do you think it'd be okay to make this task public, so to make it clear that someone is aware of the issue?

+1 Moreover, I'm quite sure there's some other old, public bug there telling how deleted images are not always purged.

Likely, it has been reported multiple times in the last years.

I made the task public, with this many duplicates filed there is really no reason not to.

@csteipp, are you currently working on this? You're set as the assignee.

another one: https://upload.wikimedia.org/wikipedia/commons/6/60/Sajid-kiptus.webm

@BBlack pointed out that if the upload.wikimedia.org URL is still accessible after deleting a file, doing action=purge on the commons.wikimedia.org deleted file's page should delete it for good (e.g. https://commons.wikimedia.org/w/index.php?title=File:Sajid-kiptus.webm&action=purge). I tried it with this file, and it didn't seem to work at first, but then the file disappeared after a few minutes; I'm not sure if it was caused by the purge action or unrelated. The next time someone runs into this problem, please try if action=purge works and report back. In general this is apparently a "known issue with no easy fix".

Another one: https://upload.wikimedia.org/wikipedia/commons/7/7f/Sajid-Monkey-Bizness.webm

There are 404 now, it's probably the time for the cache to expire. So the task goal is to trigger a removal from cache immediately after the file is deleted?

In T109331#2207412, @Dereckson wrote:

There are 404 now, it's probably the time for the cache to expire.

Though it's still visible in here but it's probably my browser.

So the task goal is to trigger a removal from cache immediately after the file is deleted?

Yes, if that's possible because there is no point of immediate deletion if they (the Facebook users) still access the files. Usually they announce in the facebook group/page that they are uploading some movie in Wikimedia than downloads took place immediately after sharing the links. So there is no other way (at this moment) to discourage them otherwise.

CTRL + SHIFT + R should do the trick in your browser. It's not possible to invalidate a cached file locally from remote as long a your browser is happy with the version and doesn't want a new one.

In T109331#2207527, @Dereckson wrote:

It's not possible to invalidate a cached file locally from remote as long a your browser is happy with the version and doesn't want a new one.

Well, the task's goal is NOT invalidate the cached version of user's browser but from the server. Similar request: T132419

In T109331#2207393, @NahidSultan wrote:

Another one: https://upload.wikimedia.org/wikipedia/commons/7/7f/Sajid-Monkey-Bizness.webm

Confirming that this file still is viewable on upload.wikimedia.org. I have never viewed the file before so it can't be my cache.

I've still a 404 for https://upload.wikimedia.org/wikipedia/commons/7/7f/Sajid-Monkey-Bizness.webm.

On IRC, I've asked 7 persons, they all have 404. Note they probably all use European upload-lb.esams.wikimedia.org.

Could you open a console/terminal, write ping upload.wikimedia.org and tell us what IP you have?

It was on a windows mobile phone on LTE data (AT&T), so I don't have a console. When viewing on my computer it gives me a 404.

In T109331#2245176, @TerraCodes wrote:

It was on a windows mobile phone on LTE data (AT&T), so I don't have a console. When viewing on my computer it gives me a 404.

Actually, I could enable internet sharing and use a laptop...

In T109331#2245057, @Dereckson wrote:

I've still a 404 for https://upload.wikimedia.org/wikipedia/commons/7/7f/Sajid-Monkey-Bizness.webm.

On IRC, I've asked 7 persons, they all have 404. Note they probably all use European upload-lb.esams.wikimedia.org.

Could you open a console/terminal, write ping upload.wikimedia.org and tell us what IP you have?

I can still access it,

ubuntu@ubuntu:~$ ping upload.wikimedia.org
PING upload.wikimedia.org (198.35.26.112) 56(84) bytes of data.
64 bytes from upload-lb.ulsfo.wikimedia.org (198.35.26.112): icmp_seq=1 ttl=49 time=354 ms

Thanks, that was useful..

I too can still see the file, and I get the same IP from ping.