[[commons:File:Things near the Nautical Museum of Litochoro 10.jpg]] only present in codfw
Closed, ResolvedPublicBUG REPORT
Actions

Assigned To

Authored By

	Lvova
	Oct 2 2025, 4:49 PM

Project Tags

Referenced Files

None

Subscribers

Description

The image https://commons.wikimedia.org/wiki/File:Things_near_the_Nautical_Museum_of_Litochoro_10.jpg is not displaying for me and other regular (non-admin) users.

The image appears to be missing or inaccessible when viewed in a browser, even after clearing the cache, trying different browsers, using incognito mode, VPNs, and different countries/IPs.
However, the image is visible to the user kaganer, who is an administrator.
This suggests that there may be some permission issue, visibility setting, or backend problem affecting file access for non-privileged users.

Related Objects

Mentioned In: T430561: file missing after move

Event Timeline

Lvova created this task.Oct 2 2025, 4:49 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 2 2025, 4:49 PM

I assume this is about https://commons.wikimedia.org/wiki/File:Things_near_the_Nautical_Museum_of_Litochoro_10.jpg ? For future reference, please fill in the bug report template - thanks!

I get a 404 error for the image file itself. Adding SRE-swift-storage as this does not look like a thumbnailing problem.

A_smart_kitten updated the task description. (Show Details)Oct 2 2025, 4:52 PM

A_smart_kitten subscribed.

Aklapper renamed this task from Image not visible to regular users, only visible to admin to HTTP 404 error for image to regular users, only visible to admin.Oct 2 2025, 4:53 PM

Aklapper updated the task description. (Show Details)

Seems to be a datacenter-specific issue:

$ curl -o /dev/null -v --connect-to ::upload-lb.eqiad.wikimedia.org https://upload.wikimedia.org/wikipedia/commons/thumb/c/cf/Things_near_the_Nautical_Museum_of_Litochoro_10.jpg/120px-Things_near_the_Nautical_Museum_of_Litochoro_10.jpg 2>&1 | grep "< HTTP"
< HTTP/2 404 
$ curl -o /dev/null -v --connect-to ::upload-lb.codfw.wikimedia.org https://upload.wikimedia.org/wikipedia/commons/thumb/c/cf/Things_near_the_Nautical_Museum_of_Litochoro_10.jpg/120px-Things_near_the_Nautical_Museum_of_Litochoro_10.jpg 2>&1 | grep "< HTTP"
< HTTP/2 200

As expected from the report, the object is in codfw, but not eqiad:

root@ms-fe1009:~# swift stat wikipedia-commons-local-public.cf c/cf/Things_near_the_Nautical_Museum_of_Litochoro_10.jpg
Object HEAD failed: http://ms-fe.svc.eqiad.wmnet/v1/AUTH_mw/wikipedia-commons-local-public.cf/c/cf/Things_near_the_Nautical_Museum_of_Litochoro_10.jpg 404 Not Found
Failed Transaction ID: tx7ac2ef9f7c79486d84f80-0068dfb6b8
root@ms-fe2009:~# swift stat wikipedia-commons-local-public.cf c/cf/Things_near_the_Nautical_Museum_of_Litochoro_10.jpg
               Account: AUTH_mw
             Container: wikipedia-commons-local-public.cf
                Object: c/cf/Things_near_the_Nautical_Museum_of_Litochoro_10.jpg
          Content Type: image/jpeg
        Content Length: 7341349
         Last Modified: Tue, 30 Sep 2025 20:53:18 GMT
                  ETag: 6383419dbeec344b83cb353b472ab95f
       Meta Sha1Base36: 957witx4ns8tlo71g1o7h4qsrpng7v9
           X-Timestamp: 1759265597.26006
         Accept-Ranges: bytes
            X-Trans-Id: tx7c819eef97524e50a8617-0068dfb6b7
X-Openstack-Request-Id: tx7c819eef97524e50a8617-0068dfb6b7

Searching the eqiad swift logs on 30th September for this image (sudo cumin -x A:swift-fe-eqiad 'zgrep -F "c/cf/Things_near_the_Nautical_Museum_of_Litochoro_10.jpg" /var/log/swift/proxy-access.log.3.gz' ) finds me only HEADs and GETs (largely of thumbs), all of which say 404.

So something went wrong during the upload, I can't tell you what (I can't find a PUT that failed, for example). As codfw is currently the primary datacenter, Monday's rclone run will copy this to eqiad, or someone could re-upload in the mean time if that's too long to wait.

As expected, the Monday rclone copied this image across:

curl -o /dev/null -v --connect-to ::upload-lb.eqiad.wikimedia.org https://upload.wikimedia.org/wikipedia/commons/c/cf/Things_near_the_Nautical_Museum_of_Litochoro_10.jpg 2>&1 | grep "< HTTP"
< HTTP/2 200

Lvova mentioned this in T430561: file missing after move.Jun 29 2026, 3:59 PM

[[commons:File:Things near the Nautical Museum of Litochoro 10.jpg]] only present in codfwClosed, ResolvedPublicBUG REPORTActions

Description

Related Objects

Event Timeline

[[commons:File:Things near the Nautical Museum of Litochoro 10.jpg]] only present in codfw
Closed, ResolvedPublicBUG REPORT
Actions