Page MenuHomePhabricator

Timeouts towards ms-fe.svc.codfw.wmnet from jobrunners
Closed, DeclinedPublicPRODUCTION ERROR

Description

Error
  • mwversion: 1.46.0-wmf.7
  • timestamp: 2026-01-01T10:18:25.105Z
  • phpversion: 8.3.28
  • reqId: f6aab498-8054-4bf2-ae80-cb52736d4229
  • Find reqId in Logstash
normalized_message
HEAD http://ms-fe.svc.codfw.wmnet/wikipedia/commons/thumb/d/dc/Number_1218_-_DPLA_-_75609bff5f93248f0acc38c3b99b4cc2_%28page_260%29.tiff/lossy-page1-320px-Number_1218_-_DPLA_-_75609bff5f93248f0acc38c3b99b4cc2_%28page_260%29.tiff.jpg HTTP/1.1 - NULL cURL e
exception.trace
Impact

TBA

Notes
HEAD http://ms-fe.svc.codfw.wmnet/wikipedia/commons/thumb/d/dc/Number_1218_-_DPLA_-_75609bff5f93248f0acc38c3b99b4cc2_%28page_260%29.tiff/lossy-page1-320px-Number_1218_-_DPLA_-_75609bff5f93248f0acc38c3b99b4cc2_%28page_260%29.tiff.jpg HTTP/1.1 - NULL cURL error 28: Connection timed out after 1001 milliseconds (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for http://ms-fe.svc.codfw.wmnet/wikipedia/commons/thumb/d/dc/Number_1218_-_DPLA_-_75609bff5f93248f0acc38c3b99b4cc2_%28page_260%29.tiff/lossy-page1-320px-Number_1218_-_DPLA_-_75609bff5f93248f0acc38c3b99b4cc2_%28page_260%29.tiff.jpg

Details

Request URL
https://mw-jobrunner.discovery.wmnet/rpc/RunSingleJob.php

Event Timeline

I think this was likely a thumbor timeout - it works for me now, but I see the thumb is newly-generated:

root@ms-fe2009:/home/mvernon# swift stat wikipedia-commons-local-thumb.dc 'd/dc/Number_1218_-_DPLA__75609bff5f93248f0acc38c3b99b4cc2_(page_260).tiff/lossy-page1-320px-Number_1218_-_DPLA__75609bff5f93248f0acc38c3b99b4cc2_(page_260).tiff.jpg'
               Account: AUTH_mw
             Container: wikipedia-commons-local-thumb.dc
                Object: d/dc/Number_1218_-_DPLA_-_75609bff5f93248f0acc38c3b99b4cc2_(page_260).tiff/lossy-page1-320px-Number_1218_-_DPLA_-_75609bff5f93248f0acc38c3b99b4cc2_(page_260).tiff.jpg
          Content Type: image/jpeg
        Content Length: 9418
         Last Modified: Fri, 02 Jan 2026 09:12:47 GMT
                  ETag: 399f13b257b10ab897322d4fa78d7eac
   Content-Disposition: inline;filename*=UTF-8''Number_1218_-_DPLA_-_75609bff5f93248f0acc38c3b99b4cc2_(page_260).tiff.jpg
           X-Timestamp: 1767345166.74425
         Accept-Ranges: bytes
            X-Trans-Id: txa64c206f7a594632aae08-0069578d06
X-Openstack-Request-Id: txa64c206f7a594632aae08-0069578d06

Swift dashboards for the last couple of days don't look unusual to me, FWIW.

thank you @MatthewVernon ! I am closing this in favour or T414967, as I have noticed more timeouts.