Timeouts towards ms-fe.svc.codfw.wmnet from jobrunners
Closed, DeclinedPublicPRODUCTION ERROR
Actions

Assigned To

None

Authored By

	jijiki
	Jan 1 2026, 10:40 AM

Description

Error

mwversion: 1.46.0-wmf.7
timestamp: 2026-01-01T10:18:25.105Z
phpversion: 8.3.28
reqId: f6aab498-8054-4bf2-ae80-cb52736d4229
Find reqId in Logstash

normalized_message

HEAD http://ms-fe.svc.codfw.wmnet/wikipedia/commons/thumb/d/dc/Number_1218_-_DPLA_-_75609bff5f93248f0acc38c3b99b4cc2_%28page_260%29.tiff/lossy-page1-320px-Number_1218_-_DPLA_-_75609bff5f93248f0acc38c3b99b4cc2_%28page_260%29.tiff.jpg HTTP/1.1 - NULL cURL e

exception.trace

Impact

TBA

Notes

HEAD http://ms-fe.svc.codfw.wmnet/wikipedia/commons/thumb/d/dc/Number_1218_-_DPLA_-_75609bff5f93248f0acc38c3b99b4cc2_%28page_260%29.tiff/lossy-page1-320px-Number_1218_-_DPLA_-_75609bff5f93248f0acc38c3b99b4cc2_%28page_260%29.tiff.jpg HTTP/1.1 - NULL cURL error 28: Connection timed out after 1001 milliseconds (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for http://ms-fe.svc.codfw.wmnet/wikipedia/commons/thumb/d/dc/Number_1218_-_DPLA_-_75609bff5f93248f0acc38c3b99b4cc2_%28page_260%29.tiff/lossy-page1-320px-Number_1218_-_DPLA_-_75609bff5f93248f0acc38c3b99b4cc2_%28page_260%29.tiff.jpg

Details

Request URL: https://mw-jobrunner.discovery.wmnet/rpc/RunSingleJob.php

Related Objects

Mentioned Here: T414967: mw-jobrunner curl errors when talking to other services

Event Timeline

jijiki created this task.Jan 1 2026, 10:40 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 1 2026, 10:40 AM

Pppery added a project: SRE-swift-storage.Jan 1 2026, 3:35 PM

I think this was likely a thumbor timeout - it works for me now, but I see the thumb is newly-generated:

root@ms-fe2009:/home/mvernon# swift stat wikipedia-commons-local-thumb.dc 'd/dc/Number_1218_-_DPLA__75609bff5f93248f0acc38c3b99b4cc2_(page_260).tiff/lossy-page1-320px-Number_1218_-_DPLA__75609bff5f93248f0acc38c3b99b4cc2_(page_260).tiff.jpg'
               Account: AUTH_mw
             Container: wikipedia-commons-local-thumb.dc
                Object: d/dc/Number_1218_-_DPLA_-_75609bff5f93248f0acc38c3b99b4cc2_(page_260).tiff/lossy-page1-320px-Number_1218_-_DPLA_-_75609bff5f93248f0acc38c3b99b4cc2_(page_260).tiff.jpg
          Content Type: image/jpeg
        Content Length: 9418
         Last Modified: Fri, 02 Jan 2026 09:12:47 GMT
                  ETag: 399f13b257b10ab897322d4fa78d7eac
   Content-Disposition: inline;filename*=UTF-8''Number_1218_-_DPLA_-_75609bff5f93248f0acc38c3b99b4cc2_(page_260).tiff.jpg
           X-Timestamp: 1767345166.74425
         Accept-Ranges: bytes
            X-Trans-Id: txa64c206f7a594632aae08-0069578d06
X-Openstack-Request-Id: txa64c206f7a594632aae08-0069578d06

Swift dashboards for the last couple of days don't look unusual to me, FWIW.

thcipriani moved this task from Untriaged to Jan–Mar 2026 on the Wikimedia-production-error board.Jan 15 2026, 5:51 PM

thank you @MatthewVernon ! I am closing this in favour or T414967, as I have noticed more timeouts.

Timeouts towards ms-fe.svc.codfw.wmnet from jobrunnersClosed, DeclinedPublicPRODUCTION ERRORActions