I noticed recently in logs an upload failed due to 401 error in swift
Log entry (req id 98f9e25c-9b70-478a-86cc-d4fd52680b90 ):
Feb 29, 2024 @ 16:03:07 HTTP 401 (Unauthorized) in 'SwiftFileBackend::doGetFileStatMulti' (given '{"srcs":["mwstore://local-swift-codfw/local-temp/7/72/20240229155755!chunkedupload_cb23bd647e85.pdf"],"concurrency":1}')
Feb 29, 2024 @ 16:03:07 HTTP 401 (Unauthorized) in 'SwiftFileBackend::doGetFileStatMulti' (given '{"srcs":["mwstore://local-swift-codfw/local-temp/7/72/20240229155755!chunkedupload_cb23bd647e85.pdf"],"concurrency":50}')
Feb 29, 2024 @ 16:03:07 HTTP 401 (Unauthorized) in 'SwiftFileBackend::doGetFileStatMulti' (given '{"srcs":["mwstore://local-swift-codfw/local-public/archive/6/64/20240229160249!\u65b0\u9078\u5404\u540d\u516c\u91d1\u7389\u5c0d\u806f.pdf"],"concurrency":1}')
Feb 29, 2024 @ 16:03:07 HTTP 401 (Unauthorized) in 'SwiftFileBackend::doGetFileStatMulti' (given '{"srcs":["mwstore://local-swift-codfw/local-public/archive/6/64/20240229160249!\u65b0\u9078\u5404\u540d\u516c\u91d1\u7389\u5c0d\u806f.pdf"],"concurrency":50}')The fact it happened multiple times in the same request suggest that MW might have been reusing bad credentials.
This seems pretty surprising. Long term, I'd like to make the publish job be auto retried on failure, which might make things like this not be an issue if they are transient. Nonetheless filing this, because it sounds like something that probably shouldn't happen.
Looking at broader logs https://logstash.wikimedia.org/goto/0fabf2fcc66a2fa897a042de4cb2489b - it seems like this issue happens on wednesdays. Maybe connected to the deploy somehow?