In T328872#10689851, @Mike_Peel wrote:

"The MediaWiki error backend-fail-internal occured: An unknown error occurred in storage backend "local-swift-eqiad"." ...

We had a couple of incidents with swift over the weekend (including about the time you posted this comment).

I'm not sure where would be a good place to put this but I think I found something weird with how uploads work. I tried uploading a very small svg file in testwiki with excimer. 857ms was spent in the swift area but very little of it to do the upload. Let me break it down: https://performance.wikimedia.org/excimer/profile/03c4168ac24eb7eb

• First column: First it tries to get the path for the file, which triggers a fileExists() which triggers a getFileStats(), Since this is multi-backend, I assume this is doing a cross-DC connection cause it's taking 94ms to just respond.
• Then it tries to do a consistencyCheck (why? The file doesn't even exist yet?) which triggers a preLoadFileStats() call which obviously makes a new cross-DC connection taking 410ms(!) to load the file information (for something that's not even uploaded yet?) but at least since it's preLoading the information it won't make the exact same call again. Right?
• In the third column, where the actual storage of the file happens, preLoadFileStats gets called twice, I think the first time is in the primary dc and the second time in the secondary dc, each call is followed by actually storing the file. The calls for uploading takes 160ms in total but getting stats takes 130ms extra on top
• The last column is another extra call to fileExists() which takes another 40ms. The reason behind this expensive call is to determine whether it's an "archived" file or a "new" file.

That way, an upload operation that supposed to take 160ms gets stretched to 850ms by adding eight extra swift calls (four of which go across the United States and come back) on top of two to store the files. Probably we can't get rid of all of them but this could be made more efficient.

I also build a flame graph of upload paths from the daily log of May 30. Here it is:
https://people.wikimedia.org/~ladsgroup/upload.svg?x=237.7&y=981

you can see the same issue in the flame graph as well.

I think removing some of these calls could reduce the load on the frontend proxies and definitely reduce the time to store a file and reduce the chance of one of the calls failing and causing issues.

Apologies if I missed something obvious, I'm new to this codebase and it's not the easiest part of mediawiki to understand.

I think the "check the file is in a consistent (presumed-to-be-absent) state" operation is intentional, and probably replicated across other file changing call paths; not least because we get tickets sometimes when these checks fail...

I presume this is/was an intentional decision that if the two backends are in an inconsistent state (with each other, or with MW's idea of what that state should be) then we leave it for an operator to fix rather than just going ahead and replacing that inconsistent-state with whatever the user was trying to do at the time.

[I am even less familiar with the mediawiki codebase...]

Okay, I made deeper investigation. I uploaded a random file on verbose mode and here is the result:

• It made more than 30 HTTP requests to swift: https://logstash.wikimedia.org/goto/8da6c87a8ed733b9a384b238e070e66e out of them, 14 are to the remote datacenter: https://logstash.wikimedia.org/goto/4061ac5482debab31857cf7d1894d04f
• It is clearly making the same requests multiple times back to back. For example, look at 14 calls to remote datacenter. HEAD (which is called from ::getContainerStat() or ::doGetFileStatMulti()) has been called on the same path back to back before the calls to PUT (there are more after it)
  • e.g. https://logstash.wikimedia.org/app/dashboards#/doc/logstash-*/logstash-mediawiki-1-7.0.0-1-2025.06.05?id=IWR2PZcBeu0zMOTS9dTJ and https://logstash.wikimedia.org/app/dashboards#/doc/logstash-*/logstash-mediawiki-1-7.0.0-1-2025.06.05?id=gtp2PZcBc9IpmxE59lGF are duplicates of each other
  • And https://logstash.wikimedia.org/app/dashboards#/doc/logstash-*/logstash-mediawiki-1-7.0.0-1-2025.06.05?id=5WR2PZcBeu0zMOTS9NKi and https://logstash.wikimedia.org/app/dashboards#/doc/logstash-*/logstash-mediawiki-1-7.0.0-1-2025.06.05?id=g9p2PZcBc9IpmxE59lGF also duplicate of each other
  • And this is just the remote dc, the local dc is the same story.
• So there are two major issues:
  • FileBackendMultiWrite::getFileStat() checks for read backend which with the option of readAffinity set in production, it should only read from the local swift. This is clearly not happening.
  • It is making the same expensive call again and again.

The reason for that seems to be calls to ::preloadFileStat(). For example here in ::consistencyCheck():

		// Preload all of the stat info in as few round trips as possible
		foreach ( $this->backends as $backend ) {
			$realPaths = $this->substPaths( $paths, $backend );
			$backend->preloadFileStat( [ 'srcs' => $realPaths, 'latest' => true ] );
		}

Ironically, it explicitly does this to save round trips and reduce calls (see the comment).

Here is the problem: Every time ::preloadFileStat() is called, the function doesn't care whether the information is already preLoaded or not, it (re-)preLoads the information again and again with no respect to the existing cache (see FileBackendStore::preloadFileStat()). It is recommended to call this function in comments (e.g. Ideally, the file stat entry should already be preloaded via preloadFileStat(). comment) and many parts of upload code path follow the guideline and call this method and as you can see, in the above excimer profile, the same call to preLoad (and then to swift) has been seen four times, just fixing this can easily brush off a decent chunk of upload time of small files and reduce the cross-dc communication of the appservers. There are currently ~60 HEAD reqs/s to swift: https://grafana.wikimedia.org/d/OPgmB1Eiz/swift?orgId=1&from=now-2d&to=now-1m&timezone=utc&var-site=$__all&viewPanel=panel-10 that can get much lower.

Of course I will debug more and see what other improvements I can do to make the system a bit more stable.

The idea of preloadFileStat() was to allow concurrent HEAD requests to a list of objects after an relevant locks were acquired. If no locks are acquired, and "latest" is not set, maybe reusing prior loaded state entries is OK. From the perspective of FileBackend, it's mostly thinking that you call doOperations or doQuickOperations, which is supposed to do one preload (within any locking) and is done. The FileBackendMultiWrite class (itself a hack due to not having a proper regional swift cluster and swift-repl only able to do periodic reconciliation) also has to write to the remote backend and has consistency checks turned on...doing a preloads of local and remote backend. It also has to repeat the write operation on the remote backend, requiring another preload to the remote. Since FileBackendMultiWrite does it's own locking, it seems like a lot of these 'stat' entries could be reused instead of reloaded.

Another matter is FileRepo batches doing a lot of getFileStat() checks before calling doOperations(), so that's more HEAD requests since it's before the other preloads. If the batch operation locks the paths, then the FileBackendMultiWrite/FileBackendStore preloads don't need to reload over existing stuff.

Generally, if 'stat' entry in the in-memory fileStatCache was loaded after the outermost lock of that path (since locks can nest), it should just magically reuse existing values sufficient for the $latest (as done elsewhere). Of course, the FileBackendStore doesn't know about the FileBackendMultiWrite locks that FileRepo acquired...this would be easy to implement if not for FileBackendMultiWrite. Maybe a preloadStatCache() could have a $knownLockedPaths array with FileBackendMultiWrite could use (with substPaths() of course). Kind of ugly though.

I understand the need to have multi write backends and doing all write operations in both dcs but from that it requires a massive leap to require practically every operation be replicated in both dcs. For example, why doing consistency check in both dcs? let's say something is corrupted in between these two swift instances, the chance of the swift reconciliation script actually finding it or overwriting the secondary dc is much higher than someone accidentally deciding to upload a new version and then getting broken. i.e. I think mediawiki at the moment should be responsible for double uploads (and other write operations) but it shouldn't try to do integrity checks of two swift clusters (doubly so during upload). To me it's like mediawiki checking primary database and a replica for data integrity during page reads, worse than that even. It tries to do that while the replica is thousands of kilometers away. File backend shouldn't do the work of the infrastructure at run time.

I don't understand discussion around batching, most uploads don't batch. I'm not seeing any batching or performance gain from batching in graphs. Can you point me to some data that batching has any effect?

I rather just remove preLoad altogether. It demonstrably makes things faster and makes the logic much simpler.

Change #1170709 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/core@master] filebackend: Remove consistency check for multi-backend

https://gerrit.wikimedia.org/r/1170709

In T328872#10889545, @Ladsgroup wrote:

I understand the need to have multi write backends and doing all write operations in both dcs but from that it requires a massive leap to require practically every operation be replicated in both dcs. For example, why doing consistency check in both dcs? let's say something is corrupted in between these two swift instances, the chance of the swift reconciliation script actually finding it or overwriting the secondary dc is much higher than someone accidentally deciding to upload a new version and then getting broken. i.e. I think mediawiki at the moment should be responsible for double uploads (and other write operations) but it shouldn't try to do integrity checks of two swift clusters (doubly so during upload). To me it's like mediawiki checking primary database and a replica for data integrity during page reads, worse than that even. It tries to do that while the replica is thousands of kilometers away. File backend shouldn't do the work of the infrastructure at run time.

One reason for the consistency checks is to quarantine object paths that have uncertain values (inconsistent between the DC clusters). The effect of operations is different if the starting state of the files is already different. If we allow more operations, the uncertainty could spread to other object paths. Another reason was to trigger the autoResync logic to fix things and unblock the operation. This was initially before swift-repl existed. When swiftrepl was new, it was also useful for unblocking user operations if didn't get around to them fixing something yet. This might involve users retrying things like re-upload/move/delete/restore *soon* after a failure. AFAIK, swift-repl used to be slow and sometimes was configured to handle excess file deletion and other times not. There was once a time when 'autoResync' was just true (handling object deletions) and swift-repl deletions were not enabled.

Anyway, I see that "autoResync" is "conservative", so it's not able to sync a list of files in a way that swift-repl cannot. Indeed most actual resyncing will be from swift-repl since it doesn't need someone to happen to try to touch the files again. I think we can just lean into the assumption that whatever is in the primary is "canon" (FileBackendMultiwrite::doOperations already does this with it's Status result) and let swift-repl make that secondary cluster match up soon if something failed.

I don't understand discussion around batching, most uploads don't batch. I'm not seeing any batching or performance gain from batching in graphs. Can you point me to some data that batching has any effect?

By batching, I just mean anything that passes 2+ operations to doOperations(). File delete/restore can involve batches with operations proportionate to the number of file versions. Every path gets a HEAD operation to check the preconditions of the operations beforehand. The actual COPY/DELETE/POST/PUT operations of use concurrency and the use of preloadFileCache() in doOperations() does the same for the HEAD requests.

I rather just remove preLoad altogether. It demonstrably makes things faster and makes the logic much simpler.

Disabling consistencyCheck() would speed things up half the way. Setting "syncChecks" to 0 in config should produce immediate results.

External callers of preloadFileCache(), basically all but the one in doOperations(), should probably get "keep existing recent cache entries" behavior. Nevertheless, the cache entries would still not be reused within doOperations() without further changes. Doing that correctly would require that things calling FileRepo::fileExistsBatch() lock the paths first and use "latest" (none of the LocalFile*Batch classes do either). In addition, FileBackend would also have to only reuse cache entries on locked paths, tagged "latest", that were fetched after the lock was acquired. Getting FileBackendMultiwrite, which takes over the locking, to cooperate would involve some more pain.

I also don't understand why LocalFile*Batch even need to call removeNonexistentFiles() ->fileExistsBatch() -> preloadFileCache(). It seems like code could be simplified to use "ignoreMissingSource", which would also knock out the HEAD request spam.

"swift-repl" (it's not actually that any more, but something based on rclone) runs only weekly (on Monday Europe-morning).

Change #1170709 merged by jenkins-bot:

[mediawiki/core@master] filebackend: Remove consistency check for multi-backend

https://gerrit.wikimedia.org/r/1170709

Change #1194781 had a related patch set uploaded (by Aaron Schulz; author: Aaron Schulz):

[mediawiki/core@master] filebackend: remove accessibility check from multi-backend

https://gerrit.wikimedia.org/r/1194781

Change #1194781 merged by jenkins-bot:

[mediawiki/core@master] filebackend: remove accessibility check from multi-backend

https://gerrit.wikimedia.org/r/1194781

Change #1196018 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/core@wmf/1.45.0-wmf.22] filebackend: Remove consistency check for multi-backend

https://gerrit.wikimedia.org/r/1196018

Change #1196018 merged by jenkins-bot:

[mediawiki/core@wmf/1.45.0-wmf.22] filebackend: Remove consistency check for multi-backend

https://gerrit.wikimedia.org/r/1196018

Mentioned in SAL (#wikimedia-operations) [2025-10-14T11:54:46Z] <ladsgroup@deploy2002> Started scap sync-world: Backport for [[gerrit:1196018|filebackend: Remove consistency check for multi-backend (T328872)]]

Mentioned in SAL (#wikimedia-operations) [2025-10-14T11:59:00Z] <ladsgroup@deploy2002> ladsgroup: Backport for [[gerrit:1196018|filebackend: Remove consistency check for multi-backend (T328872)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Mentioned in SAL (#wikimedia-operations) [2025-10-14T12:07:32Z] <ladsgroup@deploy2002> Finished scap sync-world: Backport for [[gerrit:1196018|filebackend: Remove consistency check for multi-backend (T328872)]] (duration: 12m 46s)

In T328872#10873913, @Ladsgroup wrote:

I'm not sure where would be a good place to put this but I think I found something weird with how uploads work. I tried uploading a very small svg file in testwiki with excimer. 857ms was spent in the swift area but very little of it to do the upload. Let me break it down: https://performance.wikimedia.org/excimer/profile/03c4168ac24eb7eb

I uploaded a tiny SVG and profiled it: https://performance.wikimedia.org/excimer/profile/c2c0fe4101a6962b . It's better, but still slowish.

Change #1235490 had a related patch set uploaded (by Func; author: Func):

[mediawiki/core@master] FileBackend: Clean up unused private constants

https://gerrit.wikimedia.org/r/1235490

Change #1235491 had a related patch set uploaded (by Func; author: Func):

[operations/mediawiki-config@master] filebackend: Clean up removed config params for multi-write backends

https://gerrit.wikimedia.org/r/1235491

Change #1235490 merged by jenkins-bot:

[mediawiki/core@master] FileBackend: Clean up unused private constants

https://gerrit.wikimedia.org/r/1235490

Change #1235491 merged by jenkins-bot:

[operations/mediawiki-config@master] filebackend: Clean up removed config params for multi-write backends

https://gerrit.wikimedia.org/r/1235491

Mentioned in SAL (#wikimedia-operations) [2026-02-02T21:05:24Z] <kemayo@deploy2002> Started scap sync-world: Backport for [[gerrit:1235392|Edit check: turn off the tone a/b test on frwiki, jawiki, ptwiki (T411914)]], [[gerrit:1235111|Enable suggestions BetaFeature on beta wikis (T415504)]], [[gerrit:1230462|WikimediaCustomizations: Set WMCBadEmailDomainsFile (T397244)]], [[gerrit:1235491|filebackend: Clean up removed config params for multi-write backends (T328872)]]

Mentioned in SAL (#wikimedia-operations) [2026-02-02T21:07:21Z] <kemayo@deploy2002> tgr, func, kemayo, esanders: Backport for [[gerrit:1235392|Edit check: turn off the tone a/b test on frwiki, jawiki, ptwiki (T411914)]], [[gerrit:1235111|Enable suggestions BetaFeature on beta wikis (T415504)]], [[gerrit:1230462|WikimediaCustomizations: Set WMCBadEmailDomainsFile (T397244)]], [[gerrit:1235491|filebackend: Clean up removed config params for multi-write backends (T328872)]] synced to

Mentioned in SAL (#wikimedia-operations) [2026-02-02T21:16:18Z] <kemayo@deploy2002> Finished scap sync-world: Backport for [[gerrit:1235392|Edit check: turn off the tone a/b test on frwiki, jawiki, ptwiki (T411914)]], [[gerrit:1235111|Enable suggestions BetaFeature on beta wikis (T415504)]], [[gerrit:1230462|WikimediaCustomizations: Set WMCBadEmailDomainsFile (T397244)]], [[gerrit:1235491|filebackend: Clean up removed config params for multi-write backends (T328872)]] (duration: 10

I was looking into this a bit yesterday (more general note of improving efficiency and reliability of upload) and I realized requests to swfit from mediawiki are not going through envoy but they are using HTTPS and given that it makes around 30 different requests (half of which are to the remote datacenter) this means that significant if not majority of upload time is being spent on TLS handshakes. Maybe I'm missing something obvious but that's what it looks like from reading the code and config.

Change #1269050 had a related patch set uploaded (by Ladsgroup; author: Ladsgroup):

[operations/mediawiki-config@master] Use envoy for swift inside mediawiki

https://gerrit.wikimedia.org/r/1269050

Change #1269050 merged by jenkins-bot:

[operations/mediawiki-config@master] Use envoy for swift inside mediawiki

https://gerrit.wikimedia.org/r/1269050

Mentioned in SAL (#wikimedia-operations) [2026-04-08T20:58:12Z] <ladsgroup@deploy1003> Started scap sync-world: Backport for [[gerrit:1269050|Use envoy for swift inside mediawiki (T328872)]]

Mentioned in SAL (#wikimedia-operations) [2026-04-08T21:00:04Z] <ladsgroup@deploy1003> ladsgroup: Backport for [[gerrit:1269050|Use envoy for swift inside mediawiki (T328872)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Mentioned in SAL (#wikimedia-operations) [2026-04-08T21:04:39Z] <ladsgroup@deploy1003> Finished scap sync-world: Backport for [[gerrit:1269050|Use envoy for swift inside mediawiki (T328872)]] (duration: 06m 27s)

Change #1269420 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/puppet@production] services_proxy: Bump swift timeout

https://gerrit.wikimedia.org/r/1269420

Change #1269420 merged by Clément Goubert:

[operations/puppet@production] services_proxy: Bump swift timeout

https://gerrit.wikimedia.org/r/1269420

Change #1269524 had a related patch set uploaded (by Ladsgroup; author: Ladsgroup):

[operations/mediawiki-config@master] Revert^2 "Use envoy for swift inside mediawiki"

https://gerrit.wikimedia.org/r/1269524

Change #1269524 merged by jenkins-bot:

[operations/mediawiki-config@master] Revert^2 "Use envoy for swift inside mediawiki"

https://gerrit.wikimedia.org/r/1269524

Mentioned in SAL (#wikimedia-operations) [2026-04-09T16:41:33Z] <ladsgroup@deploy1003> Started scap sync-world: Backport for [[gerrit:1269524|Revert^2 "Use envoy for swift inside mediawiki" (T328872)]]

Mentioned in SAL (#wikimedia-operations) [2026-04-09T16:43:30Z] <ladsgroup@deploy1003> ladsgroup: Backport for [[gerrit:1269524|Revert^2 "Use envoy for swift inside mediawiki" (T328872)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Mentioned in SAL (#wikimedia-operations) [2026-04-09T16:48:36Z] <ladsgroup@deploy1003> Finished scap sync-world: Backport for [[gerrit:1269524|Revert^2 "Use envoy for swift inside mediawiki" (T328872)]] (duration: 07m 02s)

Mentioned in SAL (#wikimedia-operations) [2026-04-09T16:51:48Z] <ladsgroup@deploy1003> Started scap sync-world: Backport for [[gerrit:1269524|Revert^2 "Use envoy for swift inside mediawiki" (T328872)]]

Change #1269535 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] shellbox-video: Add swift envoy listeners

https://gerrit.wikimedia.org/r/1269535

Change #1269535 merged by jenkins-bot:

[operations/deployment-charts@master] shellbox-video: Add swift envoy listeners

https://gerrit.wikimedia.org/r/1269535

Change #1270031 had a related patch set uploaded (by Ladsgroup; author: Ladsgroup):

[operations/puppet@production] envoy: Close connections to swift after 10s of inactivity

https://gerrit.wikimedia.org/r/1270031

Change #1270031 merged by Ladsgroup:

[operations/puppet@production] envoy: Close connections to swift after 10s of inactivity

https://gerrit.wikimedia.org/r/1270031

Okay, after four tries (!) we got envoy to work. Now uploads go through envoy which provides good telemetry [1] and some okay-ish performance boost (it's harder to measure given the lack of o11y tools until now. There is https://grafana.wikimedia.org/d/000000559/mediawiki-action-api-breakdown?orgId=1&from=now-24h&to=now&timezone=utc&var-module=upload&var-query=&var-Percentile=0.5 but I haven't seen any noticeable change).

[1] https://grafana.wikimedia.org/d/b1jttnFMz/envoy-telemetry-k8s?orgId=1&from=2026-04-13T12:52:57.369Z&to=2026-04-14T12:15:43.900Z&timezone=utc&var-datasource=000000026&var-site=eqiad&var-prometheus=k8s&var-kubernetes_namespace=$__all&var-app=mediawiki&var-destination=swift-codfw&var-destination=swift-eqiad I'm also looking into ways to incorporate this into trace.wikimedia.org too.

This work so far has exposed another problem. MW code for calling swift doesn't have any retry set. We (SREs) restart swift frontends from time to time. And for various reasons, a HTTP request might simply fail (network partition, etc.) and if one of many many requests it is making fails, the whole upload goes up in the air leaving the system in a inconsistent state.

It is quite annoying when the mw class decides to do the job of the infrastructure (=doing file replication across different datacenters) but it's not prepared to handle basic infrastructure failure scenarios such as HTTP request simply failing and needing a retry.

Change #1270931 had a related patch set uploaded (by Ladsgroup; author: Ladsgroup):

[operations/puppet@production] envoy: Add 1 retry for swift services

https://gerrit.wikimedia.org/r/1270931

Change #1270931 merged by Ladsgroup:

[operations/puppet@production] envoy: Add 1 retry for swift services

https://gerrit.wikimedia.org/r/1270931

Some progress report: In the past 24 hours, we had 9 cases of requests failing out of which 6 got successfully and automatically got retried via envoy. The envoy retry mechanism also useful during reboots and other issues such as one swift frontend going down as well. For the remaining three, let me investigate what is in the swift logs.

Change #1271926 had a related patch set uploaded (by CDanis; author: CDanis):

[operations/puppet@production] envoyproxy::tls_terminator: request header rewriting

https://gerrit.wikimedia.org/r/1271926

Change #1271927 had a related patch set uploaded (by CDanis; author: CDanis):

[operations/puppet@production] swift::proxy: attempt some tracing context propagation

https://gerrit.wikimedia.org/r/1271927

Mentioned in SAL (#wikimedia-operations) [2026-04-16T15:29:01Z] <cdanis> 💔cdanis@cumin1003.eqiad.wmnet ~ 🕦☕ sudo cumin 'A:swift-fe' 'disable-puppet "cdanis deploy I3aaec0 T328872"'

Change #1271926 merged by CDanis:

[operations/puppet@production] envoyproxy::tls_terminator: request header rewriting

https://gerrit.wikimedia.org/r/1271926

Change #1271927 merged by CDanis:

[operations/puppet@production] swift::proxy: attempt some tracing context propagation

https://gerrit.wikimedia.org/r/1271927

Change #1272773 had a related patch set uploaded (by CDanis; author: CDanis):

[operations/puppet@production] Revert "swift::proxy: attempt some tracing context propagation"

https://gerrit.wikimedia.org/r/1272773

Change #1272773 merged by CDanis:

[operations/puppet@production] Revert "swift::proxy: attempt some tracing context propagation"

https://gerrit.wikimedia.org/r/1272773

Change #1272775 had a related patch set uploaded (by CDanis; author: CDanis):

[operations/puppet@production] swift::proxy: re-try some tracing context propagation

https://gerrit.wikimedia.org/r/1272775

Change #1272775 merged by CDanis:

[operations/puppet@production] swift::proxy: re-try some tracing context propagation

https://gerrit.wikimedia.org/r/1272775

Mentioned in SAL (#wikimedia-operations) [2026-04-16T16:30:11Z] <cdanis> 💙cdanis@cumin1003.eqiad.wmnet ~ 🕧☕ sudo cumin 'A:swift-fe' 'disable-puppet "cdanis deploy 8ad070a466 T328872"'

Mentioned in SAL (#wikimedia-operations) [2026-04-16T16:38:37Z] <cdanis> 💙cdanis@cumin1003.eqiad.wmnet ~ 🕧☕ sudo cumin 'A:swift-fe' 'enable-puppet "cdanis deploy 8ad070a466 T328872"'

FYI: as of my Puppet patches above, you can now use an x-request-id value to find all the intra-Swift requests associated with that request.

You do need to remove the final 4 hex digits from an x-request-id as Swift truncates them. But they appear in Swift access logs now for all the ms-fe and ms-be hosts that the request touched:

💙cdanis@cumin1003.eqiad.wmnet ~ 🕐☕ sudo cumin --force 'A:swift AND A:eqiad' 'rg a94b06fe-2eb0-40a4-9ff9-a479f633 /var/log/swift/*.log | wc -l || true'
50 hosts will be targeted:
ms-be[1064-1097].eqiad.wmnet,ms-fe[1009-1024].eqiad.wmnet
FORCE mode enabled, continuing without confirmation
===== NODE GROUP =====                                                                                                                                                                                                                                                            
(1) ms-be1082.eqiad.wmnet                                                                                                                                                                                                                                                         
----- OUTPUT for command #1: 'rg a94b06fe-2eb0... | wc -l || true' -----                                                                                                                                                                                                          
4                                                                                                                                                                                                                                                                                 
===== NODE GROUP =====                                                                                                                                                                                                                                                            
(3) ms-be[1084,1095].eqiad.wmnet,ms-fe1023.eqiad.wmnet                                                                                                                                                                                                                            
----- OUTPUT for command #1: 'rg a94b06fe-2eb0... | wc -l || true' -----                                                                                                                                                                                                          
3                                                                                                                                                                                                                                                                                 
===== NODE GROUP =====                                                                                                                                                                                                                                                            
(5) ms-be[1070,1078,1085,1097].eqiad.wmnet,ms-fe1017.eqiad.wmnet                                                                                                                                                                                                                  
----- OUTPUT for command #1: 'rg a94b06fe-2eb0... | wc -l || true' -----                                                                                                                                                                                                          
1                                                                                                                                                                                                                                                                                 
===== NODE GROUP =====                                                                                                                                                                                                                                                            
(5) ms-be[1067,1069,1087,1090,1094].eqiad.wmnet                                                                                                                                                                                                                                   
----- OUTPUT for command #1: 'rg a94b06fe-2eb0... | wc -l || true' -----                                                                                                                                                                                                          
2                                                                                                                                                                                                                                                                                 
===== NODE GROUP =====                                                                                                                                                                                                                                                            
(36) ms-be[1064-1066,1068,1071-1077,1079-1081,1083,1086,1088-1089,1091-1093,1096].eqiad.wmnet,ms-fe[1009-1016,1018-1022,1024].eqiad.wmnet                                                                                                                                         
----- OUTPUT for command #1: 'rg a94b06fe-2eb0... | wc -l || true' -----                                                                                                                                                                                                          
0                                                                                                                                                                                                                                                                                 
================                                                                                                                                                                                                                                                                  
100.0% (50/50) success ratio (>= 100.0% threshold) for command #1: 'rg a94b06fe-2eb0... | wc -l || true'.
100.0% (50/50) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.

💙cdanis@cumin1003.eqiad.wmnet ~ 🕐☕ sudo cumin --force 'A:swift AND A:codfw' 'rg a94b06fe-2eb0-40a4-9ff9-a479f633 /var/log/swift/*.log | wc -l || true'
51 hosts will be targeted:
ms-be[2062-2096].codfw.wmnet,ms-fe[2009-2024].codfw.wmnet
FORCE mode enabled, continuing without confirmation
===== NODE GROUP =====                                                                                                                                                                                                                                                            
(1) ms-be2083.codfw.wmnet                                                                                                                                                                                                                                                         
----- OUTPUT for command #1: 'rg a94b06fe-2eb0... | wc -l || true' -----                                                                                                                                                                                                          
3                                                                                                                                                                                                                                                                                 
===== NODE GROUP =====                                                                                                                                                                                                                                                            
(12) ms-be[2063-2065,2074-2075,2084,2090-2091,2093,2095-2096].codfw.wmnet,ms-fe2010.codfw.wmnet                                                                                                                                                                                   
----- OUTPUT for command #1: 'rg a94b06fe-2eb0... | wc -l || true' -----                                                                                                                                                                                                          
1                                                                                                                                                                                                                                                                                 
===== NODE GROUP =====                                                                                                                                                                                                                                                            
(3) ms-be[2067,2087].codfw.wmnet,ms-fe2015.codfw.wmnet                                                                                                                                                                                                                            
----- OUTPUT for command #1: 'rg a94b06fe-2eb0... | wc -l || true' -----                                                                                                                                                                                                          
2                                                                                                                                                                                                                                                                                 
===== NODE GROUP =====                                                                                                                                                                                                                                                            
(35) ms-be[2062,2066,2068-2073,2076-2082,2085-2086,2088-2089,2092,2094].codfw.wmnet,ms-fe[2009,2011-2014,2016-2024].codfw.wmnet                                                                                                                                                   
----- OUTPUT for command #1: 'rg a94b06fe-2eb0... | wc -l || true' -----                                                                                                                                                                                                          
0                                                                                                                                                                                                                                                                                 
================

BTW -- here are two canned queries for distributed traces of uploads: https://w.wiki/LSLK and https://w.wiki/LSLn
What Jaeger sees only includes the requests from MediaWiki towards ms-fe hosts, of course -- but you can use the x-request-id from the tags to pivot into logs.
I've no idea if Swift can omit OTel, or if it could use a proxy that could.

Change #1281723 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] shellbox-media: Add envoy listeners for swift

https://gerrit.wikimedia.org/r/1281723

Change #1281723 merged by jenkins-bot:

[operations/deployment-charts@master] shellbox-media: Add envoy listeners for swift

https://gerrit.wikimedia.org/r/1281723