Change 936368 merged by jenkins-bot:

[mediawiki/core@master] filebackend: Include truncated http body for 502 on SwiftFileBackend

https://gerrit.wikimedia.org/r/936368

There is a (small) spike in grafana about 502 errors (and about 504 errors), see https://grafana.wikimedia.org/d/OPgmB1Eiz/swift?orgId=1&viewPanel=37&from=1693494000000&to=1693501200000

Maybe new logs with the added extra information are already collected.

I think this is not linked to the error rate but it makes the need for a fix more urgent. The Wiki Loves Monuments contests started today.

Yesterday, while uploading a small SVG file using bigChunkedUpload.js, the finalize stage took an unusually long time for such a small file and then logged 00019: FAILED: backend-fail-internal: An unknown error occurred in storage backend "local-swift-codfw". The file did seem to finish upload as expected despite this though.

Today, reuploading (overwriting) a small PNG, the finalize stage took a similarly unexpectedly long time, but no error message was logged. Afterwards, however, the thumbnails for the file semi-randomly fail to update. Main thumb on file description page doesn't update; mini-thumb in the upload history list does update; requesting the image thumb at any size using image markup from a client wiki (enWS) returns the old thumb but squeezed into the new narrower width. Various voodoo poking (action=purge etc.) have failed to unwedge it.

These are new after a relatively long period of no real problems experienced with uploads. Timing wise it could roughly correspond with the spiking error rates @Umherirrender notes above, but as these were my only two uploads in the time period that association is very weak at best.

I restarted the swift frontends, and that has returned the error rate to the usual low levels.

This happens again repeatedly with not so big files (around 100 MB):

Chrome console:

mwbot> internal_api_error_UploadChunkFileException: [fab58ddb-22c9-4677-adfc-786416bf388a] Caught exception of type UploadChunkFileException  {"error":{"code":"internal_api_error_UploadChunkFileException","info":"[fab58ddb-22c9-4677-adfc-786416bf388a] Caught exception of type UploadChunkFileException","errorclass":"UploadChunkFileException"},"servedby":"mw2437"}

while uploading https://commons.wikimedia.org/wiki/File:Swiss_national_map,_39_Fl%C3%BCelapass.jpg

I've spent some time checking, and nothing that looks like that filename appears in the swift proxy logs today (and the error rates look as low as normal in general).

Now I get

00411: FAILED: stashfailed: An unknown error occurred in storage backend "local-swift-eqiad"

[I'm afraid my previous comment still applies - nothing relating to a file named like this appears in any of the swift frontend logs]

Four files repeatedly failed to upload today:

• https://commons.wikimedia.org/wiki/File:Swiss_national_map,_39_Fl%C3%BCelapass.jpg
• https://commons.wikimedia.org/wiki/File:Swiss_national_map,_42_Oberwallis.jpg
• https://commons.wikimedia.org/wiki/File:Swiss_national_map,_45_Haute-Savoie.jpg
• https://commons.wikimedia.org/wiki/File:Swiss_national_map,_47_Monte_Rosa.jpg

It seems some of these files are available in Special:UploadStash, but I get T348733 while trying to publish them.

Hi, just wanted to let you guys know I'm getting this exact same error trying to upload a new revision to https://commons.wikimedia.org/wiki/File:The_Sea_Beast_(1926).webm, file size ~ 385 mb

Specific error: 00172: 44/84> FAILED: internal_api_error_UploadChunkFileException: [c7e1fd5a-e9e6-4d21-8a84-e0cb0d31d9f7] Caught exception of type UploadChunkFileException

On https://commons.wikimedia.org/wiki/File:Swiss_national_map,_42_Oberwallis.jpg I still get for every try

00293: FAILED: internal_api_error_UploadChunkFileException: [c995f30e-54b1-490d-b829-ec4c4924acc0] Caught exception of type UploadChunkFileException

In T328872#9249487, @Yann wrote:

On https://commons.wikimedia.org/wiki/File:Swiss_national_map,_42_Oberwallis.jpg I still get for every try

00293: FAILED: internal_api_error_UploadChunkFileException: [c995f30e-54b1-490d-b829-ec4c4924acc0] Caught exception of type UploadChunkFileException

Oct 13, 2023 @ 11:19:46.595 ERROR exception mw2295 commonswiki [c995f30e-54b1-490d-b829-ec4c4924acc0] /w/api.php UploadChunkFileException: Error storing file in '/tmp/phpUeTKDd': backend-fail-internal; local-swift-eqiad
Oct 13, 2023 @ 11:19:46.594 ERROR FileOperation mw2295 commonswiki HTTP 401 (Unauthorized) in 'SwiftFileBackend::doStoreInternal' (given '{"async":false,"src":"/tmp/phpUeTKDd","dst":"mwstore://local-swift-eqiad/local-temp/9/90/1af4kf8yow0c.in02i2.1.jpg.13","overwrite":true,"headers":[]}')
Oct 13, 2023 @ 11:19:46.594 ERROR FileOperation mw2295 commonswiki StoreFileOp failed: {"src":"/tmp/phpUeTKDd","dst":"mwstore://local-swift-eqiad/local-temp/9/90/1af4kf8yow0c.in02i2.1.jpg.13","overwrite":true,"headers":[],"failedAction":"attempt"}

So in this case a 401. That's pretty interesting.

In T328872#9249624, @TheDJ wrote:

In T328872#9249487, @Yann wrote:

On https://commons.wikimedia.org/wiki/File:Swiss_national_map,_42_Oberwallis.jpg I still get for every try

00293: FAILED: internal_api_error_UploadChunkFileException: [c995f30e-54b1-490d-b829-ec4c4924acc0] Caught exception of type UploadChunkFileException

Oct 13, 2023 @ 11:19:46.595 ERROR exception mw2295 commonswiki [c995f30e-54b1-490d-b829-ec4c4924acc0] /w/api.php UploadChunkFileException: Error storing file in '/tmp/phpUeTKDd': backend-fail-internal; local-swift-eqiad
Oct 13, 2023 @ 11:19:46.594 ERROR FileOperation mw2295 commonswiki HTTP 401 (Unauthorized) in 'SwiftFileBackend::doStoreInternal' (given '{"async":false,"src":"/tmp/phpUeTKDd","dst":"mwstore://local-swift-eqiad/local-temp/9/90/1af4kf8yow0c.in02i2.1.jpg.13","overwrite":true,"headers":[]}')
Oct 13, 2023 @ 11:19:46.594 ERROR FileOperation mw2295 commonswiki StoreFileOp failed: {"src":"/tmp/phpUeTKDd","dst":"mwstore://local-swift-eqiad/local-temp/9/90/1af4kf8yow0c.in02i2.1.jpg.13","overwrite":true,"headers":[],"failedAction":"attempt"}

So in this case a 401. That's pretty interesting.

As SwiftFileBackend::doStoreInternal is mention in the error, this case seems also handled in T206252: Spike of HTTP errors from SwiftFileBackend::doStoreInternal

For the 401 there are information under T228292#7490101 (Where the task description shows the same stack trace as this task description)

I'm doing a bulk upload from LIghtroom this eve and am running into "The MediaWiki error backend-fail-internal occured: An unknown error occurred in storage backend "local-swift-eqiad"." again.

While I managed to upload the files mentioned on Wed, Oct 11, 7:28 PM after many attempts, I still get this error, now with

• https://commons.wikimedia.org/wiki/File:Carte_d%27%C3%89tat-major_de_la_France,_Feuille_Sainte-Affrique_S.O.jpg
• https://commons.wikimedia.org/wiki/File:Carte_d%27%C3%89tat-major_de_la_France,_Feuille_Sainte-Affrique_S.E.jpg
• https://commons.wikimedia.org/wiki/File:Carte_d%27%C3%89tat-major_de_la_France,_Feuille_Sainte-Affrique_N.O.jpg

with the following messages, respectively:

00156: FAILED: internal_api_error_UploadChunkFileException: [0fa38f19-d4b2-49d4-b858-134dc09a519a] Caught exception of type UploadChunkFileException

00213: FAILED: internal_api_error_UploadChunkFileException: [500587b3-7b15-4621-b8a1-8416b1110235] Caught exception of type UploadChunkFileException

00318: FAILED: stashfailed: An unknown error occurred in storage backend "local-swift-eqiad".

Since the cause is not the same old failure of the proxy servers, maybe it would improve clarity if we discussed this incident on a separate task.

What I'm seeing is a total cessation of successful delete operations and a flood of MW log messages showing 401 status codes for various actions.

In T328872#9263825, @tstarling wrote:

Since the cause is not the same old failure of the proxy servers, maybe it would improve clarity if we discussed this incident on a separate task.

What I'm seeing is a total cessation of successful delete operations and a flood of MW log messages showing 401 status codes for various actions.

I don't understand what you mean. I can't say if this is related to T349127, but I got this error and T341007 in quick succession while uploading files. Now I managed to upload the files mentioned above after ~10 failed attempts. I can't say if this issue is fixed, or if I just got lucky.

I wonder if the auth token just expired while the combined file was being uploaded (but after the PUT started). By the time the chunk deletions are issued, perhaps the token expired. AFAIK, the tokens should last for a day and MediaWiki app servers only reuse them for ~15 minutes (which used to be higher, but they seemed to expire too soon). It would be interesting to see the memcached stats for the token storage on the swift servers (maybe there is eviction pressure for some reason, like some service never reusing tokens and thus flooding memcached).

You can see metrics about swift's memcached usage in grafana e.g.; I've previously looked to see if incidents (i.e. raised 5xx levels) correlated with events in memcached, but haven't found anything. There is one very hot memcached key (I think it's account_info cf headers_to_account_info in the swift source code); this is probably not ideal, but I'm not sure if it's a swift bug or just the way the system is expected to work (and I've no evidence to suggest that this is causing a problem).

index.php?action=raw&ctype=text/javascript&title=User:Rillke/MwJSBot.js&_=2:1136 mwbot> stashfailed: An unknown error occurred in storage backend "local-swift-codfw".

while uploading https://commons.wikimedia.org/wiki/File:Kaibab_trail_at_Grand_Canfon_National_Park.jpg

I am currently again experiencing lots of different error messages during file upload.

{'error': {'code': 'backend-fail-internal', 'info': 'An unknown error occurred in storage backend "local-swift-eqiad".', 'stasherrors': [{'message': 'uploadstash-exception', 'params': ['UploadStashBadPathException', 'Der Pfad ist nicht vorhanden.'], 'code': 'uploadstash-exception', 'type': 'error'}], '*': 'See https://commons.wikimedia.org/w/api.php for API usage. Subscribe to the mediawiki-api-announce mailing list at <https://lists.wikimedia.org/postorius/lists/mediawiki-api-announce.lists.wikimedia.org/> for notice of API deprecations and breaking changes.'}, 'servedby': 'mw-api-ext.eqiad.main-6464cf4f7-qkxkk'}

{'error': {'code': 'backend-fail-internal', 'info': 'An unknown error occurred in storage backend "local-swift-eqiad".', 'stasherrors': [{'message': 'uploadstash-exception', 'params': ['UploadStashBadPathException', 'Der Pfad ist nicht vorhanden.'], 'code': 'uploadstash-exception', 'type': 'error'}], '*': 'See https://commons.wikimedia.org/w/api.php for API usage. Subscribe to the mediawiki-api-announce mailing list at <https://lists.wikimedia.org/postorius/lists/mediawiki-api-announce.lists.wikimedia.org/> for notice of API deprecations and breaking changes.'}, 'servedby': 'mw-api-ext.eqiad.main-6464cf4f7-gb7xw'}

{'error': {'code': 'uploadstash-bad-path', 'info': "Path doesn't exist.", '*': 'See https://commons.wikimedia.org/w/api.php for API usage. Subscribe to the mediawiki-api-announce mailing list at <https://lists.wikimedia.org/postorius/lists/mediawiki-api-announce.lists.wikimedia.org/> for notice of API deprecations and breaking changes.'}, 'servedby': 'mw-api-ext.eqiad.main-6464cf4f7-cm6g8'}

{'error': {'code': 'uploadstash-exception', 'info': 'Could not store upload in the stash (UploadStashFileException): "Im Speicher-Backend „local-swift-eqiad“ ist ein unbekannter Fehler aufgetreten.".', '*': 'See https://commons.wikimedia.org/w/api.php for API usage. Subscribe to the mediawiki-api-announce mailing list at <https://lists.wikimedia.org/postorius/lists/mediawiki-api-announce.lists.wikimedia.org/> for notice of API deprecations and breaking changes.'}, 'servedby': 'mw-api-ext.eqiad.main-6464cf4f7-8ptjp'}

Also getting errors while uploading:

The MediaWiki error backend-fail-stat occured: Could not read the status of file "mwstore://local-multiwrite/local-public/archive/9/96/20240705164303!At_Swindon_Steam_Railway_Museum_2024_176.jpg".
The MediaWiki error backend-fail-stat occured: Could not read the status of file "mwstore://local-multiwrite/local-public/archive/3/3a/20240705173523!At_Swindon_Steam_Railway_Museum_2024_206.jpg".

https://logstash.wikimedia.org/app/dashboards#/view/AXFV7JE83bOlOASGccsT?_g=h@bbef750&_a=h@2ee6f1d
https://grafana-rw.wikimedia.org/d/OPgmB1Eiz/swift?forceLogin=&from=1719294248878&orgId=1&to=1720203381602&var-site=All&viewPanel=37

Big rise in 503s over the last days, and really taking off today.
Created T369388 for this specific occurrence of the problem

Uploads have been going much better today, thank you to whoever fixed this. :-)

This happened again while trying to upload a new version of https://commons.wikimedia.org/wiki/File:The_Lion_of_the_Moguls_-_Le_Lion_des_Mogols_(1924)_dir._Jean_Epstein.webm

04369: finalize/189> Still waiting for server to publish uploaded file
04374: FAILED: stashfailed: An unknown error occurred in storage backend "local-swift-codfw".

Thank you for reporting @Yann. I created T382705 for this one.

I am currently getting the following error for around the half of all my upload attempts.

{'error': {'code': 'lockmanager-fail-conflict', 'info': 'Could not acquire lock. Somebody else is doing something to this file.', 'filekey': '1biglw5ddlcc.q7c9ff.6579311.jpg', 'sessionkey': '1biglw5ddlcc.q7c9ff.6579311.jpg', 'docref': 'See https://commons.wikimedia.org/w/api.php for API usage. Subscribe to the mediawiki-api-announce mailing list at <https://lists.wikimedia.org/postorius/lists/mediawiki-api-announce.lists.wikimedia.org/> for notice of API deprecations and breaking changes.'}, 'servedby': 'mw-api-ext.codfw.main-74795b8fcc-r9ftw'}.

"The MediaWiki error backend-fail-internal occured: An unknown error occurred in storage backend "local-swift-eqiad"." ...

In T328872#10689851, @Mike_Peel wrote:

"The MediaWiki error backend-fail-internal occured: An unknown error occurred in storage backend "local-swift-eqiad"." ...

We had a couple of incidents with swift over the weekend (including about the time you posted this comment).

I'm not sure where would be a good place to put this but I think I found something weird with how uploads work. I tried uploading a very small svg file in testwiki with excimer. 857ms was spent in the swift area but very little of it to do the upload. Let me break it down: https://performance.wikimedia.org/excimer/profile/03c4168ac24eb7eb

• First column: First it tries to get the path for the file, which triggers a fileExists() which triggers a getFileStats(), Since this is multi-backend, I assume this is doing a cross-DC connection cause it's taking 94ms to just respond.
• Then it tries to do a consistencyCheck (why? The file doesn't even exist yet?) which triggers a preLoadFileStats() call which obviously makes a new cross-DC connection taking 410ms(!) to load the file information (for something that's not even uploaded yet?) but at least since it's preLoading the information it won't make the exact same call again. Right?
• In the third column, where the actual storage of the file happens, preLoadFileStats gets called twice, I think the first time is in the primary dc and the second time in the secondary dc, each call is followed by actually storing the file. The calls for uploading takes 160ms in total but getting stats takes 130ms extra on top
• The last column is another extra call to fileExists() which takes another 40ms. The reason behind this expensive call is to determine whether it's an "archived" file or a "new" file.

That way, an upload operation that supposed to take 160ms gets stretched to 850ms by adding eight extra swift calls (four of which go across the United States and come back) on top of two to store the files. Probably we can't get rid of all of them but this could be made more efficient.

I also build a flame graph of upload paths from the daily log of May 30. Here it is:
https://people.wikimedia.org/~ladsgroup/upload.svg?x=237.7&y=981

you can see the same issue in the flame graph as well.

I think removing some of these calls could reduce the load on the frontend proxies and definitely reduce the time to store a file and reduce the chance of one of the calls failing and causing issues.

Apologies if I missed something obvious, I'm new to this codebase and it's not the easiest part of mediawiki to understand.

I think the "check the file is in a consistent (presumed-to-be-absent) state" operation is intentional, and probably replicated across other file changing call paths; not least because we get tickets sometimes when these checks fail...

I presume this is/was an intentional decision that if the two backends are in an inconsistent state (with each other, or with MW's idea of what that state should be) then we leave it for an operator to fix rather than just going ahead and replacing that inconsistent-state with whatever the user was trying to do at the time.

[I am even less familiar with the mediawiki codebase...]

Okay, I made deeper investigation. I uploaded a random file on verbose mode and here is the result:

• It made more than 30 HTTP requests to swift: https://logstash.wikimedia.org/goto/8da6c87a8ed733b9a384b238e070e66e out of them, 14 are to the remote datacenter: https://logstash.wikimedia.org/goto/4061ac5482debab31857cf7d1894d04f
• It is clearly making the same requests multiple times back to back. For example, look at 14 calls to remote datacenter. HEAD (which is called from ::getContainerStat() or ::doGetFileStatMulti()) has been called on the same path back to back before the calls to PUT (there are more after it)
  • e.g. https://logstash.wikimedia.org/app/dashboards#/doc/logstash-*/logstash-mediawiki-1-7.0.0-1-2025.06.05?id=IWR2PZcBeu0zMOTS9dTJ and https://logstash.wikimedia.org/app/dashboards#/doc/logstash-*/logstash-mediawiki-1-7.0.0-1-2025.06.05?id=gtp2PZcBc9IpmxE59lGF are duplicates of each other
  • And https://logstash.wikimedia.org/app/dashboards#/doc/logstash-*/logstash-mediawiki-1-7.0.0-1-2025.06.05?id=5WR2PZcBeu0zMOTS9NKi and https://logstash.wikimedia.org/app/dashboards#/doc/logstash-*/logstash-mediawiki-1-7.0.0-1-2025.06.05?id=g9p2PZcBc9IpmxE59lGF also duplicate of each other
  • And this is just the remote dc, the local dc is the same story.
• So there are two major issues:
  • FileBackendMultiWrite::getFileStat() checks for read backend which with the option of readAffinity set in production, it should only read from the local swift. This is clearly not happening.
  • It is making the same expensive call again and again.

The reason for that seems to be calls to ::preloadFileStat(). For example here in ::consistencyCheck():

		// Preload all of the stat info in as few round trips as possible
		foreach ( $this->backends as $backend ) {
			$realPaths = $this->substPaths( $paths, $backend );
			$backend->preloadFileStat( [ 'srcs' => $realPaths, 'latest' => true ] );
		}

Ironically, it explicitly does this to save round trips and reduce calls (see the comment).

Here is the problem: Every time ::preloadFileStat() is called, the function doesn't care whether the information is already preLoaded or not, it (re-)preLoads the information again and again with no respect to the existing cache (see FileBackendStore::preloadFileStat()). It is recommended to call this function in comments (e.g. Ideally, the file stat entry should already be preloaded via preloadFileStat(). comment) and many parts of upload code path follow the guideline and call this method and as you can see, in the above excimer profile, the same call to preLoad (and then to swift) has been seen four times, just fixing this can easily brush off a decent chunk of upload time of small files and reduce the cross-dc communication of the appservers. There are currently ~60 HEAD reqs/s to swift: https://grafana.wikimedia.org/d/OPgmB1Eiz/swift?orgId=1&from=now-2d&to=now-1m&timezone=utc&var-site=$__all&viewPanel=panel-10 that can get much lower.

Of course I will debug more and see what other improvements I can do to make the system a bit more stable.

The idea of preloadFileStat() was to allow concurrent HEAD requests to a list of objects after an relevant locks were acquired. If no locks are acquired, and "latest" is not set, maybe reusing prior loaded state entries is OK. From the perspective of FileBackend, it's mostly thinking that you call doOperations or doQuickOperations, which is supposed to do one preload (within any locking) and is done. The FileBackendMultiWrite class (itself a hack due to not having a proper regional swift cluster and swift-repl only able to do periodic reconciliation) also has to write to the remote backend and has consistency checks turned on...doing a preloads of local and remote backend. It also has to repeat the write operation on the remote backend, requiring another preload to the remote. Since FileBackendMultiWrite does it's own locking, it seems like a lot of these 'stat' entries could be reused instead of reloaded.

Another matter is FileRepo batches doing a lot of getFileStat() checks before calling doOperations(), so that's more HEAD requests since it's before the other preloads. If the batch operation locks the paths, then the FileBackendMultiWrite/FileBackendStore preloads don't need to reload over existing stuff.

Generally, if 'stat' entry in the in-memory fileStatCache was loaded after the outermost lock of that path (since locks can nest), it should just magically reuse existing values sufficient for the $latest (as done elsewhere). Of course, the FileBackendStore doesn't know about the FileBackendMultiWrite locks that FileRepo acquired...this would be easy to implement if not for FileBackendMultiWrite. Maybe a preloadStatCache() could have a $knownLockedPaths array with FileBackendMultiWrite could use (with substPaths() of course). Kind of ugly though.

I understand the need to have multi write backends and doing all write operations in both dcs but from that it requires a massive leap to require practically every operation be replicated in both dcs. For example, why doing consistency check in both dcs? let's say something is corrupted in between these two swift instances, the chance of the swift reconciliation script actually finding it or overwriting the secondary dc is much higher than someone accidentally deciding to upload a new version and then getting broken. i.e. I think mediawiki at the moment should be responsible for double uploads (and other write operations) but it shouldn't try to do integrity checks of two swift clusters (doubly so during upload). To me it's like mediawiki checking primary database and a replica for data integrity during page reads, worse than that even. It tries to do that while the replica is thousands of kilometers away. File backend shouldn't do the work of the infrastructure at run time.

I don't understand discussion around batching, most uploads don't batch. I'm not seeing any batching or performance gain from batching in graphs. Can you point me to some data that batching has any effect?

I rather just remove preLoad altogether. It demonstrably makes things faster and makes the logic much simpler.

Change #1170709 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/core@master] filebackend: Remove consistency check for multi-backend

https://gerrit.wikimedia.org/r/1170709

In T328872#10889545, @Ladsgroup wrote:

I understand the need to have multi write backends and doing all write operations in both dcs but from that it requires a massive leap to require practically every operation be replicated in both dcs. For example, why doing consistency check in both dcs? let's say something is corrupted in between these two swift instances, the chance of the swift reconciliation script actually finding it or overwriting the secondary dc is much higher than someone accidentally deciding to upload a new version and then getting broken. i.e. I think mediawiki at the moment should be responsible for double uploads (and other write operations) but it shouldn't try to do integrity checks of two swift clusters (doubly so during upload). To me it's like mediawiki checking primary database and a replica for data integrity during page reads, worse than that even. It tries to do that while the replica is thousands of kilometers away. File backend shouldn't do the work of the infrastructure at run time.

One reason for the consistency checks is to quarantine object paths that have uncertain values (inconsistent between the DC clusters). The effect of operations is different if the starting state of the files is already different. If we allow more operations, the uncertainty could spread to other object paths. Another reason was to trigger the autoResync logic to fix things and unblock the operation. This was initially before swift-repl existed. When swiftrepl was new, it was also useful for unblocking user operations if didn't get around to them fixing something yet. This might involve users retrying things like re-upload/move/delete/restore *soon* after a failure. AFAIK, swift-repl used to be slow and sometimes was configured to handle excess file deletion and other times not. There was once a time when 'autoResync' was just true (handling object deletions) and swift-repl deletions were not enabled.

Anyway, I see that "autoResync" is "conservative", so it's not able to sync a list of files in a way that swift-repl cannot. Indeed most actual resyncing will be from swift-repl since it doesn't need someone to happen to try to touch the files again. I think we can just lean into the assumption that whatever is in the primary is "canon" (FileBackendMultiwrite::doOperations already does this with it's Status result) and let swift-repl make that secondary cluster match up soon if something failed.

I don't understand discussion around batching, most uploads don't batch. I'm not seeing any batching or performance gain from batching in graphs. Can you point me to some data that batching has any effect?

By batching, I just mean anything that passes 2+ operations to doOperations(). File delete/restore can involve batches with operations proportionate to the number of file versions. Every path gets a HEAD operation to check the preconditions of the operations beforehand. The actual COPY/DELETE/POST/PUT operations of use concurrency and the use of preloadFileCache() in doOperations() does the same for the HEAD requests.

I rather just remove preLoad altogether. It demonstrably makes things faster and makes the logic much simpler.

Disabling consistencyCheck() would speed things up half the way. Setting "syncChecks" to 0 in config should produce immediate results.

External callers of preloadFileCache(), basically all but the one in doOperations(), should probably get "keep existing recent cache entries" behavior. Nevertheless, the cache entries would still not be reused within doOperations() without further changes. Doing that correctly would require that things calling FileRepo::fileExistsBatch() lock the paths first and use "latest" (none of the LocalFile*Batch classes do either). In addition, FileBackend would also have to only reuse cache entries on locked paths, tagged "latest", that were fetched after the lock was acquired. Getting FileBackendMultiwrite, which takes over the locking, to cooperate would involve some more pain.

I also don't understand why LocalFile*Batch even need to call removeNonexistentFiles() ->fileExistsBatch() -> preloadFileCache(). It seems like code could be simplified to use "ignoreMissingSource", which would also knock out the HEAD request spam.

"swift-repl" (it's not actually that any more, but something based on rclone) runs only weekly (on Monday Europe-morning).

Change #1170709 merged by jenkins-bot:

[mediawiki/core@master] filebackend: Remove consistency check for multi-backend

https://gerrit.wikimedia.org/r/1170709

Change #1194781 had a related patch set uploaded (by Aaron Schulz; author: Aaron Schulz):

[mediawiki/core@master] filebackend: remove accessibility check from multi-backend

https://gerrit.wikimedia.org/r/1194781

Change #1194781 merged by jenkins-bot:

[mediawiki/core@master] filebackend: remove accessibility check from multi-backend

https://gerrit.wikimedia.org/r/1194781

Change #1196018 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/core@wmf/1.45.0-wmf.22] filebackend: Remove consistency check for multi-backend

https://gerrit.wikimedia.org/r/1196018

Change #1196018 merged by jenkins-bot:

[mediawiki/core@wmf/1.45.0-wmf.22] filebackend: Remove consistency check for multi-backend

https://gerrit.wikimedia.org/r/1196018

Mentioned in SAL (#wikimedia-operations) [2025-10-14T11:54:46Z] <ladsgroup@deploy2002> Started scap sync-world: Backport for [[gerrit:1196018|filebackend: Remove consistency check for multi-backend (T328872)]]

Mentioned in SAL (#wikimedia-operations) [2025-10-14T11:59:00Z] <ladsgroup@deploy2002> ladsgroup: Backport for [[gerrit:1196018|filebackend: Remove consistency check for multi-backend (T328872)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Mentioned in SAL (#wikimedia-operations) [2025-10-14T12:07:32Z] <ladsgroup@deploy2002> Finished scap sync-world: Backport for [[gerrit:1196018|filebackend: Remove consistency check for multi-backend (T328872)]] (duration: 12m 46s)

In T328872#10873913, @Ladsgroup wrote:

I'm not sure where would be a good place to put this but I think I found something weird with how uploads work. I tried uploading a very small svg file in testwiki with excimer. 857ms was spent in the swift area but very little of it to do the upload. Let me break it down: https://performance.wikimedia.org/excimer/profile/03c4168ac24eb7eb

I uploaded a tiny SVG and profiled it: https://performance.wikimedia.org/excimer/profile/c2c0fe4101a6962b . It's better, but still slowish.

Change #1235490 had a related patch set uploaded (by Func; author: Func):

[mediawiki/core@master] FileBackend: Clean up unused private constants

https://gerrit.wikimedia.org/r/1235490

Change #1235491 had a related patch set uploaded (by Func; author: Func):

[operations/mediawiki-config@master] filebackend: Clean up removed config params for multi-write backends

https://gerrit.wikimedia.org/r/1235491

Change #1235490 merged by jenkins-bot:

[mediawiki/core@master] FileBackend: Clean up unused private constants

https://gerrit.wikimedia.org/r/1235490

Change #1235491 merged by jenkins-bot:

[operations/mediawiki-config@master] filebackend: Clean up removed config params for multi-write backends

https://gerrit.wikimedia.org/r/1235491

Mentioned in SAL (#wikimedia-operations) [2026-02-02T21:05:24Z] <kemayo@deploy2002> Started scap sync-world: Backport for [[gerrit:1235392|Edit check: turn off the tone a/b test on frwiki, jawiki, ptwiki (T411914)]], [[gerrit:1235111|Enable suggestions BetaFeature on beta wikis (T415504)]], [[gerrit:1230462|WikimediaCustomizations: Set WMCBadEmailDomainsFile (T397244)]], [[gerrit:1235491|filebackend: Clean up removed config params for multi-write backends (T328872)]]

Mentioned in SAL (#wikimedia-operations) [2026-02-02T21:07:21Z] <kemayo@deploy2002> tgr, func, kemayo, esanders: Backport for [[gerrit:1235392|Edit check: turn off the tone a/b test on frwiki, jawiki, ptwiki (T411914)]], [[gerrit:1235111|Enable suggestions BetaFeature on beta wikis (T415504)]], [[gerrit:1230462|WikimediaCustomizations: Set WMCBadEmailDomainsFile (T397244)]], [[gerrit:1235491|filebackend: Clean up removed config params for multi-write backends (T328872)]] synced to

Mentioned in SAL (#wikimedia-operations) [2026-02-02T21:16:18Z] <kemayo@deploy2002> Finished scap sync-world: Backport for [[gerrit:1235392|Edit check: turn off the tone a/b test on frwiki, jawiki, ptwiki (T411914)]], [[gerrit:1235111|Enable suggestions BetaFeature on beta wikis (T415504)]], [[gerrit:1230462|WikimediaCustomizations: Set WMCBadEmailDomainsFile (T397244)]], [[gerrit:1235491|filebackend: Clean up removed config params for multi-write backends (T328872)]] (duration: 10