SVG is like HTML - there are dangerous parts but its not that big a deal. We allow <div> in wikitext but not <script>, we could do the same thing with svg potentially.

But if we don't find a replacement my assessment is that we will eventually end up in the same situation with timelines as we currently are with graphs.

Maybe, but i would argue a new task would be better, when and if that happens.

my understanding is that this worked after retrying.

Seems like this is fixed! We just had File:OBR_Hafignnover_6-2024.webm (4.88 GB) successfully uploaded (req-id: c1a5aab7-6e6b-4ef5-b20d-f9ddb095577f ). The job took 5 minutes 20 seconds to complete, so went beyond the previous 202 second limit.

Tagging for 1.39 release because there was a request on matrix to backport to REL1_39 (As in make mw core rel1_39 pull in a fixed version of library). I'll leave it to release managers if they want to do it.

per T358308 - one of the major causes of upload failure for medium-large files, is that during a deploy the job runners get killed. Making it retryable would fix that issue.

Can you try uploading again. The issue might be fixed now

Yes, this is due to T358308. Until that is fixed, you can upload large (>3GB) files via https://commons.wikimedia.org/wiki/Help:Server_side_upload

Ok, i did some more digging in logstash.

An interesting thing about this, is that it gives way more dynamic programmability to CSS. Since previously the CSS pages were static pages you had to edit to change. With variables, the variable can be set inline, so it can be set via lua, or in page previews without save, or even as a result of just a cross-domain GET request.

In T334372#9711759, @Tacsipacsi wrote:

Why would be the third one “by far” the most useful? I’d think most use cases can be satisfied by either of the first two, especially if TemplateStyles is allowed within the SVGs. Of course, there are some cases that can be satisfied only by the third option (say, using CSS variables defined in wikitext), but they aren’t that important.

Looking at recent logs, it does seem like some of the failures might be caused by things other than a timeout (There was one recently that failed after 17 seconds). However i still think the larger files are failing due to some sort of timeout as they tend to fail at around 202 second mark, which is pretty suspicious. Sample size is pretty low since users know large uploads don't work so they don't try and upload large files.

I think the better solution would be to just make MW generate the rel link. HTML should not be in messages, but that doesn't mean we shouldn't use it all.

Clearly this is no longer going to happen.

I used to be opposed to this, but I am now more a fan.

In T358308#9696868, @hnowlan wrote:

Kubernetes jobrunners were using the default max_execution_time of 180s, which might be the culprit for these behaviours. I've bumped it to match the metal jobrunners.

Existing installs arent automatically converted, but i dont think we want that anyways.

Anyone with write access to MediaWiki:Sidebar also can do what they want with MediaWiki:Common.js so it looks like this bug is not very easy to exploit.

lgtm

Looking at a recent example, it seems like the change didn't take affect (Or i was wrong about the cause).

Mostly i was looking at how many uploads get to this stage but don't get past it. I was looking at https://logstash.wikimedia.org/goto/f299fd6b01197908797af218435bd927 and comparing revieved last chunk vs chunk recombined messages.

In T361435#9675012, @bd808 wrote:

In T361435#9674842, @Bawolff wrote:

If its being moved anyways, would be kind of cool if it was sub-domain based.

Would a scheme like $TOOL.static.toolforge.org provide the desired in-browser separation or would we need $TOOL.$SOMETHING.$TLD to get good results?

lgtm :)

If its being moved anyways, would be kind of cool if it was sub-domain based.

The proposed use-case of the format is for storing the editable source-code-esque version of a layered raster image. Its not clear if anyone is using it for that, but even if they were, i would not expect them to have very many views. That use case doesn't even require thumbnailing support.

Personally i was a fan of <source>....

Even if WMF was planning to do this (i have no idea if they are) it would probably not be a top level OKR.

As an aside, i had no idea this overlapped the toolbox on new vector. Guess i should be testing on that skin every now and then

Radiopedia is using lower resolution images (At least in the linked example, they seem to be using images about 555x630 px that seem to be about 20% of the size of the images used with the ImageStack example). If you go full screen mode you get bigger issues.

Graphs are a valuable feature. They're an important feature to a sizable portion of both the editor and reader communities.

The slide numbers sometimes present a nonsensical value like “1291/136”

Fixed

Images move too fast with up/down movement on trackpad or mouse

I slowed it down. However its possible that this might vary between different mice or browsers. Chrome seems to scroll a bit faster than firefox.

I've put my modified version on my test wiki at http://bawolff.net/monstranto/index.php/IMStack

The install was old but the file was new. I didn't really investigate further. Possibly it was a timedmediahandler issue as my test file was a mp4 video.

Try https://quarry.wmcloud.org/query/81208

SBOMs are great tools to detect supply chain attacks and mitigating them.

as I may be missing deleted

It should be noted there are some files on commons with too long names. See parent task.

So it sounds like the underlying problem, is max file name size is checked at upload but not during file rename (T359294).

we are at the more philosophical point of wondering at the point of it all, and what sort of failure analysis could avoid such events in the future.

T336595 feels like a band-aid solution and a bad one at that. It does nothing to adress the underlying security problem or maintenance problem. I personally don't think we should do that except as a last resort.

In T356436#9599232, @Rockingpenny4 wrote:

@Yaron_Koren I configured the InlineComments extension on my local and went through the documentation and found out that it uses ContentHandler , however I was not able to figure out how to navigate to it on my localhost. Is there any reference for that I can go through?

Its really only uploads > 1GB that have this issue, which is a very small portion of uploads. Maybe it would make sense to split AssembleUploadChunks job into a big and small version depending on how big the file is. The big version could be handled more similarly to how webVideoTranscode jobs are handled and the small version could be handled like a normal job. (Possibly have to do the same thing for PublishStashedFile, although much of the work in that job is reverifying the file which isn't strictly needed since it has already been done, and we could possibly just make it not do that, which might make it much faster for large files).

Its been about 5 days and no new examples. Previously there was about 2 a day. I'm going to call this fixed.

Also 2 of these are above 5GB which is our current max size.

object-src is probably not too useful in modern browsers now that <object> is just a glorified iframe. It probably makes sense to just always have that set to 'none'.

I guess PublishStashedFile could be further optimized if we didn't verify the file. In theory, the file should already have been verified when uploaded to stash. If we don't need to verify the file, then there is no need to download the file locally, and all operations can take place directly on swift. However maybe that's not worth it, and its better just to be paranoid and re-verify the file at all stages.

On discord, Sazpaimon suggested having chunked upload keep a running total of the SHA1 hash so far (e.g. using https://www.php.net/manual/en/function.hash-update.php ). That way MW never has to calculate the entire hash at one time, but instead the work is split over all the uploaded chunks.

When i was testing this, the new version did still calculate some SHA1's in maybeUpgradeRow in a deferredupdate. However that is unrelated to file backend, and possibly is something about my local install or the file type handler.

I agree with Luke.

This bug is too old and too lacking in details to be actionable. Possibly https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1007983 may help with this, but who knows.

Oh, i see that deployment-charts/charts/mediawiki/values.yaml has an upstream_timeout set to 180.0 seconds. I believe mw-jobrunners will inherit that. So presumably helmfile.d/services/mw-jobrunner/values.yaml should set upstream_timeout under mesh: to be 1203 seconds (to be 1 more second than the apache timeout). Or at least be closer to that. I'm unclear if it is expected that upstream_timeout & change-propagation timeout should be above the MW $wgRequestTimeout in order to capture all errors or if the HTTP connection should close before that point in order to trigger a client abort on the PHP side.

Reading config files, it seems like the timeout should already be about 7 minutes (change-propagation DEFAULT_REQUEST_TIMEOUT = 7 minutes. job runner $wgRequestTimeout = 20 minutes).

Just checking to see if this is still a thing. Most recent example seems to be https://commons.wikimedia.org/wiki/File:NDL801250_%E5%8B%A7%E6%A5%AD%E5%8A%9F%E7%B8%BE%E9%8C%B2_%E7%AC%AC2%E7%B7%A8_part1.pdf (Feb 29). There seems to be not a whole lot of older images, but possibly that is just because they get fixed or tagged as copyvio and deleted. There is a log entry for that file of an UploadStashFileNotFoundException exception. I'm not really sure how that could happen with the file becoming actually published. Its possible there were multiple publish attempts though and that is from a later one. I'm a bit surprised there are no other log messages. It used the non-async path for assembling, so presumably it was using that path for publishing too.

Well so far so good, but may be a bit early to tell so I will keep this bug open until a week has passed to make sure the issue is really fixed. The fix should be in place as of Feb 28 19:30 UTC. The only upload since then that is a multiple of 5mb is [[File:OR-126,_Knowles_Tunnel_-_53555943932.jpg]], which appears to be a false positive, as the source image is exactly 20MB. Prior to this we were having about 1 corrupted upload a day.

The easiest thing to do here would be to make sure the end user knows that they should strip location data before uploading. Apparently we already do this in UploadWizard but not other upload workflows. This is very much a case of patch-welcome if somebody wants to do that. It could be a warning box or note on the uploading workflow (perhaps during the upload itself "This contains location data are you sure you want to upload it?" for example.

We could also add a sandbox directive to the csp policy

In T350917#9571715, @gerritbot wrote:

Change 1005965 had a related patch set uploaded (by Brian Wolff; author: Brian Wolff):

[mediawiki/core@master] Make chunked upload delete chunks in deferred job

https://gerrit.wikimedia.org/r/1005965

I wonder if the slow part of the process is deleting all the chunks. Im not 100% sure, but as far as i can tell that is not paralelized. It would also help explain the errors being file missing, instead of running out of time in middle of concatenate. Its also not super important since they will eventually get deleted anyways, maybe we should rearrange things so code reports success before doing delete step.

Ok, so new logs do show a bunch of:

Doesn't help that AssembleUploadChunksJob.php takes the specific (and even localized!) error and replaces it with a super generic one. Maybe we should stop doing that.q

As an aside some historical discussion at https://static-codereview.wikimedia.org/MediaWiki/104687.html (for context, back in the day we used mysql for job tracking which probably made the code less risky at the time)

More recent example is File:Delft_Van_Miereveltlaan_7.jpg (aka 1aqdj7jclxmc.afcsnk.1553787.jpg aka 1aqdj6vvp0a8.a3pwzj.1553787.jpg ) which is cut off at 10MB.

In T350917#9358944, @MatthewVernon wrote:

Picking a recent failure:

mvernon@cumin1001:~$ sudo cumin -x --force --no-progress --no-color -o txt O:swift::proxy "zgrep -F '0/0e/Wikidata_43.jpg' /var/log/swift/proxy-access.log.3.gz" >~/junk/T350917.txt
#Cumin output elided
mvernon@cumin1001:~$ grep " PUT " junk/T350917.txt | grep 'public'
moss-fe2001.codfw.wmnet: Nov 24 19:12:12 moss-fe2001 proxy-server: 10.192.48.101 10.192.32.51 24/Nov/2023/19/12/12 PUT /v1/AUTH_mw/wikipedia-commons-local-public.0e/0/0e/Wikidata_43.jpg HTTP/1.0 201 - wikimedia/multi-http-client%20v1.1 AUTH_tk77cb529d3... 10485760 - 574193492001fdf36bdf02cbd36887a4 tx56ca15da11c2455281325-006560f58b - 0.2022 - - 1700853131.809281588 1700853132.011436224 0
ms-fe1013.eqiad.wmnet: Nov 24 19:12:12 ms-fe1013 proxy-server: 10.192.48.101 10.64.48.149 24/Nov/2023/19/12/12 PUT /v1/AUTH_mw/wikipedia-commons-local-public.0e/0/0e/Wikidata_43.jpg HTTP/1.0 201 - wikimedia/multi-http-client%20v1.1 AUTH_tk3f3803a6c... 10485760 - 574193492001fdf36bdf02cbd36887a4 tx460fd22327174e6ba0df8-006560f58c - 0.3835 - - 1700853132.315957069 1700853132.699444294 0

So you can see there are two successful PUTs (one to each ms cluster), both from mw2279.codfw.wmnet, of the same 10485760-byte object.

So from the swift point of view, it was offered a 10M object, and stored it as requested, so I think the problem lies further up the stack (uploadwizard was used, maybe it was confused by a client interruption or somesuch?)

In T200820#9567876, @MatthewVernon wrote:

Here's the relevant logs, sorted by time:

moss-fe2001.codfw.wmnet: Feb 21 20:19:14 moss-fe2001 proxy-server: 10.192.48.105 10.192.32.51 21/Feb/2024/20/19/14 PUT /v1/AUTH_mw/wikipedia-commons-local-temp.b8/b/b8/1aqaxe3dattw.u1suhi.6080484.webm.0 HTTP/1.0 201 - wikimedia/multi-http-client%20v1.1 AUTH_tk7b6ed208d... - - - tx948576b6a72e4600a62d1-0065d65ac1 - 0.9693 - - 1708546753.577275038 1708546754.546539783 -
ms-fe1009.eqiad.wmnet: Feb 21 20:19:16 ms-fe1009 proxy-server: 10.192.48.105 10.64.0.166 21/Feb/2024/20/19/16 PUT /v1/AUTH_mw/wikipedia-commons-local-temp.b8/b/b8/1aqaxe3dattw.u1suhi.6080484.webm.0 HTTP/1.0 201 - wikimedia/multi-http-client%20v1.1 AUTH_tke5beae87e... - - - tx4683f2a851d249ea89bf1-0065d65ac2 - 1.5437 - - 1708546754.563668013 1708546756.107330084 -
ms-fe2013.codfw.wmnet: Feb 21 20:59:24 ms-fe2013 proxy-server: 10.194.152.76 10.192.0.87 21/Feb/2024/20/59/24 GET /v1/AUTH_mw/wikipedia-commons-local-temp.b8/b/b8/1aqaxe3dattw.u1suhi.6080484.webm.0 HTTP/1.0 200 - wikimedia/multi-http-client%20v1.1 AUTH_tk7b6ed208d... - 97498542 - tx8b68cb8506d34d1a9bb03-0065d66420 - 12.5442 - - 1708549152.229032040 1708549164.773195744 0
ms-fe2013.codfw.wmnet: Feb 21 21:00:02 ms-fe2013 proxy-server: 10.194.152.76 10.192.0.87 21/Feb/2024/21/00/02 DELETE /v1/AUTH_mw/wikipedia-commons-local-temp.b8/b/b8/1aqaxe3dattw.u1suhi.6080484.webm.0 HTTP/1.0 204 - wikimedia/multi-http-client%20v1.1 AUTH_tk7b6ed208d... - - - txfabad3d0f009413ea82dd-0065d66452 - 0.0458 - - 1708549202.850069761 1708549202.895848513 0
ms-fe1009.eqiad.wmnet: Feb 21 21:00:03 ms-fe1009 proxy-server: 10.194.152.76 10.64.0.166 21/Feb/2024/21/00/03 DELETE /v1/AUTH_mw/wikipedia-commons-local-temp.b8/b/b8/1aqaxe3dattw.u1suhi.6080484.webm.0 HTTP/1.0 204 - wikimedia/multi-http-client%20v1.1 AUTH_tke5beae87e... - - - tx26b0b6c03f1b4577a65b7-0065d66453 - 0.0454 - - 1708549203.070249796 1708549203.115652323 0
ms-fe2013.codfw.wmnet: Feb 21 21:03:12 ms-fe2013 proxy-server: 10.194.155.232 10.192.0.87 21/Feb/2024/21/03/12 GET /v1/AUTH_mw/wikipedia-commons-local-temp.b8/b/b8/1aqaxe3dattw.u1suhi.6080484.webm.0 HTTP/1.0 404 - wikimedia/multi-http-client%20v1.1 AUTH_tk7b6ed208d... - 70 - tx36a819fdb39c4d61bb4ee-0065d66510 - 0.0335 - - 1708549392.151752949 1708549392.185259581 0
ms-fe2013.codfw.wmnet: Feb 21 21:03:12 ms-fe2013 proxy-server: 10.194.155.232 10.192.0.87 21/Feb/2024/21/03/12 GET /v1/AUTH_mw/wikipedia-commons-local-temp.b8/b/b8/1aqaxe3dattw.u1suhi.6080484.webm.0 HTTP/1.0 404 - wikimedia/multi-http-client%20v1.1 AUTH_tk7b6ed208d... - 70 - txfffc016f7af4438eac05b-0065d66510 - 0.0284 - - 1708549392.332885504 1708549392.361268997 0

From which, I see:

• First, 10.192.48.105 (mw2283) uploads to eqiad and codfw at 20:19
• Then at 20:59, 10.194.152.76 (I guess a k8s node?) does a successful GET
• almost immediately, it issues a DELETE from both eqiad and codfw, which succeed
• Finally, at 21:03, 10.194.155.232 (another k8s node?) tries two GETs, both of which return 404 (as you'd expect, the object has been deleted)