I ended up just reading the exiftool source code.

It would be useful to link to (or upload to this task) some example files with XMP. So far, I've noticed that image magick seems to diverge from the spec when creating webp files, so it would be good to have some examples from a variety of tools just to see what they do.

Why ICC? I don't think we would normally show that type of data on a file description page?

I have no idea if anyone is still around who knows the history of why this is the way it is. @Sannita does the current upload wizard team have any thoughts or objections to changing this?

Ah, that would be the problem. Async uploads are disabled on english Wikipedia - https://github.com/wikimedia/operations-mediawiki-config/blob/master/wmf-config/InitialiseSettings.php#L729-L735

Hmm, both of those errors on the server side indicate no async. So either something is wrong with the script or maybe enwiki somehow has async disabled

Also broke the techblog link on mediawiki.org

@hinnk How are you uploading these files? (Are you using UploadWizard, a specific user script, a bot program, something else?) I suspect this may be an issue with the tool being used to do uploads.

I think there are two separate tickets here:

So recently i've been trying to collect information on cases where large uploads fail in order to try and fix the underlying issues. Did you attempt to upload this to commons in the normal way and it didn't work? If so, please let me know the details.

I think if we did deliver the wrong thumbsize, it only makes sense to deliver one larger then the requested size. Browsers will scale it client side, and its always better to downscale then upscale. The performance degredation is probably be not that bad - the larger thumbs would only be shown to first person to view the page. A small slowdown to image viewing for a single user seems not that bad, and better then never loading the image at all.

Possibly the value "auto" should also be treated as "100%"

@Aklapper did you mean to direct this to @Halfak ?

There are no interesting changes between 643fb6c (what mdwiki uses) and current master of TextExtracts. So either its a configuration difference or a change in a different component

Are you reporting that this used to work but no longer does (In which case when was the last time it worked?), or has it always been this way and you are suggesting that the module should be changed to work as you describe?

the patch at https://pastebin.com/9wVWHEpa lgtm

Maybe should just be limited just to executable files

SVG is like HTML - there are dangerous parts but its not that big a deal. We allow <div> in wikitext but not <script>, we could do the same thing with svg potentially.

But if we don't find a replacement my assessment is that we will eventually end up in the same situation with timelines as we currently are with graphs.

Maybe, but i would argue a new task would be better, when and if that happens.

my understanding is that this worked after retrying.

Seems like this is fixed! We just had File:OBR_Hafignnover_6-2024.webm (4.88 GB) successfully uploaded (req-id: c1a5aab7-6e6b-4ef5-b20d-f9ddb095577f ). The job took 5 minutes 20 seconds to complete, so went beyond the previous 202 second limit.

Tagging for 1.39 release because there was a request on matrix to backport to REL1_39 (As in make mw core rel1_39 pull in a fixed version of library). I'll leave it to release managers if they want to do it.

per T358308 - one of the major causes of upload failure for medium-large files, is that during a deploy the job runners get killed. Making it retryable would fix that issue.

Can you try uploading again. The issue might be fixed now

Yes, this is due to T358308. Until that is fixed, you can upload large (>3GB) files via https://commons.wikimedia.org/wiki/Help:Server_side_upload

Ok, i did some more digging in logstash.

An interesting thing about this, is that it gives way more dynamic programmability to CSS. Since previously the CSS pages were static pages you had to edit to change. With variables, the variable can be set inline, so it can be set via lua, or in page previews without save, or even as a result of just a cross-domain GET request.

In T334372#9711759, @Tacsipacsi wrote:

Why would be the third one “by far” the most useful? I’d think most use cases can be satisfied by either of the first two, especially if TemplateStyles is allowed within the SVGs. Of course, there are some cases that can be satisfied only by the third option (say, using CSS variables defined in wikitext), but they aren’t that important.

looks good to me

Looking at recent logs, it does seem like some of the failures might be caused by things other than a timeout (There was one recently that failed after 17 seconds). However i still think the larger files are failing due to some sort of timeout as they tend to fail at around 202 second mark, which is pretty suspicious. Sample size is pretty low since users know large uploads don't work so they don't try and upload large files.

I think the better solution would be to just make MW generate the rel link. HTML should not be in messages, but that doesn't mean we shouldn't use it all.

Clearly this is no longer going to happen.

I used to be opposed to this, but I am now more a fan.

In T358308#9696868, @hnowlan wrote:

Kubernetes jobrunners were using the default max_execution_time of 180s, which might be the culprit for these behaviours. I've bumped it to match the metal jobrunners.

Existing installs arent automatically converted, but i dont think we want that anyways.

Anyone with write access to MediaWiki:Sidebar also can do what they want with MediaWiki:Common.js so it looks like this bug is not very easy to exploit.

lgtm

Looking at a recent example, it seems like the change didn't take affect (Or i was wrong about the cause).

Mostly i was looking at how many uploads get to this stage but don't get past it. I was looking at https://logstash.wikimedia.org/goto/f299fd6b01197908797af218435bd927 and comparing revieved last chunk vs chunk recombined messages.

In T361435#9675012, @bd808 wrote:

In T361435#9674842, @Bawolff wrote:

If its being moved anyways, would be kind of cool if it was sub-domain based.

Would a scheme like $TOOL.static.toolforge.org provide the desired in-browser separation or would we need $TOOL.$SOMETHING.$TLD to get good results?

lgtm :)

If its being moved anyways, would be kind of cool if it was sub-domain based.

The proposed use-case of the format is for storing the editable source-code-esque version of a layered raster image. Its not clear if anyone is using it for that, but even if they were, i would not expect them to have very many views. That use case doesn't even require thumbnailing support.

Personally i was a fan of <source>....

Even if WMF was planning to do this (i have no idea if they are) it would probably not be a top level OKR.

As an aside, i had no idea this overlapped the toolbox on new vector. Guess i should be testing on that skin every now and then

Radiopedia is using lower resolution images (At least in the linked example, they seem to be using images about 555x630 px that seem to be about 20% of the size of the images used with the ImageStack example). If you go full screen mode you get bigger issues.

Graphs are a valuable feature. They're an important feature to a sizable portion of both the editor and reader communities.

The slide numbers sometimes present a nonsensical value like “1291/136”

Fixed

Images move too fast with up/down movement on trackpad or mouse

I slowed it down. However its possible that this might vary between different mice or browsers. Chrome seems to scroll a bit faster than firefox.

I've put my modified version on my test wiki at http://bawolff.net/monstranto/index.php/IMStack

The install was old but the file was new. I didn't really investigate further. Possibly it was a timedmediahandler issue as my test file was a mp4 video.

Try https://quarry.wmcloud.org/query/81208

SBOMs are great tools to detect supply chain attacks and mitigating them.

as I may be missing deleted

It should be noted there are some files on commons with too long names. See parent task.

So it sounds like the underlying problem, is max file name size is checked at upload but not during file rename (T359294).

we are at the more philosophical point of wondering at the point of it all, and what sort of failure analysis could avoid such events in the future.

T336595 feels like a band-aid solution and a bad one at that. It does nothing to adress the underlying security problem or maintenance problem. I personally don't think we should do that except as a last resort.

In T356436#9599232, @Rockingpenny4 wrote:

@Yaron_Koren I configured the InlineComments extension on my local and went through the documentation and found out that it uses ContentHandler , however I was not able to figure out how to navigate to it on my localhost. Is there any reference for that I can go through?

Its really only uploads > 1GB that have this issue, which is a very small portion of uploads. Maybe it would make sense to split AssembleUploadChunks job into a big and small version depending on how big the file is. The big version could be handled more similarly to how webVideoTranscode jobs are handled and the small version could be handled like a normal job. (Possibly have to do the same thing for PublishStashedFile, although much of the work in that job is reverifying the file which isn't strictly needed since it has already been done, and we could possibly just make it not do that, which might make it much faster for large files).

Its been about 5 days and no new examples. Previously there was about 2 a day. I'm going to call this fixed.

Also 2 of these are above 5GB which is our current max size.

object-src is probably not too useful in modern browsers now that <object> is just a glorified iframe. It probably makes sense to just always have that set to 'none'.

I guess PublishStashedFile could be further optimized if we didn't verify the file. In theory, the file should already have been verified when uploaded to stash. If we don't need to verify the file, then there is no need to download the file locally, and all operations can take place directly on swift. However maybe that's not worth it, and its better just to be paranoid and re-verify the file at all stages.

On discord, Sazpaimon suggested having chunked upload keep a running total of the SHA1 hash so far (e.g. using https://www.php.net/manual/en/function.hash-update.php ). That way MW never has to calculate the entire hash at one time, but instead the work is split over all the uploaded chunks.

When i was testing this, the new version did still calculate some SHA1's in maybeUpgradeRow in a deferredupdate. However that is unrelated to file backend, and possibly is something about my local install or the file type handler.