Page MenuHomePhabricator

Bawolff (Brian Wolff)
Busy-bodyAdministrator

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Tuesday

  • Clear sailing ahead.

User Details

User Since
Oct 25 2014, 1:53 AM (494 w, 1 d)
Roles
Administrator
Availability
Available
IRC Nick
Bawolff
LDAP User
Brian Wolff
MediaWiki User
Bawolff [ Global Accounts ]

Hi!

Recent Activity

Yesterday

Bawolff added a comment to T358308: AssembleUploadChunksJob & PublishStashedFile jobs seem to be timing out at about 3 minutes, but should be ~20 minutes.

Looking at recent logs, it does seem like some of the failures might be caused by things other than a timeout (There was one recently that failed after 17 seconds). However i still think the larger files are failing due to some sort of timeout as they tend to fail at around 202 second mark, which is pretty suspicious. Sample size is pretty low since users know large uploads don't work so they don't try and upload large files.

Sat, Apr 13, 9:43 PM · Patch-For-Review, WMF-JobQueue, MediaWiki-File-management

Fri, Apr 12

Bawolff added a comment to T360497: Remove "<a href" from licensing messages in WikimediaMessages.

I think the better solution would be to just make MW generate the rel link. HTML should not be in messages, but that doesn't mean we shouldn't use it all.

Fri, Apr 12, 7:48 PM · Language-Team (Language-2024-April-June), WikimediaMessages, Language-Technical Support (Language-Technical Support (Current) ), I18n, affects-translatewiki.net

Thu, Apr 11

Bawolff closed T137291: Transition all use of EasyTimeline to the Graph extension and decommission it from Wikimedia's servers as Declined.

Clearly this is no longer going to happen.

Thu, Apr 11, 9:13 PM · Technical-Debt, Wikimedia-Extension-setup, Multimedia, Epic, MediaWiki-extensions-Graph, EasyTimeline
Bawolff closed T137291: Transition all use of EasyTimeline to the Graph extension and decommission it from Wikimedia's servers, a subtask of T106123: Extensions needing to be removed from Wikimedia wikis, as Declined.
Thu, Apr 11, 9:12 PM · Wikimedia-Extension-setup, Tracking-Neverending
Bawolff added a comment to T334372: Add support for inline SVG.

I used to be opposed to this, but I am now more a fan.

Thu, Apr 11, 9:00 PM · MediaWiki-Parser
Bawolff awarded T334372: Add support for inline SVG a Love token.
Thu, Apr 11, 8:48 PM · MediaWiki-Parser

Mon, Apr 8

Bawolff added a comment to T358308: AssembleUploadChunksJob & PublishStashedFile jobs seem to be timing out at about 3 minutes, but should be ~20 minutes.

Kubernetes jobrunners were using the default max_execution_time of 180s, which might be the culprit for these behaviours. I've bumped it to match the metal jobrunners.

Mon, Apr 8, 8:23 PM · Patch-For-Review, WMF-JobQueue, MediaWiki-File-management

Thu, Apr 4

Bawolff closed T322633: Should sqlite main DB default to journal_mode=WAL ? as Resolved.
Thu, Apr 4, 12:43 AM · MW-1.42-notes (1.42.0-wmf.15; 2024-01-23), MediaWiki-General, SQLite
Bawolff added a comment to T322633: Should sqlite main DB default to journal_mode=WAL ?.

Existing installs arent automatically converted, but i dont think we want that anyways.

Thu, Apr 4, 12:43 AM · MW-1.42-notes (1.42.0-wmf.15; 2024-01-23), MediaWiki-General, SQLite

Wed, Apr 3

Bawolff changed the visibility for T361452: Foreground skin: stored XSS via MediaWiki:Sidebar.
Wed, Apr 3, 9:13 PM · security-bug, SecTeam-Processed, MediaWiki-skins-Foreground, Vuln-XSS, Security
Bawolff added a comment to T361452: Foreground skin: stored XSS via MediaWiki:Sidebar.

Anyone with write access to MediaWiki:Sidebar also can do what they want with MediaWiki:Common.js so it looks like this bug is not very easy to exploit.

Wed, Apr 3, 9:12 PM · security-bug, SecTeam-Processed, MediaWiki-skins-Foreground, Vuln-XSS, Security
Bawolff changed the visibility for T361449: Metrolook skin: stored XSS via MediaWiki:Sidebar.
Wed, Apr 3, 9:03 PM · SecTeam-Processed, security-bug, Metrolook, Vuln-XSS, Security, Security-Team
Bawolff added a comment to T358308: AssembleUploadChunksJob & PublishStashedFile jobs seem to be timing out at about 3 minutes, but should be ~20 minutes.

Looking at a recent example, it seems like the change didn't take affect (Or i was wrong about the cause).

Wed, Apr 3, 4:07 PM · Patch-For-Review, WMF-JobQueue, MediaWiki-File-management

Tue, Apr 2

Bawolff added a comment to T358308: AssembleUploadChunksJob & PublishStashedFile jobs seem to be timing out at about 3 minutes, but should be ~20 minutes.

Mostly i was looking at how many uploads get to this stage but don't get past it. I was looking at https://logstash.wikimedia.org/goto/f299fd6b01197908797af218435bd927 and comparing revieved last chunk vs chunk recombined messages.

Tue, Apr 2, 4:16 PM · Patch-For-Review, WMF-JobQueue, MediaWiki-File-management

Mon, Apr 1

Bawolff added a comment to T361435: Find a modern hostname for tools-static.wmflabs.org.

If its being moved anyways, would be kind of cool if it was sub-domain based.

Would a scheme like $TOOL.static.toolforge.org provide the desired in-browser separation or would we need $TOOL.$SOMETHING.$TLD to get good results?

Mon, Apr 1, 4:18 PM · cloud-services-team, Toolforge

Sun, Mar 31

Bawolff changed the visibility for T361365: Challenge: stored XSS on four challenge fields.
Sun, Mar 31, 4:18 AM · Vuln-XSS, Social-Tools, Challenge, Security, Security-Team
Bawolff added a comment to T361365: Challenge: stored XSS on four challenge fields.

lgtm :)

Sun, Mar 31, 3:49 AM · Vuln-XSS, Social-Tools, Challenge, Security, Security-Team
Bawolff added a comment to T361435: Find a modern hostname for tools-static.wmflabs.org.

If its being moved anyways, would be kind of cool if it was sub-domain based.

Sun, Mar 31, 2:52 AM · cloud-services-team, Toolforge

Sat, Mar 30

Bawolff added a comment to T260286: Deprecate xcf files.

The proposed use-case of the format is for storing the editable source-code-esque version of a layered raster image. Its not clear if anyone is using it for that, but even if they were, i would not expect them to have very many views. That use case doesn't even require thumbnailing support.

Sat, Mar 30, 10:51 PM · Community-consensus-needed, Commons, Thumbor, MediaWiki-File-management

Fri, Mar 29

Bawolff added a comment to T361407: Add short-to-type aliases for <syntaxhighlight> and <syntaxhighlight inline>.

Personally i was a fan of <source>....

Fri, Mar 29, 10:27 PM · Patch-For-Review, SyntaxHighlight

Tue, Mar 26

Bawolff added a comment to T334940: All Graphs broken on Wikimedia wikis (due to security issue T336556).

Even if WMF was planning to do this (i have no idea if they are) it would probably not be a top level OKR.

Tue, Mar 26, 8:12 PM · User-zeljkofilipin, Regression, User-notice, Tech Ambassadors & Translators, MediaWiki-extensions-Graph
Bawolff added a comment to T358890: Deleting the last inline comments should restore the page to default display.

As an aside, i had no idea this overlapped the toolbox on new vector. Guess i should be testing on that skin every now and then

Tue, Mar 26, 5:51 PM · Patch-For-Review, MediaWiki-extensions-InlineComments
Bawolff added a comment to T355733: Implement better CT Scan Viewer.

Radiopedia is using lower resolution images (At least in the linked example, they seem to be using images about 555x630 px that seem to be about 20% of the size of the images used with the ImageStack example). If you go full screen mode you get bigger issues.

Tue, Mar 26, 5:17 PM · Wikimedia-Medicine, User-Harej

Fri, Mar 22

Bawolff added a comment to T334940: All Graphs broken on Wikimedia wikis (due to security issue T336556).

Graphs are a valuable feature. They're an important feature to a sizable portion of both the editor and reader communities.

Fri, Mar 22, 12:38 PM · User-zeljkofilipin, Regression, User-notice, Tech Ambassadors & Translators, MediaWiki-extensions-Graph

Wed, Mar 20

alaa awarded T355733: Implement better CT Scan Viewer a Love token.
Wed, Mar 20, 4:42 PM · Wikimedia-Medicine, User-Harej
Bawolff added a comment to T355733: Implement better CT Scan Viewer.

The slide numbers sometimes present a nonsensical value like “1291/136”

Fixed

Images move too fast with up/down movement on trackpad or mouse

I slowed it down. However its possible that this might vary between different mice or browsers. Chrome seems to scroll a bit faster than firefox.

Wed, Mar 20, 1:55 PM · Wikimedia-Medicine, User-Harej

Tue, Mar 19

Bawolff awarded T360442: +2 in mediawiki/ for Siddharth (SD0001) a Like token.
Tue, Mar 19, 4:34 PM · MediaWiki-Gerrit-Group-Requests

Fri, Mar 15

TheDJ awarded T358830: Uploads fail due to 401 error from swift on wednesdays a Stroopwafel token.
Fri, Mar 15, 10:07 PM · MW-1.42-notes (1.42.0-wmf.23; 2024-03-19), MediaWiki-Engineering, Commons, MediaWiki-File-management, SRE-swift-storage

Mar 15 2024

aaron awarded T358830: Uploads fail due to 401 error from swift on wednesdays a Yellow Medal token.
Mar 15 2024, 9:38 PM · MW-1.42-notes (1.42.0-wmf.23; 2024-03-19), MediaWiki-Engineering, Commons, MediaWiki-File-management, SRE-swift-storage
Bawolff added a comment to T355733: Implement better CT Scan Viewer.

I've put my modified version on my test wiki at http://bawolff.net/monstranto/index.php/IMStack

Mar 15 2024, 6:51 AM · Wikimedia-Medicine, User-Harej

Mar 14 2024

Bawolff added a comment to T348294: FSFileBackend spends a lot of time generating uneeded sha1 hashes that are expensive for large files.

The install was old but the file was new. I didn't really investigate further. Possibly it was a timedmediahandler issue as my test file was a mp4 video.

Mar 14 2024, 8:45 PM · MW-1.42-notes (1.42.0-wmf.21; 2024-03-05), MW-Interfaces-Team, Patch-For-Review, Performance Issue, MediaWiki-File-management, Commons, Multimedia

Mar 13 2024

Bawolff added a comment to T71417: Additional sort options for categories:: Usage (most/least used pages); Creation date (newest/oldest date of original image/video).

Try https://quarry.wmcloud.org/query/81208

Mar 13 2024, 8:01 PM · Commons, MediaWiki-Categories
Bawolff created T360024: Localisation cache isn't properly detecting when it needs to be refreshed on master.
Mar 13 2024, 11:39 AM · MediaWiki-Internationalization, MW-1.42-release

Mar 8 2024

Bawolff added a comment to T359634: Adopt SBOMs for MediaWiki.

SBOMs are great tools to detect supply chain attacks and mitigating them.

Mar 8 2024, 8:48 PM · SecTeam-Processed, Security-Team, Security

Mar 6 2024

Bawolff added a comment to T359176: Long-titled archived files can get its path metadata truncated due to not having enough storage space, leading to orphan, non accesible files (was: Two files on commons have invalid UTF-8 characters in path metadata).

as I may be missing deleted

Mar 6 2024, 10:43 PM · Patch-For-Review, Commons, MediaWiki-File-management, media-backups
Bawolff closed T359294: MW allows bypassing file name restrictions when moving a file page as Resolved.

It should be noted there are some files on commons with too long names. See parent task.

Mar 6 2024, 10:42 PM · MW-1.41-notes, MW-1.40-notes, MW-1.39-notes, MW-1.42-notes (1.42.0-wmf.22; 2024-03-12), Commons, MediaWiki-File-management, media-backups
Bawolff closed T359294: MW allows bypassing file name restrictions when moving a file page, a subtask of T359176: Long-titled archived files can get its path metadata truncated due to not having enough storage space, leading to orphan, non accesible files (was: Two files on commons have invalid UTF-8 characters in path metadata), as Resolved.
Mar 6 2024, 10:41 PM · Patch-For-Review, Commons, MediaWiki-File-management, media-backups
Bawolff created T359294: MW allows bypassing file name restrictions when moving a file page.
Mar 6 2024, 9:39 AM · MW-1.41-notes, MW-1.40-notes, MW-1.39-notes, MW-1.42-notes (1.42.0-wmf.22; 2024-03-12), Commons, MediaWiki-File-management, media-backups
Bawolff added a comment to T359176: Long-titled archived files can get its path metadata truncated due to not having enough storage space, leading to orphan, non accesible files (was: Two files on commons have invalid UTF-8 characters in path metadata).

So it sounds like the underlying problem, is max file name size is checked at upload but not during file rename (T359294).

Mar 6 2024, 9:30 AM · Patch-For-Review, Commons, MediaWiki-File-management, media-backups
Bawolff added a comment to T334940: All Graphs broken on Wikimedia wikis (due to security issue T336556).

we are at the more philosophical point of wondering at the point of it all, and what sort of failure analysis could avoid such events in the future.

Mar 6 2024, 8:58 AM · User-zeljkofilipin, Regression, User-notice, Tech Ambassadors & Translators, MediaWiki-extensions-Graph

Mar 5 2024

Bawolff added a comment to T334940: All Graphs broken on Wikimedia wikis (due to security issue T336556).

T336595 feels like a band-aid solution and a bad one at that. It does nothing to adress the underlying security problem or maintenance problem. I personally don't think we should do that except as a last resort.

Mar 5 2024, 7:08 PM · User-zeljkofilipin, Regression, User-notice, Tech Ambassadors & Translators, MediaWiki-extensions-Graph
Bawolff added a comment to T356436: Improve InlineComments extension.

@Yaron_Koren I configured the InlineComments extension on my local and went through the documentation and found out that it uses ContentHandler , however I was not able to figure out how to navigate to it on my localhost. Is there any reference for that I can go through?

Mar 5 2024, 9:46 AM · MediaWiki-extensions-InlineComments, Google-Summer-of-Code (2024)
Bawolff added a comment to T358308: AssembleUploadChunksJob & PublishStashedFile jobs seem to be timing out at about 3 minutes, but should be ~20 minutes.

Its really only uploads > 1GB that have this issue, which is a very small portion of uploads. Maybe it would make sense to split AssembleUploadChunks job into a big and small version depending on how big the file is. The big version could be handled more similarly to how webVideoTranscode jobs are handled and the small version could be handled like a normal job. (Possibly have to do the same thing for PublishStashedFile, although much of the work in that job is reverifying the file which isn't strictly needed since it has already been done, and we could possibly just make it not do that, which might make it much faster for large files).

Mar 5 2024, 6:14 AM · Patch-For-Review, WMF-JobQueue, MediaWiki-File-management
Bawolff closed T350917: Incomplete files uploaded - chunked upload drops last chunk. as Resolved.

Its been about 5 days and no new examples. Previously there was about 2 a day. I'm going to call this fixed.

Mar 5 2024, 5:16 AM · MW-1.42-notes (1.42.0-wmf.20; 2024-02-27), UploadWizard, SRE-swift-storage, Commons
Bawolff added a comment to T352740: Server-side upload request for cypp0847.

Also 2 of these are above 5GB which is our current max size.

Mar 5 2024, 5:14 AM · Server-side-upload-request

Mar 2 2024

Bawolff added a comment to T327588: Consider adding more CSP directives to MediaWIki core .

object-src is probably not too useful in modern browsers now that <object> is just a glorified iframe. It probably makes sense to just always have that set to 'none'.

Mar 2 2024, 9:21 PM · Patch-For-Review, MediaWiki-General, Security, ContentSecurityPolicy
Bawolff added a comment to T200820: FAILED: stashfailed: Could not read file "mwstore://local-swift-eqiad/local-temp/a/ac/15xi9btm14os.u9p1dr.1208681.webm.0"..

I guess PublishStashedFile could be further optimized if we didn't verify the file. In theory, the file should already have been verified when uploaded to stash. If we don't need to verify the file, then there is no need to download the file locally, and all operations can take place directly on swift. However maybe that's not worth it, and its better just to be paranoid and re-verify the file at all stages.

Mar 2 2024, 8:34 PM · MW-1.42-notes (1.42.0-wmf.23; 2024-03-19), SRE-swift-storage, MediaWiki-Uploading, User-revi, Multimedia
Bawolff added a comment to T200820: FAILED: stashfailed: Could not read file "mwstore://local-swift-eqiad/local-temp/a/ac/15xi9btm14os.u9p1dr.1208681.webm.0"..

On discord, Sazpaimon suggested having chunked upload keep a running total of the SHA1 hash so far (e.g. using https://www.php.net/manual/en/function.hash-update.php ). That way MW never has to calculate the entire hash at one time, but instead the work is split over all the uploaded chunks.

Mar 2 2024, 8:49 AM · MW-1.42-notes (1.42.0-wmf.23; 2024-03-19), SRE-swift-storage, MediaWiki-Uploading, User-revi, Multimedia
Bawolff closed T348294: FSFileBackend spends a lot of time generating uneeded sha1 hashes that are expensive for large files as Resolved.

When i was testing this, the new version did still calculate some SHA1's in maybeUpgradeRow in a deferredupdate. However that is unrelated to file backend, and possibly is something about my local install or the file type handler.

Mar 2 2024, 8:39 AM · MW-1.42-notes (1.42.0-wmf.21; 2024-03-05), MW-Interfaces-Team, Patch-For-Review, Performance Issue, MediaWiki-File-management, Commons, Multimedia
Bawolff closed T348294: FSFileBackend spends a lot of time generating uneeded sha1 hashes that are expensive for large files, a subtask of T191805: Allow Mediawiki to store file size greater than 32 bits, as Resolved.
Mar 2 2024, 8:38 AM · MW-1.42-notes (1.42.0-wmf.3; 2023-10-31), Patch-For-Review, Schema-change, MediaWiki-File-management, Commons, Multimedia
Bawolff closed T14490: Warning needed if dimensions of reupload differ as Declined.

I agree with Luke.

Mar 2 2024, 6:42 AM · Commons, Multimedia, MediaWiki-File-management
Bawolff closed T59717: Systematic HTTP 503 timeouts by uploading big (<100MB) TIFF images via API action=upload, without stash, without async as Declined.

This bug is too old and too lacking in details to be actionable. Possibly https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1007983 may help with this, but who knows.

Mar 2 2024, 6:39 AM · Commons, Multimedia, MediaWiki-File-management
Bawolff created T358939: change-propogation service retries jobs that set RunnableJob::allowsRetries() false after a timeout.
Mar 2 2024, 5:01 AM · MW-1.42-notes (1.42.0-wmf.23; 2024-03-19), ChangeProp, WMF-JobQueue
Bawolff updated the task description for T358308: AssembleUploadChunksJob & PublishStashedFile jobs seem to be timing out at about 3 minutes, but should be ~20 minutes.
Mar 2 2024, 2:09 AM · Patch-For-Review, WMF-JobQueue, MediaWiki-File-management
Bawolff updated the task description for T358308: AssembleUploadChunksJob & PublishStashedFile jobs seem to be timing out at about 3 minutes, but should be ~20 minutes.
Mar 2 2024, 2:01 AM · Patch-For-Review, WMF-JobQueue, MediaWiki-File-management
Bawolff renamed T358308: AssembleUploadChunksJob & PublishStashedFile jobs seem to be timing out at about 3 minutes, but should be ~20 minutes from Increase timeout on AssembleUploadChunksJob to AssembleUploadChunksJob & PublishStashedFile jobs seem to be timing out at about 3 minutes, but should be ~20 minutes.
Mar 2 2024, 1:59 AM · Patch-For-Review, WMF-JobQueue, MediaWiki-File-management
Bawolff updated subscribers of T358308: AssembleUploadChunksJob & PublishStashedFile jobs seem to be timing out at about 3 minutes, but should be ~20 minutes.

Oh, i see that deployment-charts/charts/mediawiki/values.yaml has an upstream_timeout set to 180.0 seconds. I believe mw-jobrunners will inherit that. So presumably helmfile.d/services/mw-jobrunner/values.yaml should set upstream_timeout under mesh: to be 1203 seconds (to be 1 more second than the apache timeout). Or at least be closer to that. I'm unclear if it is expected that upstream_timeout & change-propagation timeout should be above the MW $wgRequestTimeout in order to capture all errors or if the HTTP connection should close before that point in order to trigger a client abort on the PHP side.

Mar 2 2024, 1:48 AM · Patch-For-Review, WMF-JobQueue, MediaWiki-File-management
Bawolff added a comment to T358308: AssembleUploadChunksJob & PublishStashedFile jobs seem to be timing out at about 3 minutes, but should be ~20 minutes.

Reading config files, it seems like the timeout should already be about 7 minutes (change-propagation DEFAULT_REQUEST_TIMEOUT = 7 minutes. job runner $wgRequestTimeout = 20 minutes).

Mar 2 2024, 1:39 AM · Patch-For-Review, WMF-JobQueue, MediaWiki-File-management

Mar 1 2024

Bawolff added a comment to T179884: Files occasionally getting uploaded to Commons without file pages..

Just checking to see if this is still a thing. Most recent example seems to be https://commons.wikimedia.org/wiki/File:NDL801250_%E5%8B%A7%E6%A5%AD%E5%8A%9F%E7%B8%BE%E9%8C%B2_%E7%AC%AC2%E7%B7%A8_part1.pdf (Feb 29). There seems to be not a whole lot of older images, but possibly that is just because they get fixed or tagged as copyvio and deleted. There is a log entry for that file of an UploadStashFileNotFoundException exception. I'm not really sure how that could happen with the file becoming actually published. Its possible there were multiple publish attempts though and that is from a later one. I'm a bit surprised there are no other log messages. It used the non-async path for assembling, so presumably it was using that path for publishing too.

Mar 1 2024, 4:50 AM · UploadWizard, Multimedia, SRE-swift-storage, Commons
Bawolff renamed T358830: Uploads fail due to 401 error from swift on wednesdays from Uploads fail due to 401 error from swift to Uploads fail due to 401 error from swift on wednesdays.
Mar 1 2024, 4:07 AM · MW-1.42-notes (1.42.0-wmf.23; 2024-03-19), MediaWiki-Engineering, Commons, MediaWiki-File-management, SRE-swift-storage
Bawolff updated the task description for T358830: Uploads fail due to 401 error from swift on wednesdays.
Mar 1 2024, 4:00 AM · MW-1.42-notes (1.42.0-wmf.23; 2024-03-19), MediaWiki-Engineering, Commons, MediaWiki-File-management, SRE-swift-storage
Bawolff created T358830: Uploads fail due to 401 error from swift on wednesdays.
Mar 1 2024, 3:59 AM · MW-1.42-notes (1.42.0-wmf.23; 2024-03-19), MediaWiki-Engineering, Commons, MediaWiki-File-management, SRE-swift-storage
Bawolff added a comment to T350917: Incomplete files uploaded - chunked upload drops last chunk..

Well so far so good, but may be a bit early to tell so I will keep this bug open until a week has passed to make sure the issue is really fixed. The fix should be in place as of Feb 28 19:30 UTC. The only upload since then that is a multiple of 5mb is [[File:OR-126,_Knowles_Tunnel_-_53555943932.jpg]], which appears to be a false positive, as the source image is exactly 20MB. Prior to this we were having about 1 corrupted upload a day.

Mar 1 2024, 2:40 AM · MW-1.42-notes (1.42.0-wmf.20; 2024-02-27), UploadWizard, SRE-swift-storage, Commons

Feb 27 2024

Bawolff added a comment to T22326: Option to strip some metadata on upload (GPS/geolocation privacy).

The easiest thing to do here would be to make sure the end user knows that they should strip location data before uploading. Apparently we already do this in UploadWizard but not other upload workflows. This is very much a case of patch-welcome if somebody wants to do that. It could be a warning box or note on the uploading workflow (perhaps during the upload itself "This contains location data are you sure you want to upload it?" for example.

Feb 27 2024, 3:17 PM · UploadWizard, Privacy, Multimedia, MediaWiki-Uploading

Feb 26 2024

Bawolff added a comment to T358507: XSS on doc.wikimedia.org (documentation generated by yard) (CVE-2024-27285).

We could also add a sandbox directive to the csp policy

Feb 26 2024, 6:27 PM · Patch-For-Review, Infrastructure-Foundations, SecTeam-Processed, doc.wikimedia.org, Vuln-XSS, Upstream, Security, Security-Team
Bawolff updated the task description for T358308: AssembleUploadChunksJob & PublishStashedFile jobs seem to be timing out at about 3 minutes, but should be ~20 minutes.
Feb 26 2024, 4:47 PM · Patch-For-Review, WMF-JobQueue, MediaWiki-File-management
Bawolff created T358466: AssembleUploadChunks job should be allowed to retry.
Feb 26 2024, 8:00 AM · Patch-For-Review, MediaWiki-Core-JobQueue, MediaWiki-Uploading

Feb 23 2024

Bawolff added a comment to T350917: Incomplete files uploaded - chunked upload drops last chunk..

Change 1005965 had a related patch set uploaded (by Brian Wolff; author: Brian Wolff):

[mediawiki/core@master] Make chunked upload delete chunks in deferred job

https://gerrit.wikimedia.org/r/1005965

Feb 23 2024, 11:56 AM · MW-1.42-notes (1.42.0-wmf.20; 2024-02-27), UploadWizard, SRE-swift-storage, Commons
Bawolff added a comment to T200820: FAILED: stashfailed: Could not read file "mwstore://local-swift-eqiad/local-temp/a/ac/15xi9btm14os.u9p1dr.1208681.webm.0"..

I wonder if the slow part of the process is deleting all the chunks. Im not 100% sure, but as far as i can tell that is not paralelized. It would also help explain the errors being file missing, instead of running out of time in middle of concatenate. Its also not super important since they will eventually get deleted anyways, maybe we should rearrange things so code reports success before doing delete step.

Feb 23 2024, 9:38 AM · MW-1.42-notes (1.42.0-wmf.23; 2024-03-19), SRE-swift-storage, MediaWiki-Uploading, User-revi, Multimedia
Bawolff created T358308: AssembleUploadChunksJob & PublishStashedFile jobs seem to be timing out at about 3 minutes, but should be ~20 minutes.
Feb 23 2024, 6:34 AM · Patch-For-Review, WMF-JobQueue, MediaWiki-File-management
Bawolff added a comment to T200820: FAILED: stashfailed: Could not read file "mwstore://local-swift-eqiad/local-temp/a/ac/15xi9btm14os.u9p1dr.1208681.webm.0"..

Ok, so new logs do show a bunch of:

Feb 23 2024, 6:21 AM · MW-1.42-notes (1.42.0-wmf.23; 2024-03-19), SRE-swift-storage, MediaWiki-Uploading, User-revi, Multimedia
Bawolff added a comment to T353068: Internal error: The server could not save the temporary file.

Doesn't help that AssembleUploadChunksJob.php takes the specific (and even localized!) error and replaces it with a super generic one. Maybe we should stop doing that.q

Feb 23 2024, 5:02 AM · SRE-swift-storage, UploadWizard
Bawolff added a comment to T350917: Incomplete files uploaded - chunked upload drops last chunk..

As an aside some historical discussion at https://static-codereview.wikimedia.org/MediaWiki/104687.html (for context, back in the day we used mysql for job tracking which probably made the code less risky at the time)

Feb 23 2024, 1:16 AM · MW-1.42-notes (1.42.0-wmf.20; 2024-02-27), UploadWizard, SRE-swift-storage, Commons

Feb 22 2024

Bawolff renamed T350917: Incomplete files uploaded - chunked upload drops last chunk. from Incomplete files uploaded (10 MB interruption) to Incomplete files uploaded - chunked upload drops last chunk..
Feb 22 2024, 11:59 PM · MW-1.42-notes (1.42.0-wmf.20; 2024-02-27), UploadWizard, SRE-swift-storage, Commons
Bawolff added a comment to T350917: Incomplete files uploaded - chunked upload drops last chunk..

More recent example is File:Delft_Van_Miereveltlaan_7.jpg (aka 1aqdj7jclxmc.afcsnk.1553787.jpg aka 1aqdj6vvp0a8.a3pwzj.1553787.jpg ) which is cut off at 10MB.

Feb 22 2024, 11:57 PM · MW-1.42-notes (1.42.0-wmf.20; 2024-02-27), UploadWizard, SRE-swift-storage, Commons
Bawolff added a comment to T350917: Incomplete files uploaded - chunked upload drops last chunk..

Picking a recent failure:

mvernon@cumin1001:~$ sudo cumin -x --force --no-progress --no-color -o txt O:swift::proxy "zgrep -F '0/0e/Wikidata_43.jpg' /var/log/swift/proxy-access.log.3.gz" >~/junk/T350917.txt
#Cumin output elided
mvernon@cumin1001:~$ grep " PUT " junk/T350917.txt | grep 'public'
moss-fe2001.codfw.wmnet: Nov 24 19:12:12 moss-fe2001 proxy-server: 10.192.48.101 10.192.32.51 24/Nov/2023/19/12/12 PUT /v1/AUTH_mw/wikipedia-commons-local-public.0e/0/0e/Wikidata_43.jpg HTTP/1.0 201 - wikimedia/multi-http-client%20v1.1 AUTH_tk77cb529d3... 10485760 - 574193492001fdf36bdf02cbd36887a4 tx56ca15da11c2455281325-006560f58b - 0.2022 - - 1700853131.809281588 1700853132.011436224 0
ms-fe1013.eqiad.wmnet: Nov 24 19:12:12 ms-fe1013 proxy-server: 10.192.48.101 10.64.48.149 24/Nov/2023/19/12/12 PUT /v1/AUTH_mw/wikipedia-commons-local-public.0e/0/0e/Wikidata_43.jpg HTTP/1.0 201 - wikimedia/multi-http-client%20v1.1 AUTH_tk3f3803a6c... 10485760 - 574193492001fdf36bdf02cbd36887a4 tx460fd22327174e6ba0df8-006560f58c - 0.3835 - - 1700853132.315957069 1700853132.699444294 0

So you can see there are two successful PUTs (one to each ms cluster), both from mw2279.codfw.wmnet, of the same 10485760-byte object.

So from the swift point of view, it was offered a 10M object, and stored it as requested, so I think the problem lies further up the stack (uploadwizard was used, maybe it was confused by a client interruption or somesuch?)

Feb 22 2024, 11:21 PM · MW-1.42-notes (1.42.0-wmf.20; 2024-02-27), UploadWizard, SRE-swift-storage, Commons
Bawolff added a comment to T200820: FAILED: stashfailed: Could not read file "mwstore://local-swift-eqiad/local-temp/a/ac/15xi9btm14os.u9p1dr.1208681.webm.0"..

Here's the relevant logs, sorted by time:

moss-fe2001.codfw.wmnet: Feb 21 20:19:14 moss-fe2001 proxy-server: 10.192.48.105 10.192.32.51 21/Feb/2024/20/19/14 PUT /v1/AUTH_mw/wikipedia-commons-local-temp.b8/b/b8/1aqaxe3dattw.u1suhi.6080484.webm.0 HTTP/1.0 201 - wikimedia/multi-http-client%20v1.1 AUTH_tk7b6ed208d... - - - tx948576b6a72e4600a62d1-0065d65ac1 - 0.9693 - - 1708546753.577275038 1708546754.546539783 -
ms-fe1009.eqiad.wmnet: Feb 21 20:19:16 ms-fe1009 proxy-server: 10.192.48.105 10.64.0.166 21/Feb/2024/20/19/16 PUT /v1/AUTH_mw/wikipedia-commons-local-temp.b8/b/b8/1aqaxe3dattw.u1suhi.6080484.webm.0 HTTP/1.0 201 - wikimedia/multi-http-client%20v1.1 AUTH_tke5beae87e... - - - tx4683f2a851d249ea89bf1-0065d65ac2 - 1.5437 - - 1708546754.563668013 1708546756.107330084 -
ms-fe2013.codfw.wmnet: Feb 21 20:59:24 ms-fe2013 proxy-server: 10.194.152.76 10.192.0.87 21/Feb/2024/20/59/24 GET /v1/AUTH_mw/wikipedia-commons-local-temp.b8/b/b8/1aqaxe3dattw.u1suhi.6080484.webm.0 HTTP/1.0 200 - wikimedia/multi-http-client%20v1.1 AUTH_tk7b6ed208d... - 97498542 - tx8b68cb8506d34d1a9bb03-0065d66420 - 12.5442 - - 1708549152.229032040 1708549164.773195744 0
ms-fe2013.codfw.wmnet: Feb 21 21:00:02 ms-fe2013 proxy-server: 10.194.152.76 10.192.0.87 21/Feb/2024/21/00/02 DELETE /v1/AUTH_mw/wikipedia-commons-local-temp.b8/b/b8/1aqaxe3dattw.u1suhi.6080484.webm.0 HTTP/1.0 204 - wikimedia/multi-http-client%20v1.1 AUTH_tk7b6ed208d... - - - txfabad3d0f009413ea82dd-0065d66452 - 0.0458 - - 1708549202.850069761 1708549202.895848513 0
ms-fe1009.eqiad.wmnet: Feb 21 21:00:03 ms-fe1009 proxy-server: 10.194.152.76 10.64.0.166 21/Feb/2024/21/00/03 DELETE /v1/AUTH_mw/wikipedia-commons-local-temp.b8/b/b8/1aqaxe3dattw.u1suhi.6080484.webm.0 HTTP/1.0 204 - wikimedia/multi-http-client%20v1.1 AUTH_tke5beae87e... - - - tx26b0b6c03f1b4577a65b7-0065d66453 - 0.0454 - - 1708549203.070249796 1708549203.115652323 0
ms-fe2013.codfw.wmnet: Feb 21 21:03:12 ms-fe2013 proxy-server: 10.194.155.232 10.192.0.87 21/Feb/2024/21/03/12 GET /v1/AUTH_mw/wikipedia-commons-local-temp.b8/b/b8/1aqaxe3dattw.u1suhi.6080484.webm.0 HTTP/1.0 404 - wikimedia/multi-http-client%20v1.1 AUTH_tk7b6ed208d... - 70 - tx36a819fdb39c4d61bb4ee-0065d66510 - 0.0335 - - 1708549392.151752949 1708549392.185259581 0
ms-fe2013.codfw.wmnet: Feb 21 21:03:12 ms-fe2013 proxy-server: 10.194.155.232 10.192.0.87 21/Feb/2024/21/03/12 GET /v1/AUTH_mw/wikipedia-commons-local-temp.b8/b/b8/1aqaxe3dattw.u1suhi.6080484.webm.0 HTTP/1.0 404 - wikimedia/multi-http-client%20v1.1 AUTH_tk7b6ed208d... - 70 - txfffc016f7af4438eac05b-0065d66510 - 0.0284 - - 1708549392.332885504 1708549392.361268997 0

From which, I see:

  • First, 10.192.48.105 (mw2283) uploads to eqiad and codfw at 20:19
  • Then at 20:59, 10.194.152.76 (I guess a k8s node?) does a successful GET
  • almost immediately, it issues a DELETE from both eqiad and codfw, which succeed
  • Finally, at 21:03, 10.194.155.232 (another k8s node?) tries two GETs, both of which return 404 (as you'd expect, the object has been deleted)
Feb 22 2024, 8:44 PM · MW-1.42-notes (1.42.0-wmf.23; 2024-03-19), SRE-swift-storage, MediaWiki-Uploading, User-revi, Multimedia
Bawolff merged T355433: Problem with uploading large files (2 GB) into T200820: FAILED: stashfailed: Could not read file "mwstore://local-swift-eqiad/local-temp/a/ac/15xi9btm14os.u9p1dr.1208681.webm.0"..
Feb 22 2024, 12:14 AM · MW-1.42-notes (1.42.0-wmf.23; 2024-03-19), SRE-swift-storage, MediaWiki-Uploading, User-revi, Multimedia
Bawolff merged task T355433: Problem with uploading large files (2 GB) into T200820: FAILED: stashfailed: Could not read file "mwstore://local-swift-eqiad/local-temp/a/ac/15xi9btm14os.u9p1dr.1208681.webm.0"..
Feb 22 2024, 12:14 AM · SRE-swift-storage, UploadWizard
Bawolff merged T355610: Problem uploading 4GB FLAC file in Upload Wizard to Wikimedia Commons into T200820: FAILED: stashfailed: Could not read file "mwstore://local-swift-eqiad/local-temp/a/ac/15xi9btm14os.u9p1dr.1208681.webm.0"..
Feb 22 2024, 12:12 AM · MW-1.42-notes (1.42.0-wmf.23; 2024-03-19), SRE-swift-storage, MediaWiki-Uploading, User-revi, Multimedia
Bawolff merged task T355610: Problem uploading 4GB FLAC file in Upload Wizard to Wikimedia Commons into T200820: FAILED: stashfailed: Could not read file "mwstore://local-swift-eqiad/local-temp/a/ac/15xi9btm14os.u9p1dr.1208681.webm.0"..
Feb 22 2024, 12:12 AM · SRE-swift-storage, UploadWizard
Bawolff added a comment to T355610: Problem uploading 4GB FLAC file in Upload Wizard to Wikimedia Commons.

https://gerrit.wikimedia.org/r/1005632 may help with this.

Feb 22 2024, 12:03 AM · SRE-swift-storage, UploadWizard
Bawolff added a comment to T200820: FAILED: stashfailed: Could not read file "mwstore://local-swift-eqiad/local-temp/a/ac/15xi9btm14os.u9p1dr.1208681.webm.0"..

Gerrit patch to detect the situation where jobs are racing each other. It will also abort the job if a race is detected, but if this race is actually happening with any frequency (which hopefully the logging code would detect) we would probably want to use a real lock for robustness. In principle there should never be more than one assemble job for a specific file, not just at a time, but ever, so races should not be happening 🤷

Feb 22 2024, 12:02 AM · MW-1.42-notes (1.42.0-wmf.23; 2024-03-19), SRE-swift-storage, MediaWiki-Uploading, User-revi, Multimedia

Feb 21 2024

Bawolff added a comment to T200820: FAILED: stashfailed: Could not read file "mwstore://local-swift-eqiad/local-temp/a/ac/15xi9btm14os.u9p1dr.1208681.webm.0"..

maybe what is happening is that two assemble jobs are running at the same time and the first one is successful. Previously i was investigating the possibility of two jobs where both fail and wondering why no errors happened. However if the first one succeeds there would be no error, source file would be deleted. The second would then fail due to missing files. The second one overwrites the status in db, thus the user only sees the failure. The only thing is that then the first job should make a new stash file, which while never published should still be visible in user's uploadstash.

Feb 21 2024, 9:55 PM · MW-1.42-notes (1.42.0-wmf.23; 2024-03-19), SRE-swift-storage, MediaWiki-Uploading, User-revi, Multimedia
Bawolff added a comment to T200820: FAILED: stashfailed: Could not read file "mwstore://local-swift-eqiad/local-temp/a/ac/15xi9btm14os.u9p1dr.1208681.webm.0"..

FWIW, i've been investigating this. It does seem to be happening at commons with some regularity for larger uploads. However in the modern case it is backend-fail-notexists not the read error tgr mentions above.

Feb 21 2024, 8:45 AM · MW-1.42-notes (1.42.0-wmf.23; 2024-03-19), SRE-swift-storage, MediaWiki-Uploading, User-revi, Multimedia

Feb 19 2024

stjn awarded T308160: Uncensor use of "filter" CSS property on wikitext pages a Like token.
Feb 19 2024, 3:55 PM · Parsoid, User-notice-archive, MW-1.42-notes (1.42.0-wmf.19; 2024-02-20), Security, MediaWiki-General, CSS

Feb 17 2024

Bawolff added a comment to T116701: XMPReader::checkParseSafely requires allow_url_fopen to be set enabled.

I wonder if this is still really needed. I think libxml has checks against billion laughs vulns

Feb 17 2024, 11:43 PM · XMPReader, Security
Bawolff renamed T356971: Rename help key to help-raw in HTMLForm and deprecate old key name from HTMLForm doesn't escape the help key by default to Rename help key to help-raw in HTMLForm and deprecate old key name.
Feb 17 2024, 9:08 PM · good first task, SecTeam-Processed, MediaWiki-HTMLForm, Vuln-XSS, Security
Bawolff added a comment to T356971: Rename help key to help-raw in HTMLForm and deprecate old key name.

Sorry, won't be able to work on this after all.

Feb 17 2024, 9:02 PM · good first task, SecTeam-Processed, MediaWiki-HTMLForm, Vuln-XSS, Security
Bawolff added a comment to T356768: DOM Clobbering Risk in WikiBooks.

As a practical matter, it seems like it is rather difficult to exploit when you can't control the id/name of <a> tags, and you cannot do 2 level (x.y) dom clobbering [where you fully control y]

Feb 17 2024, 12:28 AM · MW-1.42-notes (1.42.0-wmf.25; 2024-04-02), Patch-For-Review, Vuln-XSS, SecTeam-Processed, Security, Security-Team

Feb 16 2024

Bawolff added a comment to T308160: Uncensor use of "filter" CSS property on wikitext pages.

"The CSS filter keyword can now be used in html style attributes in wikitext".

Feb 16 2024, 4:57 AM · Parsoid, User-notice-archive, MW-1.42-notes (1.42.0-wmf.19; 2024-02-20), Security, MediaWiki-General, CSS
Bawolff added a comment to T191804: Allow to store files between 4 and 5 GB.

@Bawolff
Re: Tech News - What wording would you suggest as the content, and When should it be included? Thanks!

Feb 16 2024, 4:52 AM · User-notice-archive, Data-Persistence-Backup, media-backups, SRE-swift-storage, MediaWiki-File-management, Commons, Multimedia

Feb 15 2024

Bawolff awarded T357479: Stop sending X-Webkit-CSP and X-Webkit-CSP-Report-Only headers a Like token.
Feb 15 2024, 4:18 AM · SecTeam-Processed, Security-Team, ContentSecurityPolicy, Traffic
Bawolff closed T319584: Migrate bawolff from Toolforge GridEngine to Toolforge Kubernetes as Resolved.

I ended up just removing that part of the tool. It largely wasn't working for other reasons.

Feb 15 2024, 3:43 AM · Grid-Engine-to-K8s-Migration

Feb 13 2024

Bawolff closed T308160: Uncensor use of "filter" CSS property on wikitext pages as Resolved.
Feb 13 2024, 9:29 PM · Parsoid, User-notice-archive, MW-1.42-notes (1.42.0-wmf.19; 2024-02-20), Security, MediaWiki-General, CSS
Bawolff added a comment to T319584: Migrate bawolff from Toolforge GridEngine to Toolforge Kubernetes.

I didn't see this task until now. However i think i have done this.

Feb 13 2024, 5:56 PM · Grid-Engine-to-K8s-Migration
Bawolff created T357383: Feature request: Provide an S3 compatible API to do uploads.
Feb 13 2024, 9:22 AM · Commons, API Platform, MediaWiki-File-management
Bawolff added a comment to T356768: DOM Clobbering Risk in WikiBooks.

Example URL: https://en.wikibooks.org/w/index.php?title=User:XYZ&action=submit

Example Payload: The payload test<sup id=QUnit> clobbers the variable window.QUnit, which is used in multiple locations in the code as a part of a conditional check if(window.QUnit){ [...] }. This can result in client-side DoS, or simply breakage of other client-side functionalities due to triggering JS parsing issues.

Feb 13 2024, 9:11 AM · MW-1.42-notes (1.42.0-wmf.25; 2024-04-02), Patch-For-Review, Vuln-XSS, SecTeam-Processed, Security, Security-Team
Bawolff closed T191804: Allow to store files between 4 and 5 GB as Resolved.
Feb 13 2024, 8:33 AM · User-notice-archive, Data-Persistence-Backup, media-backups, SRE-swift-storage, MediaWiki-File-management, Commons, Multimedia
Bawolff added a comment to T357203: XSS through interface message in UnlinkedWikibase.

Not a security thing, but as an aside, you should probably be using ->inContentLanguage() if it ends up in the parser output (Or on newer mediawiki use $parser->msg() ). Otherwise the language of the error message might be random, as it will use the language of the user who viewed it last time the cache expired instead of the current user. [inContentLanguage of course prevents testing with ?uselang=x-xss]

Feb 13 2024, 7:18 AM · Vuln-XSS, SecTeam-Processed, MediaWiki-extensions-UnlinkedWikibase, affects-Miraheze, Security, Security-Team