Too bad, but I can see where you're coming from. T36958 sounds super interesting.

How about scraping RSS feeds every 10 minutes and caching the results? That's very little traffic. Includes titles, links, author and date.

While npr.org isn't covered by search engine cache, it seems https://www.nprillinois.org/ is. There's no obvious way to rewrite the URLs though, for example compare:

• https://www.npr.org/2024/04/18/1245573728/truth-social-shares-rally-trump-media-technology-djt
• https://www.nprillinois.org/2024-04-18/trump-media-shares-surge-after-a-miserable-run-pros-say-stay-away

In T362379#9729666, @Samwalton9-WMF wrote:

In T362379#9729651, @AlexisJazz wrote:

In T362379#9729541, @Mvolz wrote:

The NYTimes has been blocking us for a while, it briefly worked when we changed datacenters and ergo IP, but they've understandably reblocked us after a few weeks' reprieve!

There's not a whole lot we can do except to ask for IP exemptions - @Samwalton9-WMF would this be something partnerships could try?

..there's no budget to get a VPN for $2/month and spread the traffic across a dozen IPs? Do you need me to pay for it? I can do that.

I don't think hiding our traffic so that our impact is less obvious is the way to go here, not least because as far as I'm aware Citoid identifies itself in the user agent. Much better for us in the long run to work with these orgs to either educate them on what we're doing, or to figure out ways of reducing our traffic volume. We don't want to burn bridges, especially when many of these orgs are current or potential TWL partners.

In T362379#9729541, @Mvolz wrote:

The NYTimes has been blocking us for a while, it briefly worked when we changed datacenters and ergo IP, but they've understandably reblocked us after a few weeks' reprieve!

There's not a whole lot we can do except to ask for IP exemptions - @Samwalton9-WMF would this be something partnerships could try?

In T214989#9705822, @Jack_who_built_the_house wrote:

Run this in the console on Special:BlankPage:

await mw.loader.using( [ 'ext.wikiEditor', 'ext.CodeMirror.v6.WikiEditor' ] );
$( '.mw-body-content' ).append( $( '<textarea>' ) );
mw.addWikiEditor( $( 'textarea' ) );

It works. No?

Cool, that does work!

I just copy-pasted Special:Version from fawikinews, ruwiki and enwiki into a file and checked the diff. Note that I replaced all times with 00:00 because fawiki shows different times.

In T309328#9060806, @Vermont wrote:

before writing off some of the most trusted and dedicated users on Wikimedia as bad faith.

Depends on the project and individual CU.

https://commons.wikimedia.beta.wmflabs.org/w/thumb.php?f=Example.jpg&w=220 doesn't work either, but returns instantly instead of taking ~45s:

Request from (European IP redacted) via deployment-cache-text08 deployment-cache-text08, Varnish XID 396622189
Upstream caches: deployment-cache-text08 int
Error: 500, Internal Server Error at Thu, 28 Mar 2024 16:59:52 GMT

Compare to https://upload.wikimedia.beta.wmflabs.org/wikipedia/commons/a/a9/Example.jpg:

Request from (European IP redacted) via deployment-cache-upload08.deployment-prep.eqiad1.wikimedia.cloud, ATS/9.1.4
Error: 500, Cannot find server. at 2024-03-28 17:03:14 GMT

Beta cluster is actively used to test the Commons app. Must be annoying for testers that upload.wikimedia.beta.wmflabs.org has been down for weeks. I frequently use it to test gadgets. Found a train blocker or two while doing so. But I don't see a solution either. ;-(

In T359768#9656733, @jcrespo wrote:

I'm afraid SREs cannot help here- while it is ATS returning the error to the end user, it is not the source, it is upload.wikimedia.beta.wmflabs.org what is timing out (Failed to connect to upload.wikimedia.beta.wmflabs.org port 443: Connection timed out) , which means it is a beta issue, that we don't own. I couldn't find any team responsible for it at: https://www.mediawiki.org/wiki/Developers/Maintainers#Services_and_administration so returning it to the original tagging.

T354023 explains that, the Gadget: namespace has been removed globally.

Somebody changed it from 2300 to 8. See https://en.wikipedia.org/w/index.php?title=Wikipedia%3AVillage_pump_(technical)&oldid=1214586229#Gadget_being_used_by_-1_users

This is very interesting. Yes, I hack your HTML a lot. It's pretty maintenance-intensive. https://en.wikipedia.org/wiki/User:Alexis_Jazz/Factotum adds multiple buttons/links. (configurable, but it can add like six or seven max I think) Especially because I aim to support all skins, including Minerva.

Thanks for the heads up! I'll keep an eye on this. It seems okay on the patchdemo, but if anything funky happens I'll know where to look.

In T352167#9575683, @hashar wrote:

In T352167#9363925, @AlexisJazz wrote:

I use my own omgtb script, it's extremely convenient. IIRC due to CORS I can't obtain the task from https://train-blockers.toolforge.org/. If you can tell me how the correct task can be found within this domain I can try to update my script.

train-blockers is now emitting emit an Access-Control-Allow-Origin: * header (Gerrit change 978139)

In T340705#9574658, @stjn wrote:

In T340705#9574643, @AlexisJazz wrote:

The current skin has no effect on the result AFAIK? But it's true that the script doesn't filter for skins. If there's a Vector-only default gadget but you're using CologneBlue, well, that gadget isn't loading for you. If there's a Minerva-only gadget but you're using Vector, same. For the purpose of finding low-hanging fruit this doesn't really matter IMHO.

The current skin has no effect, which is a problem for figuring out what contributes to payload on mobile. E. g. I got this in ruWP https://ru.wikipedia.org/?oldid=136342483 but most are gadgets for a namespace/action/etc. or do not load in Minerva.

But this task isn't about figuring out what contributes to payload on mobile. And that pie chart seems very useful to me: it shows instantly that https://ru.wikipedia.org/wiki/MediaWiki:Gadget-wikibugs is the largest default gadget.

In T340705#9574638, @stjn wrote:

In T340705#9161969, @AlexisJazz wrote:

You might find running https://en.wikipedia.org/wiki/User:Alexis_Jazz/EverybodyLikesPie.js to be of some help on these projects. See https://it.wikisource.org/w/index.php?title=Utente:Alexis_Jazz/Sandbox&oldid=3227820 for example.

This isn’t much help because this script assumes that all default gadgets load in the current skin, which isn’t true for Minerva.

@Jdforrester-WMF I just updated a Debian 11 machine to Debian 12. Result: MediaWiki no longer works. Searching the web I stumbled upon https://discuss.freedombox.org/t/mediawiki-upgrade-notes-bullseye-to-bookworm/2609 and ran php update.php.

$ php update.php 
PHP Fatal error:  Uncaught Exception: Unable to open file /usr/share/mediawiki/extensions/LocalisationUpdate/extension.json: filemtime(): stat failed for /usr/share/mediawiki/extensions/LocalisationUpdate/extension.json in /usr/share/mediawiki/includes/registration/ExtensionRegistry.php:199
Stack trace:
#0 /usr/share/mediawiki/includes/GlobalFunctions.php(49): ExtensionRegistry->queue()
#1 /etc/mediawiki/LocalSettings.php(154): wfLoadExtension()
#2 /usr/share/mediawiki/includes/Setup.php(218): require_once('...')
#3 /usr/share/mediawiki/maintenance/doMaintenance.php(83): require_once('...')
#4 /usr/share/mediawiki/maintenance/update.php(319): require_once('...')
#5 {main}
  thrown in /usr/share/mediawiki/includes/registration/ExtensionRegistry.php on line 199

How to resolve this?
Open your LocalSettings.php and delete this line:

wfLoadExtension( 'LocalisationUpdate' );

Now run update.php. This resolved the issue for me and the wiki is loading once more.

In T357184#9530959, @Bawolff wrote:

If MW estimates the transcode will be bigger than that size, then it will refuse to do it. In essence, it will change the error on the VP9 2160P transcode of File:Politparade.webm.

Historically, $wgTranscodeBackgroundSizeLimit was passed to ulimit to prevent files larger than that size from being created, but it seems this was removed in 7d9fb4ce1.

This is the hard limit for max size of a video transcode.

It is?

$ wget "https://upload.wikimedia.org/wikipedia/commons/transcoded/d/d5/Politparade.webm/Politparade.webm.1440p.vp9.webm?download"
HTTP request sent, awaiting response... 200 OK
Length: 4168557856 (3.9G) [video/webm]
Saving to: ‘Politparade.webm.1440p.vp9.webm?download’

@Bawolff what will be affected by this limit?

In T355746#9526785, @Jdlrobson wrote:

@AlexisJazz I found this pretty easy to replicate:

• Visit https://en.m.wikipedia.org/wiki/Talk:H%E1%BB%93ng_B%C3%A0ng_dynasty in Chrome with iPhone SE emulator. Note the page renders correctly.
• Run mw.loader.load('//en.wikipedia.org/w/index.php?title=User:Alexis_Jazz/Bawl.js&action=raw&ctype=text/javascript'); in console. Note it renders incorrectly.

@jhathaway nope. Guess it was a fluke, or one of the Wikimedia mail servers (assuming that's load balanced) was malfunctioning, or maybe Cryptic did something different. I'll just decline this myself, anyone feel free to reopen if it happens again.

@DonTrung Did you clear your browser cache?

Probably nothing to be done here at this point.

In T352617 I provided some JS that could warn/correct signature issues while the user is setting it on Special:Preferences.

In T353486#9409991, @matmarex wrote:

I see, the problem is caused by https://en.wikipedia.org/wiki/User:Alexis_Jazz/Factotum included in your common.js here: https://commons.wikimedia.org/wiki/User:Donald_Trung/common.js#L-11

Considering T140606#6236721 maybe @matmarex has some thoughts about this?

In T352167#9377430, @Krinkle wrote:

@AlexisJazz Does this affect the ability to debug (ie get source maps for) any core, extension or gadget source code? It seems in each report only the startup module is implicated. I have a theory as to why, which if true, is not feasible to fix and I would likely address by silencing this particular instance of that warning.

Randomly on https://en.wikipedia.org/wiki/Special:Preferences

Source map error: Error: request failed with status 404
Resource URL: https://en.wikipedia.org/w/load.php?lang=en&modules=startup&only=scripts&raw=1&safemode=1&skin=vector
Source Map URL: /w/load.php?lang=en&modules=startup&only=scripts&skin=vector&sourcemap=1&version=6naiq

I just realized it's working again. (took me a moment to realize the "no response" error from my script vanished)

In T352167#9365090, @hashar wrote:

In T352167#9363925, @AlexisJazz wrote:

...
In my experience gadgets are cached longer than userscripts, and I have no interface admin rights anywhere on production. I'll see if I can maybe reproduce it on beta.

Indeed https://train-blockers.toolforge.org/api.php yields no CORS header :-/ The repository is https://gerrit.wikimedia.org/g/labs/tools/train-blockers , it should be an easy patch to add header('Access-Control-Allow-Origin: *');. I have proposed https://gerrit.wikimedia.org/r/c/labs/tools/train-blockers/+/978139

https://test.wikipedia.org/wiki/Main_Page?uselang=es

Source map error: Error: request failed with status 404
Resource URL: https://test.wikipedia.org/w/load.php?lang=es&modules=startup&only=scripts&raw=1&skin=vector-2022
Source Map URL: /w/load.php?lang=es&modules=startup&only=scripts&skin=vector-2022&sourcemap=1&version=qqu3e

https://commons.m.wikimedia.beta.wmflabs.org/wiki/Main%20Page?uselang=de&useskin=vector

Source map error: Error: request failed with status 404
Resource URL: https://commons.m.wikimedia.beta.wmflabs.org/w/load.php?lang=de&modules=startup&only=scripts&raw=1&skin=vector&target=mobile
Source Map URL: /w/load.php?lang=de&modules=startup&only=scripts&skin=vector&sourcemap=1&version=ds1ou

https://commons.m.wikimedia.beta.wmflabs.org/wiki/Main%20Page?uselang=pt

Source map error: Error: request failed with status 404
Resource URL: https://commons.m.wikimedia.beta.wmflabs.org/w/load.php?lang=pt&modules=ext.uls.common&skin=minerva&version=1kw47
Source Map URL: /w/load.php?lang=pt&modules=ext.uls.common&skin=minerva&sourcemap=1&version=1kw47

In T352167#9363247, @Aklapper wrote:

1.42.0-wmf.13 is six weeks away so if this should block deployment then it should be the next one instead.
We are on 1.42.0-wmf.5 currently: https://versions.toolforge.org/

In T340802#9361598, @Jdlrobson wrote:

@AlexisJazz the gadgets system is not impacted by this change. It maintains its own targets system (T328610) which I highly discourage using (as it may go away in future), but it's there in the mean time if you need it. There are no plans to deprecate Minerva. Not sure where you got any idea that this is happening. There are efforts to align the desktop Minerva/Vector skins and mobile Minerva experiences more (e.g. making gadgets work in both places) and the removal of the targets system as outlined in this ticket is part of that.

In T111565#9361839, @Jdlrobson wrote:

In T111565#9361729, @Izno wrote:

In T111565#9361493, @Jdlrobson wrote:

What exactly is the opposition here? Who is opposed to making this change (I can't see anything about this in the ticket).

T111565#9135498 points to the relevant gerrit patch review of which you were a part. I would be as happy as anyone to have it loaded 'natively', but until someone prioritizes doing so, a gadget it is. (Speaking of which, we're planning to roll that out to all mobile users Soon after the briefest of brief trials with registered users. Better hang on to your hats.)

Right but the patch didn't pose opposition - it just recognized that the fat finger concern is a problem and is hard to solve.

(Speaking of which, we're planning to roll that out to all mobile users Soon after the briefest of brief trials with registered users. Better hang on to your hats.)

It doesn't look like the gadget addresses the fat finger concern so I'm not sure why we intentionally want to roll out a bad experience to end users?

In T111565#9361493, @Jdlrobson wrote:

In T111565#9137021, @matmarex wrote:

As there are people opposed to making this change in MediaWiki core, I would suggest doing it in your wiki's MediaWiki:Common.js or a gadget, as long as your community finds that agreeable.

What exactly is the opposition here? Who is opposed to making this change (I can't see anything about this in the ticket).

My understanding of this ticket is nobody is prioritizing this work, the work is challenging and that's the only problem here.

In gadget/userscript form I made ColMask which is yet another take. Code is a bit messy though and it's JS on top of jquery.makeCollapsible. It does address the fat finger problem though, the whole collapsed object can be clicked to expand it and it uses buttons.

Nope, seems fine now.

it ded

Oh, what I wanted is actually possible, it just wasn't documented on https://www.mediawiki.org/wiki/Help:Images. (I added it now)

For me as well.

In T351876#9356445, @MatthewVernon wrote:

I think, per modules/tlsproxy/manifests/localssl.pp it was 180s when we were using nginx, so I'll adjust it accordingly.

As I don't know the cause I'm just tagging as both traffic and train blocker.

@Jdforrester-WMF @Esanders what are we supposed to do now? I've made more than one gadget that specifically targets MobileFrontend and not desktop Minerva. MakeMobileCollapsible is a default gadget now.

With limit-rate it doesn't interrupt:

$ time wget --limit-rate 10K "https://upload.wikimedia.org/wikipedia/commons/6/64/Gameplay_0_A.D._Alpha_26_Gefecht_gegen_KI_20221106_Teil_01_von_10.webm"
--2023-11-23 16:47:09--  https://upload.wikimedia.org/wikipedia/commons/6/64/Gameplay_0_A.D._Alpha_26_Gefecht_gegen_KI_20221106_Teil_01_von_10.webm
Resolving upload.wikimedia.org (upload.wikimedia.org)... 185.15.59.240, 2a02:ec80:300:ed1a::2:b
Connecting to upload.wikimedia.org (upload.wikimedia.org)|185.15.59.240|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 4213093616 (3.9G) [video/webm]
Saving to: ‘Gameplay_0_A.D._Alpha_26_Gefecht_gegen_KI_20221106_Teil_01_von_10.webm.4’

In T351876#9354956, @Lucas_Werkmeister_WMDE wrote:

I can add another datapoint to it being time-based, on my connection the download finished within less than a minute without any interruption.

In T351876#9354897, @Aklapper wrote:

Please add relevant project tags - I'm afraid that it will be hard for the Commons community to fix this entirely themselves :)

In T191804#9354264, @Bawolff wrote:

There's one more use I think: the 4K transcode of https://commons.wikimedia.org/wiki/File:Politparade.webm failed, presumably due to the 4GB limit.

More likely it hit a timeout. The lower resolution transcodes were already taking 9 hours. However if the transcode was > 4GB then the FileStoreRepo changes would fix it. Anyways, this task won't fix everything about large video files, there are still aspects that are going to be shaky for very large files.

How about both? The 1440p VP9 transcode is already 3.9GB.

In T191804#9354216, @Bawolff wrote:

As a note

My expectation is that the primary usage for a higher limit would be:

• Long (or HD) videos which previously had to use more aggressive compression to fit in 4GB
• Videos which previously had to be split up into multiple files

Right now we are averaging 12 files > 3GB a month on commons (overwritten and deleted files are negligible), and 3 files / month > 3.9GB. [enwiki has 0].

It seems for the most part large files are special cases, and I'm doubtful that increasing the limit will affect much in terms of capacity.

Latest revision as of 22:31, 22 November 2023 view source thank
Izno (talk | contribs)
→‎test: start with deploying this for users. specifically target mobile as that is where collapsible things are not enabled (rather than a skin as you should generally). see also phab:T111565

+ * MakeMobileCollapsible[ResourceLoader|targets=mobile|rights=minoredit|default]|MakeMobileCollapsible.js

In T351024#9332675, @Umherirrender wrote:

In T351024#9332620, @AlexisJazz wrote:

In T351024#9332573, @Umherirrender wrote:

The code to reduce the limit when content is parsed is in ApiQueryRevisionsBase, so this affects the following modules:

• list=alldeletedrevisions with adrlimit
• list=allrevisions with arvlimit
• prop=deletedrevisions with drvlimit
• prop=revisions with rvlimit

Also the deprecated list=deletedrevs with drlimit

Thanks for the clarification and research. You're talking about another sense of "parse" I'm guessing as the wikitext isn't actually parsed like action=parse does.

Yes, when fetching content the limit is set to 50/500, the normal limit is 500/5000. When parsing content via drvparse parameter the limit is set to 1/1.

The first number is without apihighlimit, the second with that right. There is no depending on the modul name, if you see different numbers that must depend on the rights (logged out vs. sysop/bot account)

In T351024#9332573, @Umherirrender wrote:

The code to reduce the limit when content is parsed is in ApiQueryRevisionsBase, so this affects the following modules:

• list=alldeletedrevisions with adrlimit
• list=allrevisions with arvlimit
• prop=deletedrevisions with drvlimit
• prop=revisions with rvlimit

Also the deprecated list=deletedrevs with drlimit

In T351124#9329265, @jpxg wrote:

This would seem to, in general, militate against the wisdom of implementing a spam URL blacklist as a magical totem that prevents *anybody* (including EC, including TE, including sysops) from editing the page.

Do we have some problem where template editors are spamming references to best-free-viagra.biz?