Afaik all follow up is done now

If there are other issues we should use a new bug for them.

Possibly dupe of https://phabricator.wikimedia.org/T296435

Seems like Title::getLinksTo is using LinkCache::getSelectFields which does not include page_namespace or page_title on 1.37.2 (but looks refactored on master). Maybe a backport gone wrong?

The plan is for there to be a document that will lay out the responsibilities for releases, and WMF will retain some of those responsibilities, such as providing the infrastructure and accounts.

In T185664#6941823, @Zache wrote:

Hmm, If I understand this correctly the "sighted" (stable version) is enough for Google News and it doesn't use quality tier for aggregating news. It woud be nice to confirm somehow.

• https://en.wikinews.org/wiki/User:Brian_McNeil/Google_News

well - other than GWtoolset running remotely, and pattypan requiring image downloads and spreadsheet wrangling.
interesting christmas present to commons, to turn off a tool that was developed by WMUK and unsupported for years, in favor of a java script tool supported by a single volunteer. perhaps you would care to consider a general tool management process?

I'm opposed to this. While i sympathize with MWStake wanting to increase its reach, MW should not be used for what is essentially advertising.

This was partially fixed in f3a17b7cb98b3d8

https://gerrit.wikimedia.org/r/c/mediawiki/extensions/RSS/+/787807 . I guess i shouldn't +2 myself, so if anyone wants to review...

Proposed patch

Its related to the custom strip marker scheme, i'm not sure if that's what is being referred to in the other task. The code path involved here is the one using the Sanitizer, not the one with a custom escaping function.

Looks good :)

$adMsg should probably also be ->escaped() as well for the href. FILTER_VALIDATE_URL allows urls like https://example.com/"><script>alert(1)</script>. Additionally, if you assume the message is not trusted, you can have javascript scheme urls to get xss e.g. javascript://%0aalert(1) passes FILTER_VALIDATE_URL.

Looks like it fixes the problem

In T109909#7879513, @ToBeFree wrote:

An anon-only email block on an IP address is unlikely to have any meaning other than the here-desired one, as Special:EmailUser is not available to logged-out users by default and in probably almost all MediaWiki configurations.

Is there any way to increase the priority for this to get fixed?

I think i figured out the actual cause on cluster - SpamBlacklist extension. I believe 3cb265f1 is the change that triggered this issue.

Ok, steps to reproduce locally (This is contrived, I don't know if this is what's happening on cluster. However, MediaWiki\Extension\TemplateData\Hooks::onMultiContentSave does something very similar to this, so i believe something along these is plausible):

My theory is that this is caused by https://github.com/wikimedia/mediawiki-extensions-MassMessage/blob/master/includes/Content/MassMessageListContentHandler.php#L238 - calling $parserOutput->setText('') for the case where generateHtml is false. The docs say you should do $parserOutput->setText(null) in that case. (I would also note, that ParserOutput constructor defaults to text being '' if you don't set it at all instead of null, which seems really dangerous and wrong)

This is expected behaviour on 1.37 - You need 1.38 for the extension to work with TimedMediaHandler. There's not really much i can do for earlier versions.

Just as an aside, i feel like i've seen an increase in support desk questions because of this. In particular people will try to to do the upgrade in steps, but then they will try and use an old version of mw with php8 and have problems.

So we want to add a new env check for this. The existing ones are here https://github.com/wikimedia/mediawiki/blob/master/includes/installer/Installer.php#L137 we probably also have to add appropriate i18n messages.

Hmm. We might soonhit a scaling limit with the bot, as Module:extensionJson is getting close to 2mb max page size

I'm going to guess that this is a bug in Firefox's GPU acceleration, and might depend on GPU driver and things.

Currently working around by seeing if the descriptionurl field matches the file name, but that's pretty hacky

As an aside, SemanticForms extension does do something like this.

Nothing particularly special about this extension - you could just directly link the url instead.

In T269130#7548729, @Pealhasan.x2 wrote:

Hello Dylsss,Hope you are well. I know that was valid, But I have no word to express how to prove that one! Now the Wikipedia Security team got it! I reported it last year in 2020 (December)! But who report again that is this year (2021). But my report got a duplicate tag! I don't know why! Is it possible to do anything about this issue?

Basically i suspect this is a dupe of T56033

Afaik, on (file object) cache miss, parser doesn't bulk load file objects, but loads them one at a time from db as it encounters them in the wikitext (compared to say how doing link existence checks works with batching). Even though its a local db, this probably adds up, especially if there's other complex stuff on the page. That would also explain why the null edit is fast - its probably succesfully hitting memcached much more.

In addition to the actual digits, should the thousands separator continue to be ٬ or should it be a comma , like in english?

Oh i see, we now use NumberFormatter (backed by CLDR data - https://github.com/unicode-org/cldr/blob/main/common/main/sd.xml#L4810 ) if digitTransformTable is not set.

Huh that's weird.

I changed the title of this bug to give a more clear technical summary.

Just FYI, I'm experimenting with performance improvements in the extension QuickInstantCommons. Initial tests show a 55x speed improvement, and I also have additional ideas that are not yet implemented.

It would probably make more sense to make a specific extension for that then use wikiseo (imho)

The individuals and teams on the maintainers list are separate. If a person is listed for a component and a team is listed, it doesn't mean that that person is on that team. (Assuming Aaron hasn't switched teams, he is on the performance team not sdc)

and @Bawolff for the Reading team who own PdfHandler (since PDFs have the exact same problem, but just rarely trigger it for various reasons).

To be clear, ForeignAPIRepo::httpGET is used in places where last-modified header makes sense. However ForeignAPIRepo::httpGetCached is not.

If this worked properly, it would probably be much safer to set fileMetadataExpiry to something high like a week (On the presumption that images that are edited, get edited often.

My example was randomly chosen. I think its pretty representative of enwikinews, but its possible the worst case is worse. The worst case on ruwikinews is probably quite a bit worse due to the order of magnitude size difference (but still not anywhere near as bad as the problematic dpl query)

Its similar to DPL. Main differences:

So i missed originally that the ordermethod was set to "created" (aka page_id) on the dpl query. This made me confused about the query plan chosen and i said some incorrect things about it being unideal.

Just to summarize some additional investigation that was done:

• The triggering event seems to be this edit https://ru.wikinews.org/w/index.php?title=%D0%A1%D0%BB%D1%83%D0%B6%D0%B5%D0%B1%D0%BD%D0%B0%D1%8F:%D0%96%D1%83%D1%80%D0%BD%D0%B0%D0%BB%D1%8B&logid=19137921
  • Creating that category page triggered a job which parsed a large number of pages that had a DPL query on them, instigating the DB to overload
• Its possible (although i don't have conclusive evidence) that mariadb was using an unideal query plan for this particular query, which may have exacerbated the situation. This particular DPL query may have been scaling proportional to the size of the categorylinks table (44M) instead of proportional to the size of the category (180k) i didn't originally notice that ordermethod in the query was set to created instead of categoryadd. categoryadd has more efficient execution
• ruwikinews had log entries that this particular DPL query (The one for the infobox on "В мире‏‎") had been failing due to timeouts for several days in the lead up to the incident. This meant that the mitigation introduced in the previous incident (wgDLPQueryCacheTime - Which was really a band-aid) was at least sometimes not applying, as it doesn't work for queries that take so long that they timeout.
• Possibly the mitigation from last time (wgDLPQueryCacheTime) helped us scale further, but when it did fail, it resulted in a much harder failure

In T287380#7251388, @ssr wrote:

Please enable DPL at least at Main Page of RWN. This 1 page seem to be safe for servers and it's most crucial for RWN.

A complicating factor is that the category name is parsed and people throw #switch in there. Maybe not an edge case that matters in practise though.

In T287380#7242110, @Krassotkin wrote:

@Bawolff Maybe you can try replacing database queries with CirrusSearch queries. It won't take long but we can at least test this hypothesis.

Note: extension:googlenewssitemap which powers the rss feeds on wikinews ( https://en.wikinews.org/w/index.php?title=Special:NewsFeed&format=atom ) does similar queries.

For reference, DPL queries can be grouped into the following four performance categories (from best to worst):

I'd argue we have to disable DPL from everywhere, this has potential to cause a full outage in our system but from any wiki that has it turned on. Intentionally or unintentionally.

On the other hand, in the last discussion, it was decided to rewrite the DPL to CirrusSearch.

In T287380#7236461, @Jdforrester-WMF wrote:

IIRC, Wikimedia's DPL fork was created as part of the Wikivoyage migration rush because a few of the incoming communities insisted they needed it; some of those extensions have been subsequently worked around and dropped from Wikimedia production, and that's probably the best outcome for DPL too. I appreciate that a couple of wikis have built processes that rely upon it, but I can't see it being remotely justified given the complexity and risk to the wider Wikimedia movement.

ruwikinews now has 13M pages (bigger than enwiktionary). Even if the band aid solution would have worked last time, we might be passed that at this point.

In T263220#7237177, @jcrespo wrote:

@Bawolff I think you wrote the comment at https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/645994/2/wmf-config/PoolCounterSettings.php#87 I think that suggests to reduce/tune if errors the values of 'workers'/'maxqueue' (not the timeout) in sync, but the comment is not 100% clear to me. Could you confirm that is the intention?

I will send an amend or a followup patch with clearer wording if you confirm that is the case, we don't have to think a lot if eventually we end up under the stress of an outage. E.g.:

// Note, this uses nowait:., so 'timeout' must always be 0.
// 'worker' and 'maxqueue' can be tuned, but they should be equal to each other.
// E.g. In the event DPL is causing DB problems, decrease both 'worker' AND 'maxqueue' to 2.

If oauth apps are identifying people by email adress, seems like there is an impersonation risk here if they forget to verify the confirmed flag.

Its generally expected that an extension will change the user's language if appropriate. By default the user's language will be the content language not the request language.

Text layer is generally generated on upload. It used to be refreshable by action=purge but i dont think that is the case anymore. There is a maintenance script to refresh all of a specific file type.