FTR, the removal of globals in the AF codebase wasn't still in production. While I knew this, I though I had set them to true in the default config, but I didn't. Deploying the changes above would have disabled profiling everywhere.
I'll schedule it for deployment next week, after the train.

Awesome, thanks!

@Raymond could you please move abusefilter-edit-status-profile to abusefilter-edit-status overwriting the old content in the latter? I forgot to ask before, but you already have clearance.

Only the 2 config patches are left, and they're scheduled in tomorrow's SWAT.

@MusikAnimal I'll take care of it. Scheduled in tomorrow's SWAT.

@1997kB Heh, no problem :) @Legoktm Could you please take a look at https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/AbuseFilter/+/482408/? Thanks!

@1997kB: @MarcoAurelio is correct. If what you said is actually true, it'd be very interesting. Could you please try again?

@Huji Actually, another idea came to mind. You may use new_html and check if there's a span with a cite error (the one you get when the ref identifier is not defined). So to have, for instance,

@elukey I didn't read it like that :-) Although, actually, this task deserves some attention, as it's really happening very often. Mine above was just a quick overview of the situation.

Well, the maintscript is on gerrit. However, given that it'll touch many rows, I want to include there any maintenance that we need to perform on the abuse_filter_log table. Currently, it's blocked on deciding what to do with T213478 and T34478. If someone could please take a look, it'd be great.

@Raymond Huh, I suspected that... I didn't want to change the key because it's pretty standard, but if you cannot change all messages automatically, of course I will! Unless I can be made sysop on translatewiki for 30 mins and do it manually.

FTR, even if mine is a different issue, it has the same solution: purging the local page of the message, even if it doesn't exist, fixed it.

Ah, https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/485516/ together with https://phabricator.wikimedia.org/diffusion/EABF/browse/master/includes/pagers/AbuseFilterPager.php$262. Quick fix coming.

@Raymond Could you please delete all existing abusefilter-edit-throttle-placeholder messages once the patch above is merged? The message will have a completely different content. Thanks a lot!

@daniel Well, while it's true that some JS is altering the message, that JS isn't from a user or wiki script, as the issue still happens in safemode. Maybe VE is replacing the standard message with its own? But yes, it could be a different bug. I guess we'll see after the patch above is deployed.

In T218918#5047968, @Lucas_Werkmeister_WMDE wrote:

In T218918#5047822, @Daimona wrote:

I noticed this on itwiki, too, so I'll provide another example for completeness: this week I have changed visualeditor-ca-editsource from "Modifica wikitesto" to "Modifica sorgente". Viewing any page will result in a quick flash of the new message, which is then suddenly replaced by the old one. While trying to debug I went to this page, which uses {{in:visualeditor-ca-editsource}} and showed the old version. Purging the page updated the message to the new one.

But this was on TranslateWiki, not on it.wikipedia.org directly, so if I understand @daniel’s fix correctly that shouldn’t even have been affected? [itwiki:MediaWiki:visualeditor-ca-editsource](https://it.wikipedia.org/wiki/MediaWiki:Visualeditor-ca-editsource) doesn’t exist as a page.

@Huji I wasn't totally right (it happens when you think too much right before going to sleep ;-)). I thought it would have been possible to use get_matches, but actually it would stop at the first occurrence. However, the regex is still bad (see https://regex101.com/r/ARn7U4/1) and it ends up with a catastrophic backtracking. Unfortunately I don't have a fast replacement at hand.

Now that we have a help tooltip, it's way better to move instructions there. I'll send a patch shortly.

I noticed this on itwiki, too, so I'll provide another example for completeness: this week I have changed visualeditor-ca-editsource from "Modifica wikitesto" to "Modifica sorgente". Viewing any page will result in a quick flash of the new message, which is then suddenly replaced by the old one. While trying to debug I went to this page, which uses {{in:visualeditor-ca-editsource}} and showed the old version. Purging the page updated the message to the new one.

@Huji Dang! I checked on logstash (from mobile phone, which is painful) and heck yes! AbuseFilter 101 is constantly taking ~150 seconds to execute on that page!
You are right, profiling is enabled on every wiki (and I already sent patches to remove profiling globals altogether). However, the avg execution time for that filter doesn't show up as very high due to the amount of edits (that's why there are patches to add the maximum recorded execution time). From a quick look I can see a couple of things to be optimized in that filter, please let me know if you need any help with it.

Aye, I was right! The prefix 'ARTICLE' wasn't updated to 'page'. The impact is that, in order to test uploads, users have to use article_* vars instead of page_*. Nothing else is affected and no need to backport as these values aren't saved in the DB.

TerraCodes is right, see for instance T218560.

@Legoktm I checked all the remaining versions up to 1.2.6 and I can now confirm that we only need 0.1.5 for php-ast. 0.1.2 should be kept for the current version only.

Ah, now I see. Would it still benefit from phpunit?

So I'm looking at the upgrade to PluginV2. The main change is that *Visitor classes are now instantiated by phan itself, which only wants a "get*ClassName" method instead of a constructor. This mostly means that we cannot pass in an instance of the plugin anymore. Moreover, Visitor and PreVisitor must inherit from different classes, which prevents them from having TaintednessBaseVisitor as common parent (should be fixable by turning TaintednessBaseVisitor into a trait). Plus, the MediaWiki checker runs both the generic visitor and the MW visitor, which again isn't possible in V2 because you can only provide a single class name. I'll probably do some gradual changes before the v2 upgrade.

In T218719#5039252, @hashar wrote:

We have T174339 to upgrade to phan 0.8.5, which requires a newer version of php-ast than the one we had.

@Legoktm Right. But anyway at the end of the migration we'll only have 0.1.4 for the current version, and ?[0] for the final version based on 1.2.x.

@Legoktm Cool, thanks! I think for now you can use whatever hack comes more handy for seccheck, as at the end of the upgrade we should ideally have a single version installed.
A conditional checking the required phan version seems fine.

@Legoktm I think the current version should work with 0.1.4. But actually, could you please add 0.1.5 as well? We'll need it for the next upgrade. Thanks!

@Krinkle IMHO this should be handled in core. Failures of deletion during page move should show up to the user like the ones for normal deletions do, instead of throwing.

@Legoktm could you please take a look?

$ become gerritbot
https://gerrit.wikimedia.org/r/#/c/mediawiki/tools/phan/SecurityCheckPlugin/+/497583/

Nitpicky comment: the description "Jobs restricted to trusted users. Will vote +2." for the test pipeline should be changed.

The switch to phan's PluginV2 as part of T216974 should boost the speed per PluginV2 documentation.

This would indeed be awesome. However, see my comment in T207344#5033198. Basically, phan has many breaking changes even in patch versions, so upgrading to 0.8.13 would already break a lot of stuff. Today I tried to get there, but eventually gave up due to this.
For whoever will want to proceed with this, here are the main obstacles I found while upgrading to 0.8.13. Note that I don't really know how phan works so it could be fault of mine (but I'm still happy to help).

• Clazz::getPropertyByNameInContext now has an extra required parameter, $is_static. However, I couldn't find a way to determine if the property is static where we call it.
• This commit causes 2 troubles:
  • It adds the new class UnaddressableTypedElement, which is extended by Variable. This breaks all typehints for TypedElementInterface in TaintednessBaseVisitor. Just removing them (and adding UnaddressableTypedElement to the docblock) seems to work fine, but it should be checked more deeply.
  • Parameter::getContext() is replaced by Parameter::getFileRef(), which however doesn't have the getScope() method used in setTaintedness.

Just a random comment: data actually used by existing abuse filters like the rank can be moved from added_lines to new AF variables defined via hooks.

I'm giving up with the upgrade. Phan doesn't comply with semver at all, and thus as I was saying above you'll face plenty of breaking changes even for x.x.y => x.x.z upgrades. At this point, I think seccheck needs a major rewrite in order to work with 1.2.6. If instead, you want to do it gradually, I suggest bumping to 0.9.6 first (which is roughly the same as 0.8.13), then switch to PluginV2 and slowly move on to 1.2.6.

@sbassett Well, actually I'm facing several breaking changes even with 0.8.0 => 0.8.13. The most important is the addition of UnaddressableTypedElement (which also doesn't have a true context but just a FileRef), which breaks several things in TaintednessBaseVisitor. More specifically, I'm talking about this commit.

@sbassett Thanks for the reply. I'm trying to understand how phan, AST etc. work to see if I can start bumping the phan version. For now I'm trying to get to phan ^0.8 and ast ^0.1.5, although it won't be fast and I cannot guarantee anything.

@zeljkofilipin Yes, it does. And I can also confirm that DateTime is the only input type affected by this bug.

@Bawolff Is there a specific reason to require PHP 7.0.0, or it's just because the plugin is untested with other versions?

Different message but same cause as the other task, which already has a patch.

A quick look points at https://gerrit.wikimedia.org/r/#/c/oojs/ui/+/495296/, since setFlags is a method defined in the FlaggedElement class.

In T218513#5031670, @zeljkofilipin wrote:

Train blockers are Unbreak Now!.

Great! Just a note - this setting is not for logging to Special:AbuseLog, but to recent changes. More specifically, to the places defined in $wgAbuseFilterNotifications, as this setting is only used to decide whether to include private filters there. Needless to say, having all filters logged to RC can greatly help patrollers (with or without abusefilter-modify right), who will know what edits are more likely to be bad.

This is hitting us over and over.

Train blocker because this hides form fields, high prio because there's time to fix it before the train starts.

In T218472#5030487, @MGChecker wrote:

I don't think removing these comments is a good idea, as people subscribed to this task recieve the respective email notifications anyway and are left with even more questions than before. If the intention is to deny the claims of @JruwJN, I would consider it the better approach to just say so, if the intention is to keep this hidden for now, it simply is not working. So I don't think these deletions do any good.

Hoping that this won't create more problem than it solves!