Page MenuHomePhabricator

Dry run for normalizeThrottleParameters.php
Open, HighPublic

Description

In order to fix throttle parameters, we need to execute the normalizeThrottleParameters.php script [1]. I'm hereby asking to please run it with --dry-run on all Wikimedia wikis to see how it goes. We don't need anything more than that, as the script will then be added to update.php [2].

[1]: https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/AbuseFilter/+/459368/
[2]: https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/AbuseFilter/+/459245/

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 480681 had a related patch set uploaded (by Tim Starling; owner: Daimona Eaytoy):
[mediawiki/extensions/AbuseFilter@wmf/1.33.0-wmf.8] [1.33.0-wmf.8] Report all filters with wrong throttle parameters

https://gerrit.wikimedia.org/r/480681

Change 480681 merged by jenkins-bot:
[mediawiki/extensions/AbuseFilter@wmf/1.33.0-wmf.8] [1.33.0-wmf.8] Report all filters with wrong throttle parameters

https://gerrit.wikimedia.org/r/480681

Ran it again:

bgwiki:  Throttle groups are empty for the following filters: 20, 20. Please add some groups or disable throttling, then launch the script again.
cawiki:  normalizeThrottleParameter has found 1 rows to change in abuse_filter_action for the following IDs: 9
cawiki:  normalizeThrottleParameter would insert 1 rows in abuse_filter_history for the following filters: 9
cawiki:  Throttle parameter normalization would change a total of 2 rows.
cawikinews:  normalizeThrottleParameter has found 1 rows to change in abuse_filter_action for the following IDs: 15
cawikinews:  normalizeThrottleParameter would insert 1 rows in abuse_filter_history for the following filters: 15
cawikinews:  Throttle parameter normalization would change a total of 2 rows.
dawiki:  Throttle groups are empty for the following filters: 5, 5. Please add some groups or disable throttling, then launch the script again.
enwiki:  Throttle groups are empty for the following filters: 347, 742, 347, 742. Please add some groups or disable throttling, then launch the script again.
glwiki:  Throttle groups are empty for the following filters: 15, 15. Please add some groups or disable throttling, then launch the script again.
itwiki:  Throttle groups are empty for the following filters: 148, 367, 148, 367. Please add some groups or disable throttling, then launch the script again.
mrwiki:  Throttle count and period are malformed or empty for the following filters: 9. Please fix them by hand in the way they're meant to be, then launch the script again. Throttle groups are empty for the following filters: 125, 127, 129, 130, 131, 134, 129, 130, 131, 134, 127, 125. Please add some groups or disable throttling, then launch the script again.
rowiki:  Throttle groups are empty for the following filters: 67, 67. Please add some groups or disable throttling, then launch the script again.
ruwiki:  Throttle groups are empty for the following filters: 35, 35. Please add some groups or disable throttling, then launch the script again.
ruwiktionary:  Throttle groups are empty for the following filters: 33, 33. Please add some groups or disable throttling, then launch the script again.
trwiki:  normalizeThrottleParameter has found 5 rows to change in abuse_filter_action for the following IDs: 37, 46, 47, 48, 50
trwiki:  normalizeThrottleParameter would insert 5 rows in abuse_filter_history for the following filters: 47, 48, 50, 46, 37
trwiki:  Throttle parameter normalization would change a total of 10 rows.
ukwiki:  Throttle groups are empty for the following filters: 29, 30, 53, 30, 29, 53. Please add some groups or disable throttling, then launch the script again.
zhwiki:  Throttle groups are empty for the following filters: 131, 131. Please add some groups or disable throttling, then launch the script again.

Change 480704 had a related patch set uploaded (by Daimona Eaytoy; owner: Daimona Eaytoy):
[mediawiki/extensions/AbuseFilter@master] Another couple of minor fixes, hopefully the last change to this script.

https://gerrit.wikimedia.org/r/480704

Change 480705 had a related patch set uploaded (by Daimona Eaytoy; owner: Daimona Eaytoy):
[mediawiki/extensions/AbuseFilter@REL1_32] Re-fix the throttle script

https://gerrit.wikimedia.org/r/480705

Change 480706 had a related patch set uploaded (by Daimona Eaytoy; owner: Daimona Eaytoy):
[mediawiki/extensions/AbuseFilter@wmf/1.33.0-wmf.8] Re-fix the throttle script

https://gerrit.wikimedia.org/r/480706

@tstarling Thanks. The patch above should solve the duplicate IDs problem; I guess it's not necessary to dry-run again just because of that. And now we're back to the big question: how to reduce the amount of manual work. Personally, I think the only idea worth investigating is checking past revisions of the filter (possibly the first after the offending patch was merged) to see if we can restore its parameters. Then we should write a couple of lines in T/N with a link to the above paste so that people can fix filters on their wikis.

What happened here?
Now in every wikis there is a local AbuseFilter (with localized user names) user account that is a sysop.
In Finnish language projects: https://meta.wikimedia.org/wiki/Special:CentralAuth/V%C3%A4%C3%A4rink%C3%A4ytt%C3%B6suodatin
In Finnish Wikipedia a local bureaucrat just removed user rights from this user, because they were granted to user account without consensus (as required by the local policy).

At least give a global user name and make a global user page for that account. And announce in wikis that there is going to be a new sysop, and not just surprise everyone, "oh we have now a new sysop!"

While I am sympathetic to the users feeling surprised due to the appearance of a new sysop account, I feel we're making a storm out of a teacup looking at comments of apparent outrageousness here and there. If we look at Special:ListUsers/bot the same has happened for tons of other maintenance scripts or system accounts and nobody has ever complained. Perhaps when this maintenance of AbuseFilter is done we can run a script server-side to remove the sysop flags for the everywhere should the account not need them anymore, or at least on the wikis where the filter is not allowed to block (cfr. T212268). Meanwhile I suggest that we add a User-notice to solve the apparent confusion this has caused. But I must say that I disagree strongly with the comments I read somewhat implying this was done in bad faith, because it was not; and I thank very much @Daimona for devoting his volunteer time to fix and amend this extremely complex and useful extension for all of us. Thanks.

Thanks @MarcoAurelio for your comment and your appreciation. I totally agree that this looks like much ado about nothing. Currently, our priority (or, at least, mine) should be to get this maintenance done, then we should understand what to do with the system user. Removing the flag is discussed in the linked task, but I'm not convinced in removing it only where block isn't enabled. First, because the AbuseFilter user can also degroup and block autopromotion. Second, because we could use it for other maintenance tasks in the future (like it's being done here). Again, any discussion should be in the other task.
I'm also adding tech news to inform people about the change. The message could be something like

Last week, in order to perform some technical maintenance on AbuseFilter, a new account with sysop rights and the localized name "{{int:abusefilter-blocker}}" has been created on several wikis. On wikis where AbuseFilter can perform blocks, this user already existed and is automatically used to block people. For other wikis, the account will do nothing but a one-time maintenance, so there's no reason to be worried.

And we could also add the following:

Some filters cannot be automatically edited and must be manually checked. You may find a list of affected filters [[link|here]]; if you're familiar with AbuseFilter, please take a look and fix the faulty filters.

And we should also explain what the problem is:

The "{{int:abusefilter-action-throttle}}" action takes three parameters: count, period and groups. Previously, users could freely insert everything in those fields. Now we require these parameters to be valid, i.e. they should: *not be empty; *count and period must be positive integers; * groups must be a combination of valid groups (listed here), split by commas or newlines depending on the effect you want to achieve. [The link above explains this too]

Anomie removed a subscriber: Anomie.Dec 19 2018, 3:20 PM
Daimona moved this task from To Triage to Announce in next Tech/News on the User-notice board.EditedDec 21 2018, 4:27 PM

Ah, I just realized that the next T/N issue will be on Jan 7th. Not the best period for "bad" things to happen :/ On the other hand, hopefully we have a couple of days more to decide what to do here, and what to announce.

EDIT: Looking at reports on meta and on dewiki, I guess we should also specify that:

  • I didn't think the account would have been created by this script (actually, I thought it existed even on wikis without blocks); matter of fact, on some wikis it exists since 2008 and hasn't changed since then.
  • There's no evil plan to take over any wiki: I see several complot-leaning comments, especially directed to WMF (they don't listen, they impose their decisions, don't notify communities, don't do this or that etc.). Needless to say, I'll gladly stand up for this affair, and I'm not affiliated with WMF :-)
  • Link T212268 too; specify that the user can be blocked, desysopped and anything else, but it'll continue to do what it's instructed to do.
  • Even if the wiki has block disabled, it doesn't mean that they don't need this user. First, because it could be used for maintance purposes like here. Second, because it's also used for degrouping and blocking autopromotion. The latter isn't currently logged, but it will after T49412.
  • Maybe explain that system accounts cannot be used by normal people because their token, email etc. are automatically invalidated.

Ah, I just realized that the next T/N issue will be on Jan 7th. Not the best period for "bad" things to happen :/ On the other hand, hopefully we have a couple of days more to decide what to do here, and what to announce.

@Daimona, have you decided on what to do and announce? T/N on Jan 7th means it is locked and ready for translation on Jan 4th.

@Trizek-WMF I totally forgot that we didn't have a message ready, nor I realized that the next T/N is almost here. I think we should just include as much info as possible without being too verbose. My proposal is to use two separate messages:

In abuse filters, the "{{int:abusefilter-action-throttle}}" action takes three parameters: count, period and groups. Previously, people could freely insert everything in those fields. Now we require these parameters to be valid, i.e.: *not be empty; *count and period must be positive integers; * groups must be a combination of valid groups (listed [[:mw:Extension:AbuseFilter/Actions#Throttling|here]]), separated with commas or newlines depending on the effect you want to achieve, as explained in the link above. We now want to fix all filters in WMF to make them valid, but unfortunately many of them cannot be edited automatically and must be manually checked. You may find a list of such filters [[:phab:P7956|here]]; if you're familiar with AbuseFilter, please take a look and fix them. [[:phab:T209565|]]

On December 17h, in order to perform some technical maintenance on AbuseFilter per point above, a new account with sysop rights and the localized name "{{int:abusefilter-blocker}}" has been created on several wikis. On wikis where AbuseFilter can perform blocks, this user already existed and is automatically used to block people. For other wikis, the account will only do maintenance tasks if required, so there's no reason to be worried: no real person will use that account, ever. The account creation wasn't expected to happen, and due to holidays there have been no occasions to announce this in Tech/News. See [[:phab:T212268|]] for future plans.

Trizek-WMF added a comment.EditedJan 4 2019, 1:40 PM

@Daimona, I'm sorry but I don't understood your texts. :/

You have a list (P7956) of filters that are not formatted correctly, right? Your goal is to have those filters fixed by someone? If so, that's targeting a few wikis you should contact directly for a better impact. Tech News is too global for that, except if you want everyone to be aware of the new formatting you are working on (again, IIUC).

And what about T212268? What is the future? What should be added to Tech News?

@Trizek-WMF For the moment, I added the above proposals to T/N, of course I didn't mark them for translation as they have to be improved. To answer your questions:

  • Yes, the listed filters contain invalid data and thus don't work (as expected). We cannot fix them automatically and so yes, we're asking if someone could fix them - unless we find a way to do it automatically. Looking again, the list is pretty short and I could go to each wiki and explain the problem. Still, I think we should make everyone aware of the new validation requirements (i.e., just remove the last sentence).
  • As for T212268, future plans are to make the AbuseFilter user not be a sysop (and in general create a new user group for system users, which is T212720). We should probably change that last line to read something like

Given that several communities have reported concerns about system users being sysop (mostly because we lack a reliable way to identify them), we're working on making the AbuseFilter user not be a sysop anymore, and to add a dedicated user group for other system users. [https://phabricator.wikimedia.org/T212268] [https://phabricator.wikimedia.org/T212720]

The texts you have inserted are too long, I had to cut them a bit. Please review them ASAP, since I'm wrapping up the current issue: I plan to freeze it in 2 hours starting now.

I have shorten the first item to rely on the documentation. But I don't see any of the 3 required elements in it. Can you fix it?

@Trizek-WMF Thanks, the text was meant to be amended. I updated the docs to reflect the added validation, and reviewed your changes. I think they're fine, aside from these minor things:

  • The AbuseFilter account already existed on several wikis. The first message is for wikis where it was created on Dec 17th, to let people know that there's no reason to be worried because it's not a normal account. So I'd change it to:

On several wikis, an account named "<tvar|accountname>{{int:abusefilter-blocker}} </>" has been created on December 17th to perform some technical maintenance on AbuseFilter. This account has sysop rights but it's a system user and no human can use it. The account already existed on wikis where AbuseFilter can perform blocks, which are issued using this account. See [[<tvar|T212268>:phabricator:T212268</>|T212268]] for more information and future plans.

which has roughly the same length but focuses on different aspects (feel free to further amend it, of course).

  • The other message is OK, aside from a minor rewording:

They now require parameters as listed [[:mw:Extension:AbuseFilter/Actions#Throttling|on mediawiki.org]]

should be

They must now fulfill the requirements listed [[:mw:Extension:AbuseFilter/Actions#Throttling|on mediawiki.org]]

Change done, Thank you!
I've just changed "fulfill" by "strictly respect", which is less difficult to understand for non-native speakers (and translators).

Back to the core of this task, I think that any attempt to cover more cases would only end up with imposing a default. Thus, I think we should:

  1. (Merge the missing patch)
  2. Run the script without --dry-run
  3. Change the script so that, if it finds unfixable filters, it'll still change the other ones but report the faulty ones (and don't update the updatelog table, so that further executions will still report the errors).

Thoughts?

Change 480706 abandoned by Daimona Eaytoy:
Re-fix the throttle script

Reason:
All wikis to wmf.9

https://gerrit.wikimedia.org/r/480706

Actually, there's another improvement we can do: change groups to lowercase. Throttle groups are case sensitive and must be lowercase, but some filters user them with uppercase letters (see mrwiki for instance). The patch above covers this case and maps old groups to lowercase.

greg lowered the priority of this task from Unbreak Now! to High.Jan 11 2019, 11:30 PM
greg added a subscriber: greg.

Raising priority to UBN because parent task is UBN, and this also requires some extra work (although 1. and 2. are easy). I'll try to send a new patch today/tomorrow; @tstarling any chance you'll review this again?

Lowering to High, this is not currently an UBN! issue.

Lowering to High, this is not currently an UBN! issue.

Hm, probably true. Summing up:

  • Any new filter is fine
  • Editing for the first time a filter last edited before the original offending patch will cause no troubles (specifically, throttle params won't be removed as it used to happen)
  • Filters with invalid parameters are still affected; for those, the behaviour is unclear (depends on the single filter: it could vary between "never take any action" and "completely ignore throttle") and they're responsible for several errors seen in production. Mostly, T203535 and another undefined index on AbuseFilterViewEdit.

So probably not UBN!, although defo not below high.

He7d3r added a subscriber: He7d3r.Jan 12 2019, 10:39 PM
daniel lowered the priority of this task from High to Normal.
daniel added a subscriber: daniel.

It seems like the immediate problem is fixed, so lowering prio to normal.

Daimona raised the priority of this task from Normal to High.Jan 22 2019, 3:44 PM

@daniel Not really; actually, there are still two different PHP notices and a fatal caused by malformed parameters (all seen in production), plus affected filters don't do what they should. Not UBN, but neither normal.

Change 480704 merged by jenkins-bot:
[mediawiki/extensions/AbuseFilter@master] Re-fix the throttle script

https://gerrit.wikimedia.org/r/480704

greg removed a subscriber: greg.Jan 23 2019, 4:41 AM

Change 480705 merged by Daimona Eaytoy:
[mediawiki/extensions/AbuseFilter@REL1_32] Re-fix the throttle script

https://gerrit.wikimedia.org/r/480705

Change 486034 had a related patch set uploaded (by Daimona Eaytoy; owner: Daimona Eaytoy):
[mediawiki/extensions/AbuseFilter@wmf/1.33.0-wmf.13] Re-fix the throttle script

https://gerrit.wikimedia.org/r/486034

Change 486035 had a related patch set uploaded (by Daimona Eaytoy; owner: Daimona Eaytoy):
[mediawiki/extensions/AbuseFilter@wmf/1.33.0-wmf.14] Re-fix the throttle script

https://gerrit.wikimedia.org/r/486035

Change 486034 merged by jenkins-bot:
[mediawiki/extensions/AbuseFilter@wmf/1.33.0-wmf.13] Re-fix the throttle script

https://gerrit.wikimedia.org/r/486034

Change 486035 merged by jenkins-bot:
[mediawiki/extensions/AbuseFilter@wmf/1.33.0-wmf.14] Re-fix the throttle script

https://gerrit.wikimedia.org/r/486035

Mentioned in SAL (#wikimedia-operations) [2019-01-23T12:32:06Z] <zfilipin@deploy1001> Synchronized php-1.33.0-wmf.13/extensions/AbuseFilter/: SWAT: [[gerrit:486034|Re-fix the throttle script (T209565)]] (duration: 00m 54s)

Mentioned in SAL (#wikimedia-operations) [2019-01-23T12:36:48Z] <zfilipin@deploy1001> Synchronized php-1.33.0-wmf.14/extensions/AbuseFilter: SWAT: [[gerrit:486035|Re-fix the throttle script (T209565)]] (duration: 00m 55s)

1zfilipin@mwmaint1002:~$ foreachwiki extensions/AbuseFilter/maintenance/normalizeThrottleParameters.php --dry-run
2-----------------------------------------------------------------
3cawiki: normalizeThrottleParameter has found 1 rows to change in abuse_filter_action for the following IDs: 9
4cawiki: normalizeThrottleParameter would insert 1 rows in abuse_filter_history for the following filters: 9
5cawiki: Throttle parameter normalization would change a total of 2 rows.
6-----------------------------------------------------------------
7cawikinews: normalizeThrottleParameter has found 1 rows to change in abuse_filter_action for the following IDs: 15
8cawikinews: normalizeThrottleParameter would insert 1 rows in abuse_filter_history for the following filters: 15
9cawikinews: Throttle parameter normalization would change a total of 2 rows.
10-----------------------------------------------------------------
11dawiki: Throttle groups are empty for the following filters: 5. Please add some groups or disable throttling, then launch the script again.
12-----------------------------------------------------------------
13glwiki: Throttle groups are empty for the following filters: 15. Please add some groups or disable throttling, then launch the script again.
14-----------------------------------------------------------------
15mrwiki: Throttle count and period are malformed or empty for the following filters: 9. Please fix them by hand in the way they're meant to be, then launch the script again.
16-----------------------------------------------------------------
17rowiki: Throttle groups are empty for the following filters: 67. Please add some groups or disable throttling, then launch the script again.
18-----------------------------------------------------------------
19ruwiktionary: Throttle groups are empty for the following filters: 33. Please add some groups or disable throttling, then launch the script again.
20-----------------------------------------------------------------
21trwiki: normalizeThrottleParameter has found 5 rows to change in abuse_filter_action for the following IDs: 37, 46, 47, 48, 50
22trwiki: normalizeThrottleParameter would insert 5 rows in abuse_filter_history for the following filters: 47, 48, 50, 46, 37
23trwiki: Throttle parameter normalization would change a total of 10 rows.
24-----------------------------------------------------------------
25ukwiki: Throttle groups are empty for the following filters: 29, 30, 53. Please add some groups or disable throttling, then launch the script again.
26-----------------------------------------------------------------

Change 486083 had a related patch set uploaded (by Daimona Eaytoy; owner: Daimona Eaytoy):
[mediawiki/extensions/AbuseFilter@master] Update the throttle script

https://gerrit.wikimedia.org/r/486083

Short explanation the current status

This last execution showed that a few filters (all on mrwiki) were false positive and don't need to be fixed. However, I found on logstash two different errors related to the script: an Undefined index notice and a warning about aggregating and locking queries being discouraged. The patch above is a deeper overhaul of the script, which achieves the following:

  • Better docs.
  • Recognize a possible situation where throttle parameters are truly empty (i.e. the ID is missing too), caused by a temporary bug and which can resolved in a specific fashion. Report these cases separately. This also avoids the PHP notice.
  • If there are errors with the abuse_filter_action rows, stop the script after having processed them, instead of processing abuse_filter_history rows too.
  • When updating the abuse_filter table (done twice) use Database::update() instead of doing first a Database::select() and then a Database::replace() (boosts performance avoiding two locking SELECTs)
  • When selecting abuse_filter_history rows to update, do it separately for every filter, so that we can use a single SELECT + ORDER BY, instead of a query with MAX() + GROUP BY + LOCK IN SHARE MODE (unsupported by Postgres) and a subsequent SELECT.
  • Save throttle parameters for each filter when processing abuse_filter_action rows and reuse them for abuse_filter_history, so that we don't have to re-process the same parameters again for every abuse_filter_history row.

What are we blocked on

The new patch (https://gerrit.wikimedia.org/r/486083) needs some love and review, more than the last one being a bigger change. Then, we need a backport on REL1_32, and another backport for every wmf branch currently in production. Then, the script should be executed again in dry run. At that point, we'll have to decide how to handle unfixable rows. Since the bad version of the code stayed alive only for a couple of months, in theory this problem should only affect WMF wikis. So far we have the following proposals:

  1. Go and fix every filter by hand, and leave the script as it is. We already asked communities to do that, although we'll have to ask again, and this doesn't really apply to non-WMF wikis.
    1. Variant: even if there are unfixable filters, fix the other ones instead of stopping.
  2. Add new special cases for autofixing filters. For instance, try to restore parameters from a previous history row. However this wouldn't solve all the cases and, instead, could just make the script more complex.
  3. Impose a default. For instance, the default throttle parameters (3,60,user). Which isn't really good as we don't know what those filters are meant to do.
    1. Variant: do this only for disabled and/or deleted filters, where at least we won't do any harm.

I'm unsure what would be better, though.

Reedy removed a subscriber: Reedy.Jan 23 2019, 8:26 PM

Thanks for all your work on this, @Daimona. If the number of unfixable filters is small (<50) then I would be inclined to fix them manually, since changing the script takes a while, and a bug in the script could cause data corruption. Also, I don't think you should do do any more performance optimisations on this script, since the time it takes to write and review them is presumably much larger than the amount of time they will save. IIRC the script only takes a second or two per wiki.

Change 486083 merged by jenkins-bot:
[mediawiki/extensions/AbuseFilter@master] Update the throttle script

https://gerrit.wikimedia.org/r/486083

Change 489125 had a related patch set uploaded (by Tim Starling; owner: Daimona Eaytoy):
[mediawiki/extensions/AbuseFilter@wmf/1.33.0-wmf.16] [1.33.0-wmf.16] Update the throttle script

https://gerrit.wikimedia.org/r/489125

Change 489165 had a related patch set uploaded (by Daimona Eaytoy; owner: Daimona Eaytoy):
[mediawiki/extensions/AbuseFilter@REL1_32] Update the throttle script

https://gerrit.wikimedia.org/r/489165

@tstarling Thanks for the review! Yes, the amount of unfixable filters is now very small, thanks to people who fixed faulty filters on their wiki. In fact, only 7 filters are still marked as ToDo in P7956. The last patch did improve performance (hopefully!), but the main point was to fix a couple of bugs and dbms-compat issues, as well as removing unneeded extra queries.
I have backported the fix to REL1_32 as well, and now we only need a final dry-run to guarantee no errors/warnings (I'll check logstash). If you don't have the chance to do it yourself, I'll schedule it for SWAT next week. Then some final testing on my wiki and we'll be ready for the final step (i.e. without dry-run).

Change 489165 merged by Daimona Eaytoy:
[mediawiki/extensions/AbuseFilter@REL1_32] Update the throttle script

https://gerrit.wikimedia.org/r/489165

Change 489125 merged by jenkins-bot:
[mediawiki/extensions/AbuseFilter@wmf/1.33.0-wmf.16] [1.33.0-wmf.16] Update the throttle script

https://gerrit.wikimedia.org/r/489125

Mentioned in SAL (#wikimedia-operations) [2019-02-11T04:23:12Z] <TimStarling> on mwmaint1002: running normalizeThrottleParameters.php --dry-run on all wikis (T209565)

arwiki:  Throttle count and period are malformed or empty for the following filters: 66, 96. Please fix them by hand in the way they're meant to be, then launch the script again.
cawiki:  normalizeThrottleParameter has found 1 rows to change in abuse_filter_action for the following IDs: 9
cawiki:  normalizeThrottleParameter would insert 1 rows in abuse_filter_history for the following filters: 9
cawiki:  Throttle parameter normalization would change a total of 2 rows.
cawikinews:  normalizeThrottleParameter has found 1 rows to change in abuse_filter_action for the following IDs: 15
cawikinews:  normalizeThrottleParameter would insert 1 rows in abuse_filter_history for the following filters: 15
cawikinews:  Throttle parameter normalization would change a total of 2 rows.
dawiki:  Throttle groups are empty for the following filters: 5. Please add some groups or disable throttling, then launch the script again.
glwiki:  Throttle groups are empty for the following filters: 15. Please add some groups or disable throttling, then launch the script again.
mrwiki:  Throttle count and period are malformed or empty for the following filters: 9. Please fix them by hand in the way they're meant to be, then launch the script again.
rowiki:  Throttle groups are empty for the following filters: 67. Please add some groups or disable throttling, then launch the script again.
trwiki:  normalizeThrottleParameter has found 5 rows to change in abuse_filter_action for the following IDs: 37, 46, 47, 48, 50
trwiki:  normalizeThrottleParameter would insert 5 rows in abuse_filter_history for the following filters: 37, 46, 47, 48, 50
trwiki:  Throttle parameter normalization would change a total of 10 rows.
ukwiki:  Throttle groups are empty for the following filters: 29, 30, 53. Please add some groups or disable throttling, then launch the script again.
Daimona added a comment.EditedFeb 11 2019, 9:15 AM

Thanks @tstarling! So, the output is the same as the last time, aside from:

  • Two new filters reported on arwiki (unsure why they weren't before)
  • The filter from ruwikt isn't reported anymore (fixed by Huji)
  • No errors/warning/notices on Logstash!
  • And nothing triggered the new "totallyEmpty" case, but just because I had already warned people to fix affected filters

Overall, the script looks ready to go. But first, we have to ask people to fix their filters. I'm tracking this below:

WikiFiltersRequestStatus
arwiki66, 96here
dawiki5here
glwiki15here
mrwiki9here
rowiki67here
ukwiki29, 30, 53here

In the next days I'll reach out to these communities either on IRC or on-wiki to ask them to fix their filters, providing any help they need. In the meanwhile I'll also do some final testing locally, and if no concerns arise we'll be able to go on with https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/AbuseFilter/+/459245/, so that the script will be automatically run upon updating wikis.

In the next days I'll reach out to these communities either on IRC or on-wiki to ask them to fix their filters, providing any help they need.

@Daimona Fixed in arwiki. Can you confirm that?

In the next days I'll reach out to these communities either on IRC or on-wiki to ask them to fix their filters, providing any help they need.

@Daimona Fixed in arwiki. Can you confirm that?

Yes, thanks! Of note, the UI now issues a warning if people try to save a filter with invalid parameters.

Meno25 added a subscriber: Meno25.Feb 11 2019, 12:37 PM

Change 489705 had a related patch set uploaded (by Daimona Eaytoy; owner: Daimona Eaytoy):
[mediawiki/extensions/AbuseFilter@master] Beautify old, broken abuse_filter_history rows

https://gerrit.wikimedia.org/r/489705

@tstarling While we're half done with manual fixing, a new problem showed up... I'd be grateful if you could please review the patch above, merge it, and re-dry-run the script after the remaining filters will have been fixed.

Change 493738 had a related patch set uploaded (by Daimona Eaytoy; owner: Daimona Eaytoy):
[mediawiki/extensions/AbuseFilter@master] Fix faulty query in normalizeThrottleParameters

https://gerrit.wikimedia.org/r/493738

Change 493738 merged by jenkins-bot:
[mediawiki/extensions/AbuseFilter@master] Fix faulty query in normalizeThrottleParameters

https://gerrit.wikimedia.org/r/493738

Change 497110 had a related patch set uploaded (by Daimona Eaytoy; owner: Daimona Eaytoy):
[mediawiki/extensions/AbuseFilter@REL1_32] Remove normalizeThrottleParameters script

https://gerrit.wikimedia.org/r/497110

Change 489705 had a related patch set uploaded (by Daimona Eaytoy; owner: Daimona Eaytoy):
[mediawiki/extensions/AbuseFilter@master] Beautify old, broken abuse_filter_history rows

https://gerrit.wikimedia.org/r/489705

Change 489705 merged by jenkins-bot:
[mediawiki/extensions/AbuseFilter@master] Beautify old, broken abuse_filter_history rows

https://gerrit.wikimedia.org/r/489705

Well, so we're waiting for mr.wiki filter 9 to be fixed. I've been told to contact the only active local admin, although I don't know when they'll be able to fix the filter... Furthermore, the wiki has opted out from global sysop, and thus stewards cannot easily change the filter. I'm unsure if that would be OK to do for someone in the staff group.

mrwiki done! @tstarling You acn dry-run one last time whenever you want, and thanks a lot for your help with this. One last question: is it fine to add the script to update.php so that it's automatically run upon updating, or do we want to run it manually?