Page MenuHomePhabricator

Devise a process for finding and fixing filters that will be affected by changes in AbuseFilter
Open, NormalPublic

Description

Every so often, we need to make a change in AbuseFilter code that could potentially affect the meaning of existing filters. For instance:

  • In T191715 we learned that some system variable names were not properly reserved by AbuseFilter, and before we fix it we need to identify all filters on WMF wikis that use a homonym local variable and change their code to use a different name for that local variable.
  • In T181024 we found out that arrays may be improperly cast to strings, and before fixing the code we want to fix the filters that use this incorrect approach.
  • A similar issue was also observed in T190639 about the way string() function is used by filters to cast lists into strings. Again, we would like to fix the filters that use that function before updating the code.
  • In T187973 we want to find all filters that use deprecated variables and replace the variable names with their non-deprecated alternatives.

For each of these we would like to run a query against production databases (for all WMF wikis, possibly one wiki at a time) to identify filters that need to be modified, and then use a global sysop account to go into each and every one of these filters and apply a fix.

To that end, we need to have a very clear process for the following:

  1. To decide when each of those steps is worth doing. The alternative is to not find/modify the affected filters, and let the local admins do it. Each approach has cons and pros, and we need to devise a guideline to decide which approach to use when.
  2. To request for the queries to be run (who asks it, who runs it, does it need legal approval each time, is there a Phabricator tag to use, etc)
  3. To request for the affected filters to be edited (who does that, do we ask one of the existing global sysops or do we temporarily promote one of the developers to a global sysop each time)
  4. To communicate and document the filter modifications (the last thing we want is for the local sysops to see a random person come and change something with their filters and freak out)

Since Brian and Aeryn were (briefly) engaged with some of the tasks enumerated above, I have copied them here as well.

Event Timeline

Huji created this task.Apr 11 2018, 2:30 PM
Restricted Application added subscribers: Scoopfinder, Aklapper. · View Herald TranscriptApr 11 2018, 2:30 PM
Huji updated the task description. (Show Details)Apr 11 2018, 2:30 PM
Huji added a subscriber: Krinkle.Apr 11 2018, 2:35 PM

Of note, @Krinkle has proposed in T187973#3994713 that we should use the tag #wikimedia-site-requests for the query task and he believes that "the task of a global sysop performing on-wiki changes is outside the scope for any Phabricator task" which leaves me unsure as to how to manage that step.

As a side note, I tried to search for existing examples (already done), but I could only find T140791.

Huji added a subscriber: kaldari.Apr 11 2018, 4:16 PM

Good work! Of note, in T140791 @kaldari elects to post results in a private manner because some filters are private. I assume the results included more than just the filter ID?

Huji triaged this task as Normal priority.Apr 11 2018, 4:16 PM
Huji updated the task description. (Show Details)

Yeah, probably the results included the pattern as well. I think the needed queries, if they only return IDs, shouldn't be private.

I had some thoughts about the error-fixing part. What if we create a maintenance account (like "AbuseFilterManager"), with shared login and global sysop rights, in order to keep all contribs on the same account and maybe provide a userpage with some lines of explanation?

I was told you could also tag #DBA but I think #wikimedia-site-requests probably makes more sense. One issue here we currently have to do this manually, wiki by wiki (I think?). I'm not sure if there's a script to run a query across all wikis, but if there isn't, we should create one!

Created as T193894. I saw this a bit too late but added site-requests as well. Of course a script should be used for this, but I don't think it should be too hard to create one.

Huji added a comment.Jul 27 2018, 6:25 PM

Of note, @Krinkle has proposed in T187973#3994713 that we should use the tag #wikimedia-site-requests for the query task and he believes that "the task of a global sysop performing on-wiki changes is outside the scope for any Phabricator task" which leaves me unsure as to how to manage that step.

As for the latter part, I have made a request to become a global sysop. I think having a global sysop that is extensively familiar with AbuseFilter and is dedicated to the filter cleanup can tremendously boost the process of cross-wiki filter updates