Okay, now that we have T152934 resolved, and there is documented support form CUs to have this access, what is the next step? I mean, besides submitting a patch for wikimedia/operations-mediawiki-config. Like do we need legal to review and approve?

For what it's worth, this would *not* need any change to the Privacy Policy, or ANPI policy.

I've been testing the feature on Beta. So far it looks good except for some apparent logging problem which I've reported on T152934.

I have concerns however with regards to data retention. Looking at the patch I don't see (sorry if there's there or elsewhere) how data is purged after 90 days per our privacy policy. If there's no mechanism to do so, we can't enable the feature at production I'm afraid. Scratch that. I've found that the purges are done via purgeOldLogIPData.php which I guess it'll be added to a crontab in our puppet, right? But we need to define first T179131.

I think we can get the $wgAbuseFilterPrivateLog done now as it's kinda of a no-op for now as it just allows the log to work. I'll submit a patch.

Change 409445 had a related patch set uploaded (by MarcoAurelio; owner: MarcoAurelio):
[operations/mediawiki-config@master] Log accessing private abusefilter details

https://gerrit.wikimedia.org/r/409445

Patch above enables the private log but disables anyone being given out the permissions on production wikis.

Ensure the new entries in logging table are not replicated into the public dumps, and are filtered out of the Cloud-Services (or does that happen automatically?)

@bd808 @Marostegui Can you answer that question? Thanks!

In T160357#3959543, @MarcoAurelio wrote:

Ensure the new entries in logging table are not replicated into the public dumps, and are filtered out of the Cloud-Services (or does that happen automatically?)

Entries in the logging table are shown in the Wiki Replicas based on [[https://phabricator.wikimedia.org/source/operations-puppet/browse/production/modules/role/templates/labs/db/views/maintain-views.yaml;da9169bacb465b4db1a3d95006b830a8a4491f23$145-197|a list of allowed logging.log_type values]]. If the log_type is in this list then it will be shown; if it is not then it will be excluded from query results by the view layer. The abusefilter type is currently in that list. I don't see anywhere in this ticket that actually explains what new data will be in the logging table and how we would filter it out.

Dumps is @ArielGlenn's department. I am unsure of where to look to find that out.

We grab entries from the logging table by requesting them as an anonymous user and asking MediaWiki to export them. Anything we aren't permitted to see, won't show up.

• Schedule maintenance/purgeOldLogIPData.php to be run daily

purge_abusefilter.pp is cron'd now to run daily at 01:15 (UTC) everyday on the Wikimedia Puppet - https://gerrit.wikimedia.org/r/#/c/410072/

Get legal to sign off on this task (is it needed?)

I think its probably a good idea to get legal to sign off (Its not that different from checkuser, so it probably will be a straight forward thing for them to look into)

Ensure the new entries in logging table are not replicated into the public dumps

AbuseFilter log has its own table, abuse_filter_log; does it also log anything on logging?

Set $wgAbuseFilterPrivateLog to true

I'm scheduling https://gerrit.wikimedia.org/r/#/c/409445/ for SWAT. It's a NO-OP for now.

In T160357#3975380, @MarcoAurelio wrote:

Ensure the new entries in logging table are not replicated into the public dumps

AbuseFilter log has its own table, abuse_filter_log; does it also log anything on logging?

MariaDB [metawiki_p]> select distinct log_type from logging;
+-------------------+
| log_type          |
+-------------------+
| abusefilter       |
| block             |
| contentmodel      |
| delete            |
| gblblock          |
| gblrename         |
| gblrights         |
| globalauth        |
| import            |
| lock              |
| managetags        |
| massmessage       |
| merge             |
| move              |
| mwoauthconsumer   |
| newusers          |
| notifytranslators |
| pagelang          |
| pagetranslation   |
| patrol            |
| protect           |
| renameuser        |
| review            |
| rights            |
| tag               |
| thanks            |
| timedmediahandler |
| translationreview |
| upload            |
| usermerge         |
+-------------------+
30 rows in set (0.24 sec)

I don't see the abusefilterprivatelogdetails for now. Once https://gerrit.wikimedia.org/r/#/c/409445/ is deployed, I'll check that again.

Change 409445 merged by jenkins-bot:
[operations/mediawiki-config@master] Log accessing private abusefilter details

https://gerrit.wikimedia.org/r/409445

Mentioned in SAL (#wikimedia-operations) [2018-02-15T15:14:45Z] <zfilipin@tin> Synchronized wmf-config/abusefilter.php: SWAT: [[gerrit:409445|Log accessing private abusefilter details (T160357)]] (duration: 01m 12s)

Theres a whitelist i puppet for which log types are allowed on labs. Anything not on the list is not replicated to labs.

The Special:Log/abusefilterprivatedetails is now avalaible. Those in the staff global group should check if accessing private abusefilter log details is properly logged in that log. For now, they're the only ones with access.

@Bawolff I guess you mean maintain-views?

I've filled T187455: Totally exclude 'abusefilterprivatedetails' from 'logging' in public replicas to verify that really that log is not avalaible on public replicas, data dumps, etc.

Added a note on maintain-views.yaml to avoid abusefilterprivatedetails to ever be whitelisted.

@APalmer_WMF we are now ready for a review and blessing from Legal. Would you like me to create a separate task for it?

TL;DR: We want to allow CUs to have the ability to see the IP information that is already stored along with each AbuseFilter log entry. This has been historically not allowed because we did not have a way to log a CU's access to this type of IP information. But now, we have a log for it implemented. Also, CUs already have access to this data through the CheckUser, but only if the log was about the activities of a registered user, or if they already knew the IP address to look up. With this new permission, those AbuseFilter log entries that are caused by, e.g. preventing multiple account creation, can easily lead to an IP that can then be checked on CheckUser to find a potential abuser behind it.

Hi @Huji, thanks for the ping. I don't think we need to create a second ticket. I will review the information here, and get back to you shortly.

Grabbed this because in addition to walking through it with Aeryn tomorrow going to do some production testing from the SuSa side. Once we're all set I'll submit the patch to turn on for CUs etc. (assuming the old data purge stuff is set up? I believe that is now but will check on).

We'll need to cron it daily instead of weekly. @Dzahn has a patch for that already and we also have another task to unify the indexes across the numerous DBs.

The situation before was to run daily. Since there were issues with it i temp. reduced it to weekly. Then "Reedy fixed it(tm)" happened and i simply reverted to how it was before just now. So it's back to daily.

https://gerrit.wikimedia.org/r/#/q/fc41cb753c1589e48c731030aba0d8d6d133fdbf

Met with @Jalexander to discuss. Approved by Legal for rollout once SuSa has tested and approved.

We're pretty much good, done a bit of testing and will do a bit more but should be ready to roll out a patch to give to CUs on Tuesday (US Holiday on Monday and I'd rather not launch with folks either on their weekend or going to it in case there are issues/questions).

@Huji @MarcoAurelio Do you know if there is any documentation created for this somewhere? Perhaps with screenshots? We can create that if needed (I'll probably want to create some for the SuSa team anyway) but might push the rollout back to Thursdayish. We'll want to send out announcements to the CUs/Ombudsman/Stewards and having the documentation would be really helpful. I'm happy to do that announcement or to let either of you as desired :).

Don't forget assigning the permission to view the new log to ombdusmen and stewards as well :)

@Jalexander I can create some screenshots in the next 2-3 days. I will also look to see what pages on Meta/MediaWiki wiki need to be updated. Lastly, I will prepare a piece of test to be emailed on the CU mailing list. and get it to you for approval

@Jalexander: Here is a piece of text + some screenshots.

Imagine a filter exists that prevent creation of more than 3 accounts per day by the same IP address. A user tries to create a fourth account from his IP, and is stopped by the filter:

Here "TestAccount" is the account name that person used and failed to create. Running a check user on this account name gives you an error because the account does not exist:

Up until now, there was no way to retrieve any CheckUser information about this event. With the new permission, CheckUsers can retrieve the private details associated with the AbuseFilter log entry:

Currently the private information only includes an IP address:

CheckUsers can then run a check on that IP address if necessary. Every time a CheckUser accesses the private information of an AbuseLog entry, this action itself is stored in the logs, so other CheckUsers can be aware of it and ensure that accessing private data is only conducted in accordance with the Privacy Policy and the CheckUser policy:

@Jalexander I also reviewed existing documentation:

• m:CheckUser policy already states that the CheckUser tool can be used to "Determine from which IPs a user has edited or done a logged action or password reset on the Wikimedia wiki". Since the bold part already exists, and this new feature is just extension of what logged actions can be investigated by the CheckUsers, no update is necessary.
  • Recall that this same statement also applies to when a CheckUser is run for all other types of logged actions (e.g. page deletion, user right adjustment, etc.)
• mw:Extension:CheckUser does not seem to intend to provide details of how the tool works, but it did mention the "Get Edits" feature. There, it only described that it retrieves a user's edits; this is wrong, even before the current task, non-edit actions (like page deletions, user right adjustments, etc.) were also shown through Get Edits. I updated the page to indicate that logged actions are also mentioned.

In sum, no further documentation change is necessary. The text and screenshots above can be sent to cu-l listserv as an educational email.

Add the abusefilter-private and abusefilter-private-log rights to the set of rights held by the checkuser group
Add the abusefilter-private and abusefilter-private-log rights to the set of rights held by the ombudsmen group
Add the abusefilter-private-log right to the set of rights held by the steward global group
Add the abusefilter-private right to the set of rights held by the steward local group at Meta-Wiki only.

I'll take care of all of this (1 and 4 in a patch, 2 & 3 via Meta-Wiki); all when we have the final placet from the WMF.

This is approved from the Trust & Safety side now (and hence the WMF). @MarcoAurelio is doing the global changes now and submitting the local patch which I'll shepherd though SWAT this afternoon.

For now I think I'll use @Huji's images and put up some documentation on CUwiki. We might want to think about some comments on the AbuseFilter extension too though that isn't urgent.

• Ombudsmen and Staff: abusefilter-private and abusefilter-private-log added
• Stewards: abusefilter-private-log added
• Checkusers: patch incoming.
• Stewards: patch incoming

Change 413062 had a related patch set uploaded (by MarcoAurelio; owner: MarcoAurelio):
[operations/mediawiki-config@master] Allow CheckUsers and Stewards to access private data from the AbuseLog

https://gerrit.wikimedia.org/r/413062

Change 413062 merged by jenkins-bot:
[operations/mediawiki-config@master] Allow CheckUsers and Stewards to access private data from the AbuseLog

https://gerrit.wikimedia.org/r/413062

Mentioned in SAL (#wikimedia-operations) [2018-02-21T00:43:31Z] <thcipriani@tin> Synchronized wmf-config/abusefilter.php: SWAT: [[gerrit:413062|Allow CheckUsers and Stewards to access private data from the AbuseLog]] T160357 (duration: 01m 12s)

I get "Originating IP address Not Available" on Wikispecies, when trying to use this function, and it doesnt come up in the Check user log.

In T160357#3988173, @Dan_Koehl wrote:

I get "Originating IP address Not Available" on Wikispecies, when trying to use this function, and it doesnt come up in the Check user log.

I was able to get it to work on https://species.wikimedia.org/wiki/Special:AbuseLog/968 (chosen since the IP was public anyway) Is there a specific one where you got that message?

Please note that abuselog entries older than 90 days will not have any private data attached as per our privacy policy. The access log is not kept in the checkuser log but in a sublog at Special:Log.

There are a few permission that should still be added to Ombudsman group [1], so they can see everything correctly. It was reported that one of them can't see the details of a log.

I would add:

• abusefilter-hidden-log
• abusefilter-log
• abusefilter-log-detail
• abusefilter-log-private

I could add if none opposes that.

[1] - https://meta.wikimedia.org/wiki/Special:GlobalGroupPermissions/ombudsman
These permission above are only for viewing and are already available to stewards.

As a former Ombudsman, I think what @Teles suggests is appropriate and should be done.

In T160357#3992683, @Teles wrote:

There are a few permission that should still be added to Ombudsman group [1], so they can see everything correctly. It was reported that one of them can't see the details of a log.

I would add:

• abusefilter-hidden-log
• abusefilter-log
• abusefilter-log-detail
• abusefilter-log-private

I could add if none opposes that.

[1] - https://meta.wikimedia.org/wiki/Special:GlobalGroupPermissions/ombudsman
These permission above are only for viewing and are already available to stewards.

As I said to Patrik I'm ok with this for OS and would actually add abusefilter-view and abusefilter-view-private as well because I think it could be important for the Ombuds to see the underlying filter as well.

Done: 21:13, 22 February 2018 MarcoAurelio (talk | contribs | block) changed global group permissions for Special:GlobalUsers/ombudsman: added abusefilter-log-detail, abusefilter-view, abusefilter-log, abusefilter-view-private, abusefilter-log-private, abusefilter-hidden-log; removed (none) (expand per phab:T160357#3994200) - hope that's alright now.