Page MenuHomePhabricator

Work out what global/cross-wiki user rights changed while logging was broken
Closed, ResolvedPublic

Description

The global rights changes (CentralAuth) and cross-wiki rights changes (MW core) systems rely on separate objects that are like User but called UserRightsProxy and CentralAuthGroupMembershipProxy to carry out rights changes. They basically rely on the Special:UserRights code in core to do a lot of the work, like calling addGroup/removeGroup and logging and things.
Global user rights (and to a slightly lesser extent cross-wiki user rights changes) are important - they manage things like staff, steward and system administrator access on-wiki, often covering things restricted by https://wikimediafoundation.org/wiki/Access_to_nonpublic_data_policy.

https://gerrit.wikimedia.org/r/#/c/180328/ started requiring User::addGroup to return a boolean true before changes would be logged, but these other classes did not get updated. This was deployed to Wikimedia wikis as part of 1.25wmf16 and hit metawiki (where stewards tend to use these functions) on the 10th of February (aka, yesterday) at 20:28 UTC, as part of group1.

A user (Ajraddatz) noted in https://meta.wikimedia.org/?diff=11227546 that he'd made a global user rights change but it did not get logged. I noticed this and a couple of us talked about it in #wikimedia-stewards - @hoo worked out why it was happening and came up with a fix, which I've reviewed and deployed (while Greg was unavailable - https://gerrit.wikimedia.org/r/#/c/189888/ and https://gerrit.wikimedia.org/r/#/c/189879/). We've since tested that the fix worked, but now we need to work out what changed that should have been logged.

For global user rights changes we just need to know what changed in centralauth.global_user_groups since 20:28 UTC 2015-02-10, but for cross-wiki rights I think we'd need to go through each wiki's local user_groups table changes since then and manually filter out changes that were made and logged locally (i.e., not changes made via meta).

Event Timeline

Krenair created this task.Feb 11 2015, 2:31 AM
Krenair assigned this task to Springle.
Krenair raised the priority of this task from to Needs Triage.
Krenair updated the task description. (Show Details)
Krenair added subscribers: Krenair, greg, hoo and 3 others.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 11 2015, 2:31 AM
Springle added a comment.EditedFeb 11 2015, 3:13 AM

03:06 < hoo> springle: Should be ok to ahve them public if they're only affecting that table

Found 5 writes to centralauth.global_user_groups since 2015-02-10 20:28:00:

REPLACE /* CentralAuthUser::addToGlobalGroups Ajraddatz */ INTO `global_user_groups` (gug_user,gug_group) VALUES ('12903269','OTRS-member')
/*!*/;
--
REPLACE /* CentralAuthUser::addToGlobalGroups Hoo man */ INTO `global_user_groups` (gug_user,gug_group) VALUES ('7684853','apihighlimits-requestor')
/*!*/;
--
DELETE /* CentralAuthUser::removeFromGlobalGroups Hoo man */ FROM `global_user_groups` WHERE gug_user = '7684853' AND gug_group = 'apihighlimits-requestor'
/*!*/;
--
REPLACE /* CentralAuthUser::addToGlobalGroups Ajraddatz */ INTO `global_user_groups` (gug_user,gug_group) VALUES ('35673140','OTRS-member')
/*!*/;
--
DELETE /* CentralAuthUser::removeFromGlobalGroups Ajraddatz */ FROM `global_user_groups` WHERE gug_user = '35673140' AND gug_group = 'OTRS-member'
/*!*/;

user_groups will need some light scripting...

Okay, that matches the test logs that were made after I deployed a fix + the one we knew was missing (@hoo has since manually logged that for Ajraddatz, though it has the wrong timestamp I think).

S7 wikis:

-- arwiki
-- cawiki
-- eswiki
-- fawiki
INSERT /* User::addGroup Mirzapouri */ IGNORE INTO `user_groups` (ug_user,ug_group) VALUES ('293976','uploader')
/*!*/;
-- frwiktionary
-- hewiki
-- huwiki
-- kowiki
-- metawiki
INSERT /* User::addGroup Jalexander-WMF */ IGNORE INTO `user_groups` (ug_user,ug_group) VALUES ('7293923','sysop')
/*!*/;
--
INSERT /* User::addGroup Jalexander-WMF */ IGNORE INTO `user_groups` (ug_user,ug_group) VALUES ('4334568','sysop')
/*!*/;
-- rowiki
-- ukwiki
-- viwiki

S6 wikis:

-- frwiki
-- jawiki
-- ruwiki
INSERT /* User::addGroup Дмитрий Ромашов */ IGNORE INTO `user_groups` (ug_user,ug_group) VALUES ('1287991','uploader')
/*!*/;
--
INSERT /* User::addGroup Игорь Петров СП... */ IGNORE INTO `user_groups` (ug_user,ug_group) VALUES ('1526892','uploader')
/*!*/;
--
INSERT /* User::addGroup БЕЛОРУС и Я */ IGNORE INTO `user_groups` (ug_user,ug_group) VALUES ('1291227','uploader')
/*!*/;
INSERT /* User::addGroup Jwh */ IGNORE INTO `user_groups` (ug_user,ug_group) VALUES ('567134','uploader')
/*!*/;

S7 and S6 entries - all local changes which were logged, great.

Springle added a comment.EditedFeb 11 2015, 4:02 AM

Nothing in S5 or S4 wikis:

-- dewiki
-- wikidatawiki
-- commonswiki

S3 is going to take a while. Or I'll improve my script...

S2 wikis:

-- bgwiki
-- bgwiktionary
-- cswiki
INSERT /* User::addGroup Tchoř */ IGNORE INTO `user_groups` (ug_user,ug_group) VALUES ('203299','autopatrolled')
/*!*/;
--
INSERT /* User::addGroup Tchoř */ IGNORE INTO `user_groups` (ug_user,ug_group) VALUES ('17452','autopatrolled')
/*!*/;
--
INSERT /* User::addGroup Tchoř */ IGNORE INTO `user_groups` (ug_user,ug_group) VALUES ('228759','autopatrolled')
/*!*/;
-- enwikiquote
-- enwiktionary
-- eowiki
-- fiwiki
-- idwiki
-- itwiki
-- l10nwiki
-- nlwiki
-- nowiki
-- plwiki
-- ptwiki
INSERT /* User::addGroup Ruy Pugliesi */ IGNORE INTO `user_groups` (ug_user,ug_group) VALUES ('968960','autoreviewer')
/*!*/;
--
INSERT /* User::addGroup Zoldyick */ IGNORE INTO `user_groups` (ug_user,ug_group) VALUES ('1184012','autoreviewer')
/*!*/;
-- svwiki
-- thwiki
-- trwiki
INSERT /* User::addGroup Kibele */ IGNORE INTO `user_groups` (ug_user,ug_group) VALUES ('63386','autoreview')
/*!*/;
--
INSERT /* User::addGroup Kibele */ IGNORE INTO `user_groups` (ug_user,ug_group) VALUES ('572993','autoreview')
/*!*/;
INSERT /* User::addGroup Kibele */ IGNORE INTO `user_groups` (ug_user,ug_group) VALUES ('579915','autoreview')
/*!*/;
--
INSERT /* User::addGroup Kibele */ IGNORE INTO `user_groups` (ug_user,ug_group) VALUES ('444255','autoreview')
/*!*/;
-- zhwiki
DELETE /* UserRightsProxy::removeGroup Avraham */ FROM `user_groups` WHERE ug_user = '137629' AND ug_group = 'checkuser'
/*!*/;

S1 enwiki:

-- enwiki
DELETE /* User::removeGroup Cenarium */ FROM `user_groups` WHERE ug_user = '24062666' AND ug_group = 'rollbacker'
/*!*/;
--
DELETE /* User::removeGroup Cenarium */ FROM `user_groups` WHERE ug_user = '24062666' AND ug_group = 'confirmed'
/*!*/;
INSERT /* User::addGroup Gilliam */ IGNORE INTO `user_groups` (ug_user,ug_group) VALUES ('5450916','rollbacker')
/*!*/;
--
DELETE /* User::removeGroup 28bytes */ FROM `user_groups` WHERE ug_user = '9356143' AND ug_group = 'sysop'
/*!*/;

No writes to user_groups on any S3 wiki within the time period.

Springle reassigned this task from Springle to Krenair.Feb 11 2015, 4:09 AM
Springle set Security to None.
Springle added a subscriber: Springle.
Alexia removed a subscriber: Alexia.Feb 11 2015, 4:23 AM

I made a mistake hacking the script to make S3 faster.

The true S3 results are:

-- anwiktionary
DELETE /* UserRightsProxy::removeGroup Rschen7754 */ FROM `user_groups` WHERE ug_user = '8' AND ug_group = 'sysop'
/*!*/;
-- srwikisource
INSERT /* User::addGroup Dungodung */ IGNORE INTO `user_groups` (ug_user,ug_group) VALUES ('18','sysop')
/*!*/;
-- testwiki
INSERT /* UserRightsProxy::addGroup Hoo man */ IGNORE INTO `user_groups` (ug_user,ug_group) VALUES ('14788','sysop')
/*!*/;
DELETE /* UserRightsProxy::removeGroup Hoo man */ FROM `user_groups` WHERE ug_user = '14788' AND ug_group = 'sysop'
/*!*/;

Timestamps for UserRightsProxy hits:

anwiktionary UserRightsProxy::removeGroup Rschen7754
2015-02-11 03:56:31

testwiki UserRightsProxy::addGroup Hoo man
2015-02-11 01:57:32

textwiki UserRightsProxy::removeGroup Hoo man
2015-02-11 03:56:31

zhwiki UserRightsProxy::removeGroup Avraham
2015-02-11 01:45:04

Krenair reassigned this task from Krenair to hoo.Feb 11 2015, 4:36 AM

Thanks @Springle!

The S3 entries were made after I deployed the fix, and got logged on meta just fine.

REPLACE /* CentralAuthUser::addToGlobalGroups Ajraddatz */ INTO `global_user_groups` (gug_user,gug_group) VALUES ('12903269','OTRS-member')

2015-02-10 23:48:15 (from IRC) - already manually logged by @hoo, but with the wrong timestamp, please fix

DELETE /* UserRightsProxy::removeGroup Avraham */ FROM `user_groups` WHERE ug_user = '137629' AND ug_group = 'checkuser'

2015-02-11 01:45:04 - target User:Bencmq@zhwiki, needs logging

Assigning to @hoo so he can handle these, then this can be closed.

FYI: Same bug now occurring again with interwiki rights changes. https://meta.wikimedia.org/wiki/Special:UserRights/Bencmq@zhwiki is an example.

Krenair added a comment.EditedFeb 11 2015, 10:44 AM

Did you perform a rights change against that user since I deployed the fix? It doesn't look like it. Pretty sure we already caught a change you made to that user before the fix, please read the previous comments.

Aklapper triaged this task as High priority.Feb 11 2015, 12:26 PM

Potentially not, it was reported by another steward and it looked like it happened after the fix. I can't replicate as I did a rights change since and it worked fine.

Sigh. How many rights changes have you made against User:Bencmq@zhwiki within the past two days?

AFAIK only one: namely the removal of CU rights from Bencmq@zhwiki. He had them since March 2013 and now he doesn't have them anymore though it isn't logged. When will this be fixed? (and maybe for more rights removals, or were the others already done?)

Rights, that's what we thought. Please read T89205#1029906

Rights, that's what we thought. Please read T89205#1029906

Yeah, I saw that, but it doesn't give me the answer about when it will be fixed. ;-)

Krenair claimed this task.Feb 12 2015, 2:19 PM
REPLACE /* CentralAuthUser::addToGlobalGroups Ajraddatz */ INTO `global_user_groups` (gug_user,gug_group) VALUES ('12903269','OTRS-member')

2015-02-10 23:48:15 (from IRC) - already manually logged by @hoo, but with the wrong timestamp, please fix

DELETE /* UserRightsProxy::removeGroup Avraham */ FROM `user_groups` WHERE ug_user = '137629' AND ug_group = 'checkuser'

2015-02-11 01:45:04 - target User:Bencmq@zhwiki, needs logging

I just went ahead and fixed these myself.

Krenair closed this task as Resolved.Feb 12 2015, 2:19 PM