Page MenuHomePhabricator

Some administrators can no longer add/remove users from the autochecked/editor user groups
Closed, ResolvedPublicBUG REPORT

Description

I and two further dewiki admins (LexICon, Hgzh) and a bureaucrat (Itti) can no longer assign users to or remove users from the autochecked/editors ("passiver Sichter"/"Sichter") user groups. Those groups are suddenly listed under "Groups you cannot change" for us. It works fine for at least some other administrators. I first noticed this today. It was working fine a few days ago.

Dewiki discussion

Steps to Reproduce:

Actual Results:

  • The groups are listed under "Groups you cannot change" and grayed out.

Expected Results:

  • The groups should be listed under "Groups you can change" and changable.

Event Timeline

Reedy renamed this task from Some administrators can no longer add/remove users from the autchecked/editor user groups to Some administrators can no longer add/remove users from the autochecked/editor user groups.Jan 29 2021, 3:43 PM

Adding editor or autoreview rights using the Action API (action=userrights) does not work either, while adding ip-block-exempt using the API works fine, just like what the UI promises.

Trying to add editor or autoreview rights rights results in the same JSON as when trying to add sysop rights, no special error message:

{
    "userrights": {
        "user": "CountCountSocke",
        "userid": 3343530,
        "added": [],
        "removed": []
    }
}

I played with this interesting bug for a while, and I noticed the following:

I tried to give my sock +sysop at test2wiki, and I am unable to change anything but IPBE, despite I should be able to grant Editors and Autochecked users, too. Interface looks like this:

However:

>>> \MediaWiki\MediaWikiServices::getInstance()->getUserGroupManager()->getGroupsChangeableBy(User::newFromName('Martin Urbanec (test)'))
=> [
     "add" => [
       "ipblock-exempt",
       "editor",
       "autoreview",
     ],
     "remove" => [
       "ipblock-exempt",
       "editor",
       "autoreview",
     ],
     "add-self" => [],
     "remove-self" => [],
   ]
>>>

I changed the code at mwdebug1003, and added wfDebug( __METHOD__ . ": res: " . print_r( $res, true) ); to SpecialUserrights::changeableGroups. The verbose logs say:

2021-01-29 16:34:57 [YBQ5MOgUPfJ@mF7lXPpFfQAAAAs] mwdebug1003 test2wiki 1.36.0-wmf.28 wfDebug DEBUG: UserrightsPage::changeableGroups: res: Array
(
    [add] => Array
        (
            [0] => ipblock-exempt
        )

    [remove] => Array
        (
            [0] => ipblock-exempt
        )

    [add-self] => Array
        (
        )

    [remove-self] => Array
        (
        )

)

It sounds FlaggedRevs might be having troubles adding its rights to core?

To look for why that happens, I added a couple of other debug lines to SpecialUserrights:

$ugm = \MediaWiki\MediaWikiServices::getInstance()->getUserGroupManager();
wfDebug( __METHOD__ . ": options: addGroups: " . print_r( $ugm->options->get( 'AddGroups' ), true ) ); // I made options public at mwdebug1003
$config = \MediaWiki\MediaWikiServices::getInstance()->getMainConfig();
wfDebug( __METHOD__ . ": config: addGroups: " . print_r( $config->get( 'AddGroups' ), true ) );

This is what got logged:

2021-01-29 17:25:25 [YBRFBLlESy6xbLxR9peOFwAAAEo] mwdebug1003 test2wiki 1.36.0-wmf.28 wfDebug DEBUG: UserrightsPage::changeableGroups: options: addGroups: Array                                                     (                                                                                                                                                                                                                        [sysop] => Array                                                                                                                                                                                                         (                                                                                                                                                                                                                        [0] => ipblock-exempt                                                                                                                                                                                            )                                                                                                                                                                                                                                                                                                                                                                                                                                     [bureaucrat] => Array                                                                                                                                                                                                    (                                                                                                                                                                                                                        [0] => flow-bot                                                                                                                                                                                                      [1] => import                                                                                                                                                                                                        [2] => transwiki
            [3] => accountcreator
            [4] => sysop
            [5] => interface-admin
            [6] => bureaucrat
            [7] => bot
            [8] => confirmed
            [9] => copyviobot
        )

    [qa_automation] => Array
        (
            [0] => qa_automation
        )

)
2021-01-29 17:25:25 [YBRFBLlESy6xbLxR9peOFwAAAEo] mwdebug1003 test2wiki 1.36.0-wmf.28 wfDebug DEBUG: UserrightsPage::changeableGroups: config: addGroups: Array
(
    [sysop] => Array
        (
            [0] => ipblock-exempt
            [1] => editor
            [2] => autoreview
        )

    [bureaucrat] => Array
        (
            [0] => flow-bot
            [1] => import
            [2] => transwiki
            [3] => accountcreator
            [4] => sysop
            [5] => interface-admin
            [6] => bureaucrat
            [7] => bot
            [8] => confirmed
            [9] => copyviobot
            [10] => reviewer
        )

    [qa_automation] => Array
        (
            [0] => qa_automation
        )

)

I then verified in ServiceOptions that it indeed gets malformed GlobalVarConfig. The key question is how... Posting to share investigation made so far. This looks like a very interesting bug :).

Change 660006 had a related patch set uploaded (by Reedy; owner: Reedy):
[mediawiki/core@master] Revert "Remove usages and hard deprecate User::changeable(By)Group"

https://gerrit.wikimedia.org/r/660006

Change 660007 had a related patch set uploaded (by Reedy; owner: Reedy):
[mediawiki/core@wmf/1.36.0-wmf.28] Revert "Remove usages and hard deprecate User::changeable(By)Group"

https://gerrit.wikimedia.org/r/660007

Change 660007 merged by jenkins-bot:
[mediawiki/core@wmf/1.36.0-wmf.28] Revert "Remove usages and hard deprecate User::changeable(By)Group"

https://gerrit.wikimedia.org/r/660007

Majavah triaged this task as Unbreak Now! priority.Jan 29 2021, 10:42 PM
Reedy lowered the priority of this task from Unbreak Now! to High.Jan 29 2021, 10:47 PM
Reedy added a project: Platform Engineering.

I then verified in ServiceOptions that it indeed gets malformed GlobalVarConfig. The key question is how... Posting to share investigation made so far. This looks like a very interesting bug :).

Hm.. Maybe UserGroupManager somehow gets initialized before extension registration is fully completed - then it would get not the full state of globals, but that would be a pretty brutal bug.. Would be interesting to put log a trace from ServiceWiring 'UserGroupManager' initializer..

Urbanecm raised the priority of this task from High to Unbreak Now!.Jan 30 2021, 2:55 PM

This actively blocks wmf.28 and wmf.29 to roll forward, marking as UBN.

FlaggedRevs is unusual since it modifies wgAddGroup/wgRemoveGroup and other relevant config via a $wgExtensionFunction. Extension functions run quite late in MW initialization sequence, so my running theory is that a UserGroupManager was already initialized by the time FlaggedRevs extension function runs, thus already has a copy of the config before FlaggedRevs modifies it. I've recently made a few other services depend on UserGroupManager, potentially moving the initialization earlier, plus the change to make User::changeableGroups depend on UserGroupManager has made the problem visible. This doesn't happen in a vanilla MW with only FlaggedRevs installed locally, so some other extension might be interfering. To confirm the theory we need to log from ServiceWiring UserGroupManager initialization callback and from config/flaggedrevs extension function, and see what comes first in production.

The correct fix, imho, would be to stop using extension function to modify globals in FlaggedRevs. the comment in that code is

// Configs that must be set after FlaggedRevsHooks::onExtensionFunctions for some reason???

added by @matmarex in https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/627332 where part of the extension function was moved out. Originally it the config was moved into an extension function by @Reedy in https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/518396 to workaround yet another extension registration issue, when the permissions were not unset properly, cause they were reset back by extension registration. Fun!

If we wanna pile a workaround on top of a workaround, we could add a new implementation of ServiceOptions - DynamicServiceOptions, which would consult real globals on every call instead of making a copy. But that's probably a nuclear option of last resort.

I then verified in ServiceOptions that it indeed gets malformed GlobalVarConfig. The key question is how... Posting to share investigation made so far. This looks like a very interesting bug :).

Hm.. Maybe UserGroupManager somehow gets initialized before extension registration is fully completed - then it would get not the full state of globals, but that would be a pretty brutal bug.. Would be interesting to put log a trace from ServiceWiring 'UserGroupManager' initializer..

This sounds about right. I tried (and failed) to reproduce the bug at localhost. Hence, I moved test2wiki only to wmf.28 at mwdebug1003, and added three lines to ServiceWiring's UserGroupManager initialization:

global $wgAddGroups;
var_dump( $wgAddGroups );
throw new \MWException( 'Test livehacking T273296' );  // To get a stack trace

This is what I got:

array(3) { ["sysop"]=> array(1) { [0]=> string(14) "ipblock-exempt" } ["bureaucrat"]=> array(10) { [0]=> string(8) "flow-bot" [1]=> string(6) "import" [2]=> string(9) "transwiki" [3]=> string(14) "accountcreator" [4]=> string(5) "sysop" [5]=> string(15) "interface-admin" [6]=> string(10) "bureaucrat" [7]=> string(3) "bot" [8]=> string(9) "confirmed" [9]=> string(10) "copyviobot" } ["qa_automation"]=> array(1) { [0]=> string(13) "qa_automation" } } array(3) { ["sysop"]=> array(1) { [0]=> string(14) "ipblock-exempt" } ["bureaucrat"]=> array(10) { [0]=> string(8) "flow-bot" [1]=> string(6) "import" [2]=> string(9) "transwiki" [3]=> string(14) "accountcreator" [4]=> string(5) "sysop" [5]=> string(15) "interface-admin" [6]=> string(10) "bureaucrat" [7]=> string(3) "bot" [8]=> string(9) "confirmed" [9]=> string(10) "copyviobot" } ["qa_automation"]=> array(1) { [0]=> string(13) "qa_automation" } }

[YBV1Gq25u4xs@pD9OYkZ1gAAABI] /wiki/Special:UserRights/Martin_Urbanec_(test) MWException: Test livehacking T273296

Backtrace:

from /srv/mediawiki/php-1.36.0-wmf.28/includes/ServiceWiring.php(1421)
#0 /srv/mediawiki/php-1.36.0-wmf.28/vendor/wikimedia/services/src/ServiceContainer.php(447): Wikimedia\Services\ServiceContainer->{closure}(MediaWiki\MediaWikiServices)
#1 /srv/mediawiki/php-1.36.0-wmf.28/vendor/wikimedia/services/src/ServiceContainer.php(416): Wikimedia\Services\ServiceContainer->createService(string)
#2 /srv/mediawiki/php-1.36.0-wmf.28/includes/MediaWikiServices.php(255): Wikimedia\Services\ServiceContainer->getService(string)
#3 /srv/mediawiki/php-1.36.0-wmf.28/includes/MediaWikiServices.php(1456): MediaWiki\MediaWikiServices->getService(string)
#4 /srv/mediawiki/php-1.36.0-wmf.28/includes/ServiceWiring.php(1029): MediaWiki\MediaWikiServices->getUserGroupManager()
#5 /srv/mediawiki/php-1.36.0-wmf.28/vendor/wikimedia/services/src/ServiceContainer.php(447): Wikimedia\Services\ServiceContainer->{closure}(MediaWiki\MediaWikiServices)
#6 /srv/mediawiki/php-1.36.0-wmf.28/vendor/wikimedia/services/src/ServiceContainer.php(416): Wikimedia\Services\ServiceContainer->createService(string)
#7 /srv/mediawiki/php-1.36.0-wmf.28/includes/MediaWikiServices.php(255): Wikimedia\Services\ServiceContainer->getService(string)
#8 /srv/mediawiki/php-1.36.0-wmf.28/includes/MediaWikiServices.php(1183): MediaWiki\MediaWikiServices->getService(string)
#9 /srv/mediawiki/php-1.36.0-wmf.28/includes/ServiceWiring.php(149): MediaWiki\MediaWikiServices->getPermissionManager()
#10 /srv/mediawiki/php-1.36.0-wmf.28/vendor/wikimedia/services/src/ServiceContainer.php(447): Wikimedia\Services\ServiceContainer->{closure}(MediaWiki\MediaWikiServices)
#11 /srv/mediawiki/php-1.36.0-wmf.28/vendor/wikimedia/services/src/ServiceContainer.php(416): Wikimedia\Services\ServiceContainer->createService(string)
#12 /srv/mediawiki/php-1.36.0-wmf.28/includes/MediaWikiServices.php(255): Wikimedia\Services\ServiceContainer->getService(string)
#13 /srv/mediawiki/php-1.36.0-wmf.28/includes/MediaWikiServices.php(552): MediaWiki\MediaWikiServices->getService(string)
#14 /srv/mediawiki/php-1.36.0-wmf.28/extensions/CentralAuth/includes/ServiceWiring.php(11): MediaWiki\MediaWikiServices->getAuthManager()
#15 /srv/mediawiki/php-1.36.0-wmf.28/vendor/wikimedia/services/src/ServiceContainer.php(447): Wikimedia\Services\ServiceContainer->{closure}(MediaWiki\MediaWikiServices)
#16 /srv/mediawiki/php-1.36.0-wmf.28/vendor/wikimedia/services/src/ServiceContainer.php(416): Wikimedia\Services\ServiceContainer->createService(string)
#17 /srv/mediawiki/php-1.36.0-wmf.28/includes/MediaWikiServices.php(255): Wikimedia\Services\ServiceContainer->getService(string)
#18 /srv/mediawiki/php-1.36.0-wmf.28/extensions/CentralAuth/includes/CentralAuthServices.php(20): MediaWiki\MediaWikiServices->getService(string)
#19 /srv/mediawiki/php-1.36.0-wmf.28/extensions/CentralAuth/includes/CentralAuthUtils.php(11): CentralAuthServices::getUtilityService(NULL)
#20 /srv/mediawiki/php-1.36.0-wmf.28/extensions/CentralAuth/includes/CentralAuthUtils.php(121): CentralAuthUtils::getUtilityService()
#21 /srv/mediawiki/php-1.36.0-wmf.28/extensions/CentralAuth/includes/session/CentralAuthSessionProvider.php(129): CentralAuthUtils::getCentralSessionById(string)
#22 /srv/mediawiki/php-1.36.0-wmf.28/includes/session/SessionManager.php(488): CentralAuthSessionProvider->provideSessionInfo(WebRequest)
#23 /srv/mediawiki/php-1.36.0-wmf.28/includes/session/SessionManager.php(214): MediaWiki\Session\SessionManager->getSessionInfoForRequest(WebRequest)
#24 /srv/mediawiki/php-1.36.0-wmf.28/includes/WebRequest.php(827): MediaWiki\Session\SessionManager->getSessionForRequest(WebRequest)
#25 /srv/mediawiki/php-1.36.0-wmf.28/includes/session/SessionManager.php(136): WebRequest->getSession()
#26 /srv/mediawiki/php-1.36.0-wmf.28/includes/Setup.php(720): MediaWiki\Session\SessionManager::getGlobalSession()
#27 /srv/mediawiki/php-1.36.0-wmf.28/includes/WebStart.php(89): require_once(string)
#28 /srv/mediawiki/php-1.36.0-wmf.28/index.php(44): require(string)
#29 /srv/mediawiki/w/index.php(3): require(string)
#30 {main}

The most important part is that $wgAddGroups is indeed copied before FlaggedRevs has a chance to get inicialized. Thanks to the stacktrace, it's easy to answer why:

  1. Setup.php invokes SessionManager before FlaggedRevs's permissions get incialized, which (thanks to being tied up by CentralAuth) consults CentralAuthUtilityService, which gets inicialized
  2. CentralAuthUtilityService requires AuthManager, which requires PermissionManager
  3. PermissionManager requires UserGroupManager, which gets outdated version of $wgAddGroups (and maybe other variables)

This means we have an easy workaround: revert https://gerrit.wikimedia.org/r/c/mediawiki/core/+/654718, and use global $wgAddGroups instead, as before the change. We can think about how to fix this properly after wmf.28 is deployed again.

I don't think the super-delayed configuration of FlaggedRevs in flaggedrevs.php (part of our own configuration) @Pchelolo mentions plays a role. For test2wiki, which is where i successfullly reproduced the issue, AddGroups is not touched at all in flaggedrevs.php, leaving us at the defaults set by FlaggedRevs' extension.json. That probably means our init sequence inicializes SessionManager before it loads extension.json files.

Change 660023 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[mediawiki/core@master] Revert "Move User::changeable(By)Groups methods to UserGroupManager"

https://gerrit.wikimedia.org/r/660023

Fun. Thank you @Urbanecm for the stack trace. Agreed, let's revert the whole thing and unblock the train, properly fixing it will be tricky.

Heh.. I think I know how to workaround the problem in a nicer way. It doesn't change the decision to revert everything for now, will do it separately. Now, as we're introducing Authority interface it is going to be very easy to break dependency from AuthManager to PermissionManager, thus breaking up the too early initialization sequence.

Change 660023 abandoned by Urbanecm:
[mediawiki/core@master] Revert "Move User::changeable(By)Groups methods to UserGroupManager"

Reason:
a dupe of 660006

https://gerrit.wikimedia.org/r/660023

Change 660023 restored by Ppchelko:
[mediawiki/core@master] Revert "Move User::changeable(By)Groups methods to UserGroupManager"

https://gerrit.wikimedia.org/r/660023

Change 660006 merged by jenkins-bot:
[mediawiki/core@master] Revert "Remove usages and hard deprecate User::changeable(By)Group"

https://gerrit.wikimedia.org/r/660006

Change 660024 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[mediawiki/core@wmf/1.36.0-wmf.28] Revert "Move User::changeable(By)Groups methods to UserGroupManager"

https://gerrit.wikimedia.org/r/660024

Change 660024 abandoned by Ppchelko:
[mediawiki/core@wmf/1.36.0-wmf.28] Revert "Move User::changeable(By)Groups methods to UserGroupManager"

Reason:

https://gerrit.wikimedia.org/r/660024

Change 660023 merged by jenkins-bot:
[mediawiki/core@master] Revert "Move User::changeable(By)Groups methods to UserGroupManager"

https://gerrit.wikimedia.org/r/660023

Change 660532 had a related patch set uploaded (by Urbanecm; owner: Urbanecm):
[mediawiki/core@wmf/1.36.0-wmf.28] Revert "Revert "Remove usages and hard deprecate User::changeable(By)Group""

https://gerrit.wikimedia.org/r/660532

Change 660532 abandoned by Urbanecm:
[mediawiki/core@wmf/1.36.0-wmf.28] Revert "Revert "Remove usages and hard deprecate User::changeable(By)Group""

Reason:

https://gerrit.wikimedia.org/r/660532

Change 660532 restored by Urbanecm:
[mediawiki/core@wmf/1.36.0-wmf.28] Revert "Revert "Remove usages and hard deprecate User::changeable(By)Group""

https://gerrit.wikimedia.org/r/660532

Change 660532 abandoned by Urbanecm:
[mediawiki/core@wmf/1.36.0-wmf.28] Revert "Revert "Remove usages and hard deprecate User::changeable(By)Group""

Reason:

https://gerrit.wikimedia.org/r/660532

Change 660533 had a related patch set uploaded (by Urbanecm; owner: Urbanecm):
[mediawiki/core@wmf/1.36.0-wmf.28] Revert "Revert "Revert "Remove usages and hard deprecate User::changeable(By)Group"""

https://gerrit.wikimedia.org/r/660533

Change 660024 restored by Urbanecm:
[mediawiki/core@wmf/1.36.0-wmf.28] Revert "Move User::changeable(By)Groups methods to UserGroupManager"

https://gerrit.wikimedia.org/r/660024

Change 660649 had a related patch set uploaded (by Urbanecm; owner: Urbanecm):
[mediawiki/core@wmf/1.36.0-wmf.28] Revert "Move User::changeable(By)Groups methods to UserGroupManager"

https://gerrit.wikimedia.org/r/660649

Change 660024 abandoned by Urbanecm:
[mediawiki/core@wmf/1.36.0-wmf.28] Revert "Move User::changeable(By)Groups methods to UserGroupManager"

Reason:
can't figure out how to un-WIP it, uploading new one

https://gerrit.wikimedia.org/r/660024

If we want to re-deploy wmf.28 on Monday, we need to merge&deploy the following two patches:

Since both are already in master, this is no longer a blocker for wmf.29 (yet unbranched).

Change 660533 merged by jenkins-bot:
[mediawiki/core@wmf/1.36.0-wmf.28] Revert "Revert "Revert "Remove usages and hard deprecate User::changeable(By)Group"""

https://gerrit.wikimedia.org/r/660533

Change 660649 merged by jenkins-bot:
[mediawiki/core@wmf/1.36.0-wmf.28] Revert "Move User::changeable(By)Groups methods to UserGroupManager"

https://gerrit.wikimedia.org/r/660649

Mentioned in SAL (#wikimedia-operations) [2021-02-01T10:39:40Z] <urbanecm@deploy1001> Synchronized php-1.36.0-wmf.28/includes/user//User.php: Fixing T273317 T273296 (duration: 00m 58s)

Mentioned in SAL (#wikimedia-operations) [2021-02-01T10:41:04Z] <urbanecm@deploy1001> sync-file aborted: Fixing T273317 T273296 (duration: 00m 12s)

Mentioned in SAL (#wikimedia-operations) [2021-02-01T10:42:14Z] <urbanecm@deploy1001> Synchronized php-1.36.0-wmf.28/includes/: Fixing T273317 T273296 (duration: 01m 01s)

So, backports applied, group0 (that does not include test2wiki for some reason) promoted to wmf.28.

Urbanecm lowered the priority of this task from Unbreak Now! to High.Feb 1 2021, 1:32 PM

@Pchelolo Hi, so, the reverts were backported into .28, and so far this issue didn't re-occur. I removed this as a train blocker, and I'll leave it for you to deal with, probably by breaking the too-early inicialization sequence, at which point the reverts can be reverted.

Hm, I think the real reason for this bug and it's companion is T270828 - now we have a codepath that ends up initializing PermissionManager/UserGroupManager from SessionManager::getGlobalSessionInfo. It's called during the setup way before extension functions are called, since extension functions have to have fully initialized MW by their contract. So I would assume that PermissionManager probably gets wrong $wgGroupPermissions as well.

Pchelolo claimed this task.

Ive created followup tasks T273510 and T273511 that would allow us to fix the issue properly and unrevert it.