Stewards didn’t get GTAIV because it’s redundant to our steward permissions. But it turned out most of us still got the permissions once our local groups changed in a wiki where we hold CU/OS permissions (even when the group change was unrelated to CU/OS like in the example above) or when temporarily granting CU/OS permissions to ourselves to perform actions on a wiki without local CU/OS.
As of today there are just four stewards left without GTAIV https://meta.wikimedia.org/wiki/Special:GlobalUsers?username=&group=steward&limit=100

I suppose this is because the new global group was not initially populated with the users belonging to relevant local groups, so the change took effect the next time the local group membership was changed in any way?

Yes, that would explain it.

I think we should fix it by populating the groups with a maintenance script. I am already working on one for the 'local-bot' group from T415588: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/1234540. Should we run the script for the 'global-temporary-account-viewer' group as well?

Thanks. Yes please, it makes sense to run it for 'global-temporary-account-viewer' too.

Change #1234540 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):

[mediawiki/extensions/CentralAuth@master] Add maintenance script to update automatic global group membership

https://gerrit.wikimedia.org/r/1234540

Change #1234540 merged by jenkins-bot:

[mediawiki/extensions/CentralAuth@master] Add maintenance script to update automatic global group membership

https://gerrit.wikimedia.org/r/1234540

Next step: wait for train rollout, then run foreachwikiindblist sul CentralAuth:UpdateAutomaticGlobalGroupMembership --local-group=checkuser --local-group=suppress. This will take care of any outstanding memberships in global-temporary-account-viewer that weren't processed before, per this config for local groups: https://gerrit.wikimedia.org/g/operations/mediawiki-config/+/a13355b072b61df5be4e319f3fdaec6479fb3276/wmf-config/CommonSettings.php#4568 (I checked and all users in the global group global-sysop are already members of the new global group, so there's no need to handle that). Changes will be logged at https://meta.wikimedia.org/wiki/Special:Log/gblrights/Maintenance_script.

Done on the beta cluster: https://meta.wikimedia.beta.wmcloud.org/wiki/Special:Log/gblrights/Maintenance_script

There are some discrepancies in the numbers reported by the script, and the number of log entries. The script says that a total of 62 users were added to the group. The log only shows 9 entries.

The current list of users has 70 of them. I didn't think to check it before running the script, and I haven't found any way to find the previous data. My best guess, based on the gblrights log and the order of entries in the global_user_groups table, is that there were 14 users in the group and 56 were added.

My best guess is that the problems were likely caused by the script reading from the replica databases. In some cases this caused it to try to add a user again because it appeared that they were not group members yet, and in other cases to skip generating the log entry because it appeared that no changes were made.

Change #1242458 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):

[mediawiki/extensions/CentralAuth@master] UpdateAutomaticGlobalGroupMembership: Read user data from primary

https://gerrit.wikimedia.org/r/1242458

You could always just configure a global-temporary-account-viewer-2 group on beta, re-run the script, and see what happens. That's guaranteed to be empty at start.

I was going to run a query to delete user group memberships, excluding those that seem to have been there before today:

DELETE FROM global_user_groups 
WHERE gug_group='global-temporary-account-viewer' 
AND gug_user NOT IN (202091, 55618, 33432, 296022, 13970, 196695, 17, 296122, 296023, 296016, 296009, 252629, 24596, 34246);

This should be fine too, right?

Change #1242458 merged by jenkins-bot:

[mediawiki/extensions/CentralAuth@master] UpdateAutomaticGlobalGroupMembership: Read user data from primary

https://gerrit.wikimedia.org/r/1242458

I ran that DELETE.

Second attempt to run the script on the beta cluster: https://meta.wikimedia.beta.wmcloud.org/wiki/Special:Log/gblrights/Maintenance_script?limit=100

This looks much better. Scripts says 56 users added, and there are 56 new log entries. I think we can go ahead with production now.

Change #1242469 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):

[mediawiki/extensions/CentralAuth@wmf/1.46.0-wmf.16] UpdateAutomaticGlobalGroupMembership: Read user data from primary

https://gerrit.wikimedia.org/r/1242469

Change #1242469 merged by jenkins-bot:

[mediawiki/extensions/CentralAuth@wmf/1.46.0-wmf.16] UpdateAutomaticGlobalGroupMembership: Read user data from primary

https://gerrit.wikimedia.org/r/1242469