Page MenuHomePhabricator

Document that groups with IP reveal rights must not be changed without making changes to the cache for Special:GlobalContributions
Closed, ResolvedPublic1.5 Estimated Story Points

Description

Summary

Special:GlobalContributions permissions checks assume that the groups with IP reveal rights never change. This is currently the case according to the IP reveal access policy.

Explanation of the issue

If the IP reveal rights are assigned to different groups, then the Special:GlobalContributions permissions check may become inaccurate for users in those groups, for up to 1 month. This means:

  • If IP reveal rights were added to a group, then users in that group may not be able to reveal IPs via Special:GlobalContributions if they have an existing cache entry (meaning if they have checked Special:GlobalContributions within the last month). Eventually this will fix itself.
  • If IP reveal rights are removed from a group, then users in that group may still be able to reveal IPs via Special:GlobalContributions if they have an existing cache entry (meaning if they have checked Special:GlobalContributions within the last month). Eventually this will fix itself.

This is also the case for other permissions, e.g. seeing deleted edits.

This only affects changing which rights are assigned to groups. Special:GlobalContributions will show the correct data to a user straight away if they are added to or removed from a group with IP reveal rights.

Recommendations if changing group rights

As summarized in T396217#11065904, we advise that it is probably OK for some cache entries to be out of date for a while given the types of changes we expect to happen. We don't think it is likely that sensitive rights will be removed from groups, thus enabling users to see data that they shouldn't. We expect that assigning these rights to additional groups will be very infrequent.

However, this task should be read through and a judgement should be made about this, if ever the group rights assignments do change.

Technical notes

For more details, including possible solutions to invalidate the cache when the group rights assignment changes, see T396217#11065904.

Special:GlobalContributions caches its external permissions lookups because they're expensive but necessary. This cached value relies on being invalidated if the user ever gains or losses permissions in order to ensure that if the permission can affect their lookup permissions, Special:GC will accurately reflect that. The only way to programmatically declare this invalidation is on user group change (local and global). However, rights can be added/removed to groups via config and if a relevant right is added/removed this way, the cache invalidation has no way of seeing this.

The only protection against this drift is policy and institutional knowledge not to do the thing. If policy changes or a config changes, we need to decide and document:

  1. what to do preemptively (eg. in case of policy change and we have time to prepare)
  2. what to do reactively (eg. in case of a config change we need to respond to)
Acceptance Criteria

From T396217#11065904:

Event Timeline

Change #1154308 had a related patch set uploaded (by Tchanders; author: Tchanders):

[operations/mediawiki-config@master] Document that IP reveal permissions can't just be reassigned

https://gerrit.wikimedia.org/r/1154308

For local groups, the patch above points to this task from where the IP reveal rights are assigned in the config repo.

For global groups we could invalidate the affected cache entries automatically when permissions assignments are changed by running a new hook from SpecialGlobalGroupPermissions::invalidateRightsCache.

Tchanders renamed this task from Document response steps to potential permissions drift for Special:GlobalContributions access to Document that groups with IP reveal rights must not be changed without making changes to the cache for Special:GlobalContributions.Jun 6 2025, 3:14 PM
Tchanders updated the task description. (Show Details)

For now, we could define a cache version in code and manually update it if mediawiki-config changes.

If we want to avoid that, we could go the RL way and integrate a simple maintenance script with the deployment process that would touch a check key either unconditionally, or if the permissions configuration changes (if that can be determined).

Tchanders set the point value for this task to 1.5.Aug 1 2025, 9:01 AM

Here's my take on what to do next, having discussed with @STran, @Dreamy_Jazz , @mszabo and @OKryva-WMF (thanks!).

Summary
  • The permissions cache for GlobalContributions may become inaccurate, if the permissions are assigned to different groups.
  • We will make a change to how we cache, to make sure this only happens occasionally and for a limited duration.
  • We will document clearly on this task what to do if assigning the IP reveal permission to different groups: our advice is to do nothing, but also to read this task and reassess, in case something has changed between now and then.
The problem

The permissions cache is not invalidated when the rights that are looked up are assigned to different groups, only when the user's groups change.

There is no way to automatically invalidate the cache when local groups' rights change. (There's nothing to hook into, since these are only defined in config, which is updated by changing the config files.)

Therefore the cache entries may become inaccurate.

Exacerbating factors

The way we build the cache entry means the data can be older than the TTL (time to live - 1 month) would suggest.

What happens:

  • The first time a user visits Special:GlobalContributions, we look up their permissions on the relevant wikis and cache the results under a key made from the user's name, with a TTL of 1 month
  • If the user visits Special:GlobalContributions again within the next month, if we need to look up permissions at new wikis, we add these results to the user's existing cache object and set a new TTL of 1 month from now

Effect:

  • The older results can then last more than 1 month. (Arbitrarily long, e.g. if new wikis keep getting added to the cache entry every few weeks.) That exacerbates the problem of not invalidating when the group rights change, because an inaccurate cache entry can last arbitrarily long.
Mitigating factors

(The rights that are checked are checkuser-temporary-account, checkuser-temporary-account-no-preference, deletedtext, deletedhistory, suppressrevision and viewsuppressed.)

  • deletedtext, deletedhistory, suppressrevision and viewsuppressed are changed only very infrequently (once or twice a year [1]), so the cache entries would not often be wrong.
  • If the cache says a user can't see something when they should be able to, they can still see it locally via Special:Contributions.
  • The cache would only say a user can see something when they shouldn't be able to, if one of these rights is removed from a group. That hasn't happened in the last 5+ years [1]. This would only affect users in these trusted groups.
  • checkuser-temporary-account and checkuser-temporary-account-no-preference cannot be assigned to different groups without a policy change. We can improve documentation that cache entries could become inaccurate if these are changed.
What should we do?

Actions to take

  • Update the way we cache so that the data can't get arbitrarily old:
    • Segment on the user/wiki combination rather than just the user
    • Make a new cache entry for each user/wiki, rather than updating an existing entry
  • Update the config documentation patch from T396217#10891325 to advise reading through this task before changing the rights assignment for the IP reveal rights.
  • Update the description of this task to make it clear that, if the IP reveal rights assignments change, we advise doing nothing by default, but we advise reading through this task and deciding whether doing nothing is still the best course of action.

With the above actions and the mitigating factors, the cache inaccuracies could only affect a small number of trusted users for a limited time. If we're comfortable with this, we don't need to take further action.

Other solutions we discussed but didn't go with

Noting these here for reference.

Technical approaches:

  • Invalidate all cache entries whenever wmf-config changes. <- Not feasible since it changes multiple times a day
  • Store local group permissions in a database that is updated by a maintenance script each time permissions config change. Have this script invalidate the cache entries when needed. <- A lot of work for not much gain: storage space, two-sources-of-truth risk, running the script that almost always does nothing. (However, if other projects need this, it might be worth considering, since we do store global group rights in the DB. But that's beyond the scope of this task.)
  • Reduce the TTL, so the cache is wrong for less long. <- Too many cache misses is frustrating to users, since this can block the page from loading properly (we had bug reports about this before we implemented the cache). 1 month seems to be working (good stats, no bug reports).
  • Only look up the non-public-visibility rights for the wikis that return non-publicly visible results <- This is worth exploring if ever we decide the cache needs to be perfectly accurate. We'd need to consider how/whether to fix pagination if we need to remove hidden revisions after querying the wikis for all revisions.

Product approaches:

  • Only include contributions that are visible to everyone, so that the rights don't need to be looked up. Show which wikis the usr is active on, so this can be followed up locally on Special:Contributions. The non-public revisions can be seen there. <- Having to swap tools degrades the user experience, so it would be nice to avoid this if we can.
  • Don't show the non-public revisions by default, but add a checkbox for adding them. The IP reveal right is cached as before, and the other rights are looked up only when needed, and only for the wikis that have results on this page. <- This is probably the strongest product-based solution, but does cause some inconvenience to users and may be unnecessary if we don't mind the cache being occasionally wrong for a limited time and a small number of users.

[1] According to git log -G "deletedtext|deletedhistory|suppressrevision|viewsuppressed" --oneline --pretty=format:'%C(auto)%h%d (%cr) %s' on the mediawiki-config repo (thanks @STran for sharing this)

Change #1176511 had a related patch set uploaded (by Tchanders; author: Tchanders):

[mediawiki/extensions/CheckUser@master] CheckUserGlobalContributionsLookup: Improve caching strategy

https://gerrit.wikimedia.org/r/1176511

Change #1176511 merged by jenkins-bot:

[mediawiki/extensions/CheckUser@master] CheckUserGlobalContributionsLookup: Improve caching strategy

https://gerrit.wikimedia.org/r/1176511

Change #1179697 had a related patch set uploaded (by Tchanders; author: Tchanders):

[mediawiki/extensions/CheckUser@master] Test check key building in CheckUserGlobalContributionsLookup

https://gerrit.wikimedia.org/r/1179697

Change #1179697 merged by jenkins-bot:

[mediawiki/extensions/CheckUser@master] Test check key building in CheckUserGlobalContributionsLookup

https://gerrit.wikimedia.org/r/1179697

Change #1154308 merged by jenkins-bot:

[operations/mediawiki-config@master] Document that IP reveal permissions can't just be reassigned

https://gerrit.wikimedia.org/r/1154308

Mentioned in SAL (#wikimedia-operations) [2025-09-02T13:04:58Z] <stran@deploy1003> Started scap sync-world: Backport for [[gerrit:1154308|Document that IP reveal permissions can't just be reassigned (T396217)]], [[gerrit:1180532|Enable temporary accounts on remaining small-sized projects (T402181)]]

Mentioned in SAL (#wikimedia-operations) [2025-09-02T13:11:47Z] <stran@deploy1003> tchanders, stran: Backport for [[gerrit:1154308|Document that IP reveal permissions can't just be reassigned (T396217)]], [[gerrit:1180532|Enable temporary accounts on remaining small-sized projects (T402181)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

jijiki raised the priority of this task from High to Unbreak Now!.EditedSep 2 2025, 3:25 PM
jijiki subscribed.

<comment removed as it belongs to T402181>

The patch for this ticket that was merged then was just a change in PHP comments. I'm not sure how it could have caused that?

The patch for this ticket that was merged then was just a change in PHP comments. I'm not sure how it could have caused that?

It got deployed at the same time as https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1180532 (enabling temp accounts on small wikis), so I think there was a mixup on which task was associated with the change.

It can be because of the temp accounts being enabled? (deployed at the same time: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1180532) I need to go to a meeting very soon but this needs investigation.

Sorry for the mixup, I will reopen T402181 and remove my comment!

jijiki changed the task status from Open to In Progress.Sep 2 2025, 3:42 PM
jijiki lowered the priority of this task from Unbreak Now! to High.