akhatun opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2212

Temporary increase of userWrongRowsRatioThreshold

akhatun closed https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2212

Temporary increase of userWrongRowsRatioThreshold

Slack thread

Error log:

26/05/02 21:44:21 WARN DenormalizedHistoryChecker: DenormMetricsGrowthErrors ratio 0.08317929759704251 is higher than expected threshold 0.05 -- Writing errors

From @JAllemandou

I have investigated and found the problem. I don't have a good solution though, and this issue opens the door for a larger variety of problems I had not foreseen.
Data investigation:

• Claude is right, the problem is with user entities, altergroups event-type, for the specific anonymous_user subgroup
• The data growth is real, we have new group-change events in the new snapshot, for many wikis and across the entire past 25 years time period
• The new events performers are flagged as anonymous but their usertext values are in the form metawiki>USERTEXT, not usual IPs
• The underlying logging table also contains the new events, with their logging_actor defined, and the actors not referencing any user, actor_user values being NULL.

Searching Phabricator for related tasks, I found T6055: Interwiki rights logs should be duplicated at related wikis
 * TIL that group-change (and also other user actions probably) can be initiated from metawiki for users in other wikis. Now that I write it, it makes sense with centralauth, but I had not thought about it.
 * Originally the action were logged in metawiki only, not in the other wikis affected by the change. The task above is making this change, duplicating the logs to reflect the change on the other wikis too.
 * The task has a backfill step for historical data, that happened on April 14th 2026, hence explaining why we see a change in this snapshot

Yay! it all makes sense :cat_yay:

The problem we need to solve for MWH is that the group-change performers are not anonymous while they are reported as so.
I suggest we use the actor name pattern metawiki>USERTEXT to find those cases, and add a event_user_is_cross_wiki field to make this explicit.
I think we need a task for the solution I suggest if we wish to make it happen, and that for this month we should go with accepting the data as it is.
Let's talk about this more.

Additional info

The problem occurs only for a very small portions of events, where:
event_entity = 'user' AND event_type = 'altergroups'
Within those events, we had almost no events where event_user_is_anonymous = true, because anonymous users don't have the right to change groups for users.
In the new snapshot, we now have some. This is due to a data-backfill of altergroups  changes in the logging table where the change was initiated from centralauth (metawiki)

Example: An admin wants to change groups for a specific user, and uses centralauth to do so. Until recently, only metawiki received logging row for the action. Now, sister projects where the change applies also receive logging rows.
This means a cross-wiki event in terms of performer (the user having performed the action is logged into metawiki, the action happens on a sister project). And this means for us that we don't have a user to reference as the performer. Therefore we mark it as anonymous.
A better way would be for us to have a dedicated flag where we would mark the user as "crosswiki_user"
But for now we don't have that.

The metawiki>USERTEXT event you can find for revision and page were already present in the previous MWH snapshots (checked the six snapshots, the number almost doesn't change)

Interesting!

BTW, I am dealing with a similar thing in T423952: Create mediawiki.user_change event stream. GlobalUserRenames result in local wiki group_change events with an anonymous performer. I'm currently setting performer to null, but perhaps I should try to find some way to represent a system/job/async performer?

Hm. will consider.

What we really need is a GlobalUserRename DomainEvent or hook that gives us the global change event, not just all the triggered local ones.