Accounts taking 30+ minutes to autocreate on metawiki/loginwiki (2023-05)
Open, Needs TriagePublicBUG REPORT
Actions

Assigned To

None

Authored By

	AntiCompositeNumber
	May 14 2023, 12:52 AM

Description

It is taking over 30 minutes for most new account creations to autocreate accounts on metawiki and loginwiki. Some will create the account on loginwiki or metawiki as expected, but the creation of the other will be delayed. Most seem to not have either. Some create fine.

Examples:

Previous occurrance: T314442

Details

	Subject	Repo	Branch	Lines +/-
	team-sre: Add warning for CentralAuth job lag	operations/alerts	master	+46 -0

Customize query in gerrit

Related Objects

Mentioned In: T348206: Improve logging, monitoring and test coverage for MediaWiki Platform team authentication extensions
Mentioned Here: T314442: New accounts not attaching to meta/loginwiki

Event Timeline

AntiCompositeNumber created this task.May 14 2023, 12:52 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 14 2023, 12:52 AM

Xaosflux subscribed.May 14 2023, 1:01 AM

taavi added a project: WMF-JobQueue.May 14 2023, 10:05 AM

As usual, this resolved itself without intervention after a few days. However, every time it happens it prevents us from appropriately responding to abuse. Given that there is no reason to think it will not reoccur, I am leaving this task open.

Not sure if there's anything we can do about the job queue being slow, but let's at leask ask that from the correct (I think) people.

Change 935078 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/alerts@master] team-sre: Add warning for CentralAuth job lag

https://gerrit.wikimedia.org/r/935078

gerritbot added a project: Patch-For-Review.Jul 3 2023, 2:16 PM

These issues seem to have occurred before we added capacity to the jobrunner cluster. I'm in the process of adding an alert to quantify if it still happens often, and if it does we'll try to figure out a fix.

Possible things to think about in the meantime are:

It is low volume, but relatively time-sensitive, does it make sense to give it its own lane in cp-jobqueue?
If it is very time sensitive, maybe it should be something else than a job (we can potentially have to shut off the jobqueue at times). If so, and since it is cross-wiki, what could be the mechanism?

Clement_Goubert moved this task from Incoming 🐫 to 🌻Mediawiki on the serviceops board.Sep 18 2023, 11:31 AM

Restricted Application added a project: MediaWiki-Platform-Team. · View Herald TranscriptSep 18 2023, 11:31 AM

akosiaris subscribed.Sep 18 2023, 11:35 AM

larissagaulia moved this task from Inbox, needs triage to Radar on the MediaWiki-Platform-Team board.Sep 18 2023, 12:24 PM

larissagaulia edited projects, added MediaWiki-Platform-Team (Radar); removed MediaWiki-Platform-Team.

Tgr mentioned this in T348206: Improve logging, monitoring and test coverage for MediaWiki Platform team authentication extensions.Oct 5 2023, 12:52 AM

This is currently happening! I'm seeing some users (also LTAs) created like 30 minutes ago, and I cannot check on loginwiki/metawiki because they have only one local user in the SUL (their homewiki). While I can see other, more recent users, with the SUL created without problems! Just for example this one (don't want to add others, since there are some who are abusive usernames), thanks.
P.S. FTR Some hours ago I also noticed this spambot (without autocreated user on loginwiki after almost 2 days)!

The p99 job backlog for CentralAuthCreateLocalAccountJob is 30s (with a few spikes going up to 4 min). That doesn't seem so bad.

In T336627#8985967, @Clement_Goubert wrote:

It is low volume, but relatively time-sensitive, does it make sense to give it its own lane in cp-jobqueue?

What counts as low volume? The insertion rate is 0.2/s on average.

Change 935078 abandoned by Clément Goubert:

[operations/alerts@master] team-sre: Add warning for CentralAuth job lag

Reason:

Abandoned for gitlab migration