Depending on whether you also need to unlink users who have recently unset the preference, either setting wikimedia-donor to a timestamp rather than a boolean, or having a separate wikimedia-donor-last-touched field seems like the easiest way of keeping track of the time.

Communication:

• mediawiki-api-announce
• wikitech-l
• Tech News
• API changelog

In T440873#12425149, @STei-WMF wrote:

Hello @Tgr @Vesperio_ofc, responding to the usernotice tag. How will this be worded? Also are we featuring this in this week's tech news?

This is the only thing in the job output that seems like an error:

arwiki Unknown component: ext.cite.visualEditor, ext.cite.wikiEditor, ext.cite.community-configuration-autoname, ext.cite.referencePreviews, ext.confirmEdit.CaptchaWidget, ext.confirmEdit.hCaptcha, ext.confirmEdit.hCaptcha.styles, ext.confirmEdit.hCaptcha.gradeC, ext.globalCssJs.user, ext.globalCssJs.user.styles, ext.wikimediaMessages.ipInfo.hooks, ext.abuseFilter.ace, ext.visualEditor.editCheck.checks, ext.wikimediaEvents.visualEditor, mw.cx.externalmessages, ext.checkUser.tempAccountOnboarding, ext.checkUser.ipInfo.hooks, ext.checkUser.suggestedInvestigations, ext.quicksurveys.survey.ReadingLists.beta.feature.survey, ext.quicksurveys.survey.Automatic.Translation.Feedback, mediawiki.messagePoster

The new MediaWiki version is on group 0, but the authevents Logstash channel is sampled 1:1000 and has no group0 temp account creation events from the last day at all, and the Prometheus metric is not split by wiki (it made the dashboard slow to the point of uselessness). So I'm not sure there's a good way to assess impact other than waiting for the group 1 deployment, unless we can identify some non-sampled log event that would accompany the error.

Also a request ID would be useful.

If you can still reproduce, what's the chain of top-level HTTP requests / redirects?

Works for me on Chrome 154.0.8037.92 / Ubuntu in incognito mode. The browser is not redirected to the central domain though, and ends up with no central cookies. As far as I can see, that's a JS issue (the VE could should handle the redirect).

Probably a train blocker?

Those seem like two very different errors.

Still (or again?) happening. Should be fixed (well, ignored) by T432613: backfilLocalAccounts.php should not send alerts when the account is blocked though.

Mesh timeout was updated in T390972: Restart CronJobs on failure of the service mesh, this can probably be closed now.

Mesh timeout was updated in T390972: Restart CronJobs on failure of the service mesh, this can probably be closed now.

Mesh timeout was updated in T390972: Restart CronJobs on failure of the service mesh, this can probably be closed now.

Mesh timeout was updated in T390972: Restart CronJobs on failure of the service mesh, this can probably be closed now.

This is a documented limitation of the minifier; similar changes have been rejected in the past to keep the code simpler (T37492: CSSMin::minify should not interfere with string values). Is there a real-world use case this bug is interfering with? The example in the task description is not something I'd expect to see in real code - it's brittle and there are almost certainly better ways to target a given hidden div.

To clarify, when you say "old notifications become re-highlighted", you don't mean individual notifications (as in the number on the icon changing, or the blue dot reappearing in the upper right corner of a notification entry in the dropdown) but rather one of the menu icons becoming re-highlighted (going from gray to red/blue), right?

Naming isn't the main problem. In any case let's continue this thread in a dedicated task -> T440867: Make new OAuth 2 client-credentials apps authenticated by default (set $wgOAuth2ClientCredentialsOwnerAuthCutover default-on)

WikiMap has changed recently, we should check whether this will be still an issue after T440863: Remove most uses of MockWikiMapTrait-

(Tagging as MediaWiki-Core-Platform-Team as we created the mock mainly for CentralAuth.)

We since a few days ago now not setting callback URLs on owner-only and client-credentials apps since they have no purpose, but yeah it should be displayed in a more meaningful way (probably by just hiding the whole field). If you are seeing it on an app with auth code grant or an OAuth 1 app, that's a bug in app validation.

09:09:14 1) MediaWiki\Extension\OATHAuth\Tests\Integration\Special\OATHManageTest::testDisplaysIrremovableWarning
09:09:14 Failed asserting that "
<snip>
(oathauth-2fa-groups-notice-multiple-links-entry: 2, (group-interface-admin-member: TestUser 94aa7c.09d)(and)(word-separator)(group-sysop-member: TestUser 94aa7c.09d), 127.0.0.1, <a rel="nofollow" class="external text" href="http://127.0.0.1:9413/wiki/SomePage">SomePage</a>)
<snip>
(oathauth-2fa-groups-notice-multiple-links-entry: 1, (group-suppress-member: TestUser 94aa7c.09d), 127.0.0.1, <a rel="nofollow" class="external text" href="http://127.0.0.1:9413/wiki/DefaultPage">DefaultPage</a>)
<snip>
(oathauth-2fa-groups-notice-multiple-links-entry: 1, (group-interface-admin-member: TestUser 94aa7c.09d), remote-wiki.local, <a rel="nofollow" class="external text" href="http://remote-wiki.local/wiki/SomePage">SomePage</a>)
<snip>
"
contains "(oathauth-2fa-groups-notice-multiple-links-entry: 2, (group-interface-admin-member: TestUser 94aa7c.09d)(and)(word-separator)(group-sysop-member: TestUser 94aa7c.09d), wikidb-unittest_.local".

so the $entry['wiki'] parameter differs. Not sure if that means the test finds the user on a different wiki than expected or something about wiki handling is broken (we know eg. WikiMap doesn't work well in unit tests).

(Wrt the current high number of memcached errors, there is an investigation ongoing in T420223: Increase in number of memcached errors in eqiad. For replag, I have no idea what would count as unusually high.)

In T438570#12401037, @herron wrote:

Re: false alarms: having a look at throttle_warning tagged logs over the past 7 days, would a spike in any of these logs be actionable if raised during a deploy?

5,346,462 Cookies set on {url} with Cache-Control "{cache-control}"

We'd need a fast system user check (e.g. T212720: System users should be in a dedicated user group).

What is the scenario in which an adversary can intercept the submitted code, but cannot intercept the challenge? Otherwise the combinatorics is entirely pointless, and each fragment can only be used once.

That seems like a more confusing variant of issuing recovery codes (which we do).

CentralAuthUnattachUserJob jobs stopped being scheduled when the train reached group 2, so whatever was happening (still not sure what resulted in multiple jobs per minute) is fixed.

Another approach would be to ignore all tokens where the iat (issued at) field is older than some given threshold. That's robust against accidental minting of tokens with no / far future expiry.

I wonder if we should have some kind of suppression system (along the lines of the current production hack - an array with normalized message => sampling rate that overrides normal sampling settings).

The number of events with throttle warning is always going to be 1000 (or more generally 1000 x number of unique message strings that are being throttled), since that's the throttle limit, right? The real (unthrottled) number of events would be more informative as that tells you how much burden it is putting on the event intake, but I don't think that number is available within Logstash.

In T388247#10665436, @Tgr wrote:

git-review automatically installs a commit-msg hook from https://gerrit.wikimedia.org/r/tools/hooks/commit-msg when setting up a new repository (to handle Change-Id); maybe that can be altered somehow.

Pushing Gerrit patches for low-risk security issues (which are nevertheless private) is not uncommon, and I think in that case we'd still want to link to the Phab task, so warn reject rather than reject would better. (Or allow the patch through when an I-Know-What-I-Am-Doing: Yes footer is present, or whatever.)

How does it put pressure on the logging infra if it's a relatively small increase in total log traffic? 200K / 30 min is about 100/sec, IIRC the total MediaWiki log volume is a couple thousand per sec.

Should be in backlog, not radar, since it's about a component we own.

Oh, right, the service factory could be reinitialized at any time. Guess we'll just have to use Utils::getCentralWiki() everywhere then.

OTOH if unapproved client-credentials apps would be usable by their owner, that would make approval pointless since client-credentials apps can only be used by their owner.

Some ideas:

• group information visually - "title information" (owner, name and version; should be made more prominent), description, protocol details (OAuth version, flow/grants, confidential flag, callback URI), abilities (scopes, applicable wiki, restrictions if user is allowed to view it), status (stage, maybe a log snippet on the detail page), IDs (client key, DB ID), navigation / action links
• make it clear at a glance whether the app is for everyone of for the owner
• make it clear at a glance whether the app has dangerous scopes
• indicate whether the user viewing the page has authorized this application

onMediaWikiServices is running well before onSetupAfterCache. Maybe something else is setting $wgMWOAuthCentralWiki in your wiki setup?

Not sure how this ended up on MediaWiki-Core-Platform-Team (Radar), it's clearly within scope for the team.

I tried to get all unattached accounts but the query didn't turn up any results. I'm probably messing it up somehow.

hive (wmf_raw)> select count(*) from centralauth_localuser right join mediawiki_user where centralauth_localuser.snapshot='2026-08' and centralauth_localuser.wiki_db = 'centralauth' and mediawiki_user.snapshot=2026-08 and lu_name = user_name and lu_wiki = mediawiki_user.wiki_db and lu_name is null;
_c0
0

Similar to T437261: "Wikimedia\Services\ContainerDisabledException: Container disabled!" in JobQueue / WANObjectCache->getWithSetCallback / getDBLoadBalancer().

Probably before the cleanup we should fix system users:
T275931: Have new system users be automatically attached to CentralAuth
T111686: Attach existing system users to CentralAuth

Spot-checking some random job entries:

tgr@mwlog1003:/srv/mw-log$ ack CentralAuthUnattachUserJob JobExecutor.log | head -n100 | ack 'username=([^=]+) wiki=([^ ]+)' --output='$2 $1' | sort | uniq -c
     12 cswiki Hovhannes Kirakosyan
      2 dewiki ~2026-52145-13
      2 enwiki ~2026-51810-00
      2 enwiki ~2026-52041-04
     14 fiwiki Hovhannes Kirakosyan
      2 frwiki Emmabsha.256
      6 hewiki Hovhannes Kirakosyan
      2 idwiki Hovhannes Kirakosyan
      2 itwiki ~2026-52026-69
     13 kowiki Hovhannes Kirakosyan
      2 metawiki ~2026-51810-00
      6 nowiki Hovhannes Kirakosyan
      6 ptwiki Hovhannes Kirakosyan
     15 rowiki Hovhannes Kirakosyan
     14 ukwiki Hovhannes Kirakosyan

The only code path leading to CentralAuthUnattachUserJob is LocalUserNotFoundException thrown from localUserData(), which will log either Requested to load user data for unknown wiki {wikiId}. Was it deleted without data migration? (zero instances in the last 90 days) or Could not find local user data for {username}@{wikiId} (4 instances, all during global vanishing, but it does not include SHB). But it won't log if we are reading the replica, and the callers will schedule the job anyway. Oops.

There was a single spike of ~1500 events starting around 17:30 Wednesday and ending around 18:10. That matches the codfw depool time for T439010: codfw power outage, so probably it's just a bunch of memcached hosts being impacted by the power outage.

Returned to old levels a while ago:

All the metrics seem to be in the normal range now, except for T438380: Account_on_central_wiki_is_not_attached_this_shouldn_t_happen errors during autologin which is still generating a huge number of autologin errors, but presumably that doesn't affect established users who haven't been renamed recently.

This should be fixed in RemoteBoxedExecutor. The logger could be any PSR-3 implementation, not just Monolog or LegacyLogger; we explicitly allow site administrators to customize that. PSR-3 says

@LucasWerkmeister three months have passed, should we just assume rEOAUe4424057b1be: Add a grace period for reusing refresh tokens has fixed the issue?

Discussing all session loss problems this year in the same task isn't really useful, so here's a dedicated one: T439139: Investigate session losses during 2026 September eqiad sessionstore depool

I mean it doesn't show anything that would explain why the release tools or patchdemo repos would trigger that test.

In T425216#12351266, @BBlack wrote:

To not copy any more application-specific business logic into CDN edge logic than we absolutely have to (because it binds them together and makes future changes more complicated).

Maybe caused by T438591: Commons account not attached with CentralAuth in account rename (SHB) although that was just one user, and the volume is quite large (50-100 errors per minute).

Same happening at Patchdemo (https://gitlab.wikimedia.org/tgr/patchdemo/-/pipelines/229200/failures)

Why would that involve auth.wikimedia.org though? We don't do temp user autologin until after the save was successful.

In hindsight, maybe better to go the opposite direction and replace the DB ID with the key everywhere, so it becomes a completely internal detail.

The (not yet implemented on any browser) DBSC JS API explainer says