Page MenuHomePhabricator
Create Task
Maniphest T402808

Deploy separate anonymous session backend to Wikimedia production, in log-only mode
Closed, ResolvedPublic
Actions

Assigned To
DAlangi_WMF
Authored By
Tgr
Aug 25 2025, 1:55 PM
Tags
Referenced Files
F66719572: Screenshot Capture - 2025-10-02 - 12-44-04.png
Oct 2 2025, 10:46 AM
F66719557: Screenshot Capture - 2025-10-02 - 12-37-21.png
Oct 2 2025, 10:46 AM
F66719555: Screenshot Capture - 2025-10-02 - 12-35-25.png
Oct 2 2025, 10:46 AM
Subscribers
Aklapper
Tgr

Description

Set the anonymous and authenticated session backend to the same service but use two separate BagOStuff represenatitions for the backend, and check for every write/read that the handle being used matches the content of the read/write.

Depends on the patches in T394075: Investigate using different stores for different kinds of sessions, and on T399195: Update logging and monitoring for multiple session storage backends.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
session: Avoid logging in session store if UserInfo is availablemediawiki/coremaster+32 -23
session: Enable MultiBackendSessionStore on `group2` wikisoperations/mediawiki-configmaster+1 -3
session: Improve logging for MultiBackendSessionStoremediawiki/corewmf/1.45.0-wmf.22+17 -18
session: Improve logging for MultiBackendSessionStoremediawiki/corewmf/1.45.0-wmf.21+17 -18
session: Improve logging for MultiBackendSessionStoremediawiki/coremaster+17 -18
session: Lookup authenticated store first before anon storemediawiki/corewmf/1.45.0-wmf.20+15 -8
session: Lookup authenticated store first before anon storemediawiki/corewmf/1.45.0-wmf.21+15 -8
session: Handle an edge-case in MultiBackendSessionStore::set()mediawiki/corewmf/1.45.0-wmf.21+15 -0
session: Handle an edge-case in MultiBackendSessionStore::set()mediawiki/corewmf/1.45.0-wmf.20+15 -0
tests: Expand tests for MultiBackendSessionStoremediawiki/coremaster+82 -0
session: Lookup authenticated store first before anon storemediawiki/coremaster+15 -8
session: Handle an edge-case in MultiBackendSessionStore::set()mediawiki/coremaster+15 -0
session: Enable MultiBackendSessionStore on `group1` wikisoperations/mediawiki-configmaster+1 -0
session: Enable MultiBackendSessionStore on `group0` wikisoperations/mediawiki-configmaster+1 -1
session: Improve logging and monitoring in SessionStore implementationsmediawiki/corewmf/1.45.0-wmf.20+97 -38
session: Improve logging and monitoring in SessionStore implementationsmediawiki/corewmf/1.45.0-wmf.19+97 -38
Enable multibackend session store on beta and testwikioperations/mediawiki-configmaster+8 -0
session: Improve logging and monitoring in SessionStore implementationsmediawiki/coremaster+97 -38
session: Follow-up on I10101c8b928a12 (3fde556f95b4b1ce43)mediawiki/coremaster+683 -52
session: Segregate anonymous sessions from authenticated sessions (p2)mediawiki/coremaster+361 -32
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Tgr added a parent task: T400372: Separate storage backend for anonymous sessions.Aug 25 2025, 7:30 PM
Tgr added a project: OKR-Work.
Tgr mentioned this in T402853: Deploy separate anonymous session backend to Wikimedia production.Aug 25 2025, 7:39 PM
gerritbot added a comment.Aug 26 2025, 6:56 PM

Change #1178870 had a related patch set uploaded (by D3r1ck01; author: Derick Alangi):

[mediawiki/core@master] session: Segregate anonymous sessions from authenticated sessions (p2)

https://gerrit.wikimedia.org/r/1178870

gerritbot added a project: Patch-For-Review.Aug 26 2025, 6:56 PM
gerritbot added a comment.Aug 29 2025, 2:49 PM

Change #1183132 had a related patch set uploaded (by D3r1ck01; author: Derick Alangi):

[operations/mediawiki-config@master] session: Enable MultiBackendSessionStore on `group0` wikis only

https://gerrit.wikimedia.org/r/1183132

gerritbot added a comment.Sep 11 2025, 6:49 PM

Change #1178870 merged by jenkins-bot:

[mediawiki/core@master] session: Segregate anonymous sessions from authenticated sessions (p2)

https://gerrit.wikimedia.org/r/1178870

ReleaseTaggerBot added a project: MW-1.45-notes (1.45.0-wmf.19; 2025-09-16).Sep 11 2025, 7:00 PM
gerritbot added a comment.Sep 12 2025, 11:24 AM

Change #1187513 had a related patch set uploaded (by D3r1ck01; author: Derick Alangi):

[mediawiki/core@master] session: Follow-up for I10101c8b928a12 (3fde556f95b4b1ce43)

https://gerrit.wikimedia.org/r/1187513

gerritbot added a comment.Sep 12 2025, 12:05 PM

Change #1187779 had a related patch set uploaded (by D3r1ck01; author: Derick Alangi):

[operations/mediawiki-config@master] session: Enable MultiBackendSessionStore on `group1` wikis

https://gerrit.wikimedia.org/r/1187779

gerritbot added a comment.Sep 12 2025, 12:09 PM

Change #1187781 had a related patch set uploaded (by D3r1ck01; author: Derick Alangi):

[operations/mediawiki-config@master] session: Enable MultiBackendSessionStore on `group2` wikis

https://gerrit.wikimedia.org/r/1187781

gerritbot added a comment.Sep 15 2025, 8:14 PM

Change #1187513 merged by jenkins-bot:

[mediawiki/core@master] session: Follow-up on I10101c8b928a12 (3fde556f95b4b1ce43)

https://gerrit.wikimedia.org/r/1187513

gerritbot added a comment.Sep 16 2025, 9:34 AM

Change #1188733 had a related patch set uploaded (by D3r1ck01; author: Derick Alangi):

[mediawiki/core@master] session: Improve logging and monitoring in SessionStore implementations

https://gerrit.wikimedia.org/r/1188733

gerritbot added a comment.Sep 25 2025, 1:10 AM

Change #1188733 merged by jenkins-bot:

[mediawiki/core@master] session: Improve logging and monitoring in SessionStore implementations

https://gerrit.wikimedia.org/r/1188733

ReleaseTaggerBot edited projects, added MW-1.45-notes (1.45.0-wmf.21; 2025-09-30); removed MW-1.45-notes (1.45.0-wmf.19; 2025-09-16).Sep 25 2025, 2:00 AM
gerritbot added a comment.Sep 25 2025, 11:58 AM

Change #1191351 had a related patch set uploaded (by D3r1ck01; author: Derick Alangi):

[mediawiki/core@wmf/1.45.0-wmf.20] session: Improve logging and monitoring in SessionStore implementations

https://gerrit.wikimedia.org/r/1191351

gerritbot added a comment.Sep 25 2025, 12:14 PM

Change #1191360 had a related patch set uploaded (by D3r1ck01; author: Derick Alangi):

[mediawiki/core@wmf/1.45.0-wmf.19] session: Improve logging and monitoring in SessionStore implementations

https://gerrit.wikimedia.org/r/1191360

gerritbot added a comment.Sep 25 2025, 12:33 PM

Change #1191378 had a related patch set uploaded (by D3r1ck01; author: Derick Alangi):

[operations/mediawiki-config@master] Enable multibackend session store on beta and testwiki

https://gerrit.wikimedia.org/r/1191378

gerritbot added a comment.Sep 25 2025, 1:16 PM

Change #1191360 merged by jenkins-bot:

[mediawiki/core@wmf/1.45.0-wmf.19] session: Improve logging and monitoring in SessionStore implementations

https://gerrit.wikimedia.org/r/1191360

gerritbot added a comment.Sep 25 2025, 1:16 PM

Change #1191351 merged by jenkins-bot:

[mediawiki/core@wmf/1.45.0-wmf.20] session: Improve logging and monitoring in SessionStore implementations

https://gerrit.wikimedia.org/r/1191351

Stashbot mentioned this in T399195: Update logging and monitoring for multiple session storage backends.Sep 25 2025, 1:27 PM

Mentioned in SAL (#wikimedia-operations) [2025-09-25T13:27:03Z] <tgr@deploy1003> Started scap sync-world: Backport for [[gerrit:1191359|objectcache: Add a hit/miss flag to CachedBagOStuff]], [[gerrit:1191360|session: Improve logging and monitoring in SessionStore implementations (T399195 T402808)]], [[gerrit:1191361|hCaptcha: Fix mock for StatsFactory]], [[gerrit:1191362|NewcomerTasks: Use StatsFactory unit test helper]], [[gerrit:1191350|objectcache: Add a hit/miss flag to CachedB

Stashbot added a comment.Sep 25 2025, 1:33 PM

Mentioned in SAL (#wikimedia-operations) [2025-09-25T13:33:33Z] <tgr@deploy1003> d3r1ck01, wmde-fisch, tgr: Backport for [[gerrit:1191359|objectcache: Add a hit/miss flag to CachedBagOStuff]], [[gerrit:1191360|session: Improve logging and monitoring in SessionStore implementations (T399195 T402808)]], [[gerrit:1191361|hCaptcha: Fix mock for StatsFactory]], [[gerrit:1191362|NewcomerTasks: Use StatsFactory unit test helper]], [[gerrit:1191350|objectcache: Add a hit/miss flag to Cache

Stashbot added a comment.Sep 25 2025, 1:43 PM

Mentioned in SAL (#wikimedia-operations) [2025-09-25T13:43:12Z] <tgr@deploy1003> Finished scap sync-world: Backport for [[gerrit:1191359|objectcache: Add a hit/miss flag to CachedBagOStuff]], [[gerrit:1191360|session: Improve logging and monitoring in SessionStore implementations (T399195 T402808)]], [[gerrit:1191361|hCaptcha: Fix mock for StatsFactory]], [[gerrit:1191362|NewcomerTasks: Use StatsFactory unit test helper]], [[gerrit:1191350|objectcache: Add a hit/miss flag to Cached

gerritbot added a comment.Sep 25 2025, 1:47 PM

Change #1191378 merged by jenkins-bot:

[operations/mediawiki-config@master] Enable multibackend session store on beta and testwiki

https://gerrit.wikimedia.org/r/1191378

Stashbot added a comment.Sep 25 2025, 1:48 PM

Mentioned in SAL (#wikimedia-operations) [2025-09-25T13:48:12Z] <tgr@deploy1003> Started scap sync-world: Backport for [[gerrit:1191378|Enable multibackend session store on beta and testwiki (T402808)]], [[gerrit:1191370|Pre-deploy Design Research participant recruitment survey on jawiki (T405577)]]

Stashbot mentioned this in T405577: Deploy QuickSurvey for research participant registration drive on jawiki.Sep 25 2025, 1:48 PM
Stashbot added a comment.Sep 25 2025, 1:54 PM

Mentioned in SAL (#wikimedia-operations) [2025-09-25T13:54:44Z] <tgr@deploy1003> tgr, d3r1ck01, dani: Backport for [[gerrit:1191378|Enable multibackend session store on beta and testwiki (T402808)]], [[gerrit:1191370|Pre-deploy Design Research participant recruitment survey on jawiki (T405577)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

ReleaseTaggerBot edited projects, added MW-1.45-notes (1.45.0-wmf.19; 2025-09-16); removed MW-1.45-notes (1.45.0-wmf.21; 2025-09-30).Sep 25 2025, 2:00 PM
Stashbot added a comment.Sep 25 2025, 2:04 PM

Mentioned in SAL (#wikimedia-operations) [2025-09-25T14:04:23Z] <tgr@deploy1003> Finished scap sync-world: Backport for [[gerrit:1191378|Enable multibackend session store on beta and testwiki (T402808)]], [[gerrit:1191370|Pre-deploy Design Research participant recruitment survey on jawiki (T405577)]] (duration: 16m 11s)

DAlangi_WMF changed the task status from Open to In Progress.Sep 26 2025, 10:53 AM
DAlangi_WMF added a comment.Sep 26 2025, 10:56 AM

Rough plan

Thursday, September 25th -> rollout to beta cluster and testwiki ✅
Monday, September 29th -> rollout to group0 wikis
Tuesday, September 30th -> rollout to group1 wikis
Wednesday, September 31st -> rollout to group2 wikis

For each day's rollout, we'll monitor to ensure everything is working smoothly as expected. If not, we'll have to block, fix the issues, then resume or potentially roll back.

DAlangi_WMF changed the status of subtask T405824: Develop Grafana dashboard for MediaWiki session store sessions from Open to In Progress.Sep 27 2025, 8:12 AM
gerritbot added a comment.Sep 29 2025, 8:04 AM

Change #1192051 had a related patch set uploaded (by D3r1ck01; author: Derick Alangi):

[mediawiki/core@master] tests: Expand tests for MultiBackendSessionStore

https://gerrit.wikimedia.org/r/1192051

gerritbot added a comment.Sep 29 2025, 1:03 PM

Change #1183132 merged by jenkins-bot:

[operations/mediawiki-config@master] session: Enable MultiBackendSessionStore on `group0` wikis

https://gerrit.wikimedia.org/r/1183132

Stashbot added a comment.Sep 29 2025, 1:03 PM

Mentioned in SAL (#wikimedia-operations) [2025-09-29T13:03:53Z] <lucaswerkmeister-wmde@deploy2002> Started scap sync-world: Backport for [[gerrit:1183132|session: Enable MultiBackendSessionStore on group0 wikis (T402808)]]

Stashbot added a comment.Sep 29 2025, 1:09 PM

Mentioned in SAL (#wikimedia-operations) [2025-09-29T13:09:43Z] <lucaswerkmeister-wmde@deploy2002> d3r1ck01, lucaswerkmeister-wmde: Backport for [[gerrit:1183132|session: Enable MultiBackendSessionStore on group0 wikis (T402808)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Stashbot added a comment.Sep 29 2025, 1:21 PM

Mentioned in SAL (#wikimedia-operations) [2025-09-29T13:21:45Z] <lucaswerkmeister-wmde@deploy2002> Finished scap sync-world: Backport for [[gerrit:1183132|session: Enable MultiBackendSessionStore on group0 wikis (T402808)]] (duration: 17m 52s)

gerritbot added a comment.Sep 30 2025, 1:38 PM

Change #1187779 merged by jenkins-bot:

[operations/mediawiki-config@master] session: Enable MultiBackendSessionStore on `group1` wikis

https://gerrit.wikimedia.org/r/1187779

Stashbot added a comment.Sep 30 2025, 1:38 PM

Mentioned in SAL (#wikimedia-operations) [2025-09-30T13:38:41Z] <lucaswerkmeister-wmde@deploy2002> Started scap sync-world: Backport for [[gerrit:1187779|session: Enable MultiBackendSessionStore on group1 wikis (T402808)]]

Stashbot added a comment.Sep 30 2025, 1:45 PM

Mentioned in SAL (#wikimedia-operations) [2025-09-30T13:45:07Z] <lucaswerkmeister-wmde@deploy2002> lucaswerkmeister-wmde, d3r1ck01: Backport for [[gerrit:1187779|session: Enable MultiBackendSessionStore on group1 wikis (T402808)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Stashbot added a comment.Sep 30 2025, 1:53 PM

Mentioned in SAL (#wikimedia-operations) [2025-09-30T13:53:21Z] <lucaswerkmeister-wmde@deploy2002> Finished scap sync-world: Backport for [[gerrit:1187779|session: Enable MultiBackendSessionStore on group1 wikis (T402808)]] (duration: 14m 40s)

gerritbot added a comment.Sep 30 2025, 4:32 PM

Change #1192585 had a related patch set uploaded (by D3r1ck01; author: Derick Alangi):

[mediawiki/core@master] session: Attempt to fetch authentication store first

https://gerrit.wikimedia.org/r/1192585

gerritbot added a comment.Oct 1 2025, 12:23 PM

Change #1192876 had a related patch set uploaded (by D3r1ck01; author: Derick Alangi):

[mediawiki/core@master] session: Handle `set()` edge-case with `false` as value to in-process cache

https://gerrit.wikimedia.org/r/1192876

gerritbot added a comment.Oct 1 2025, 1:03 PM

Change #1192876 merged by jenkins-bot:

[mediawiki/core@master] session: Handle an edge-case in MultiBackendSessionStore::set()

https://gerrit.wikimedia.org/r/1192876

gerritbot added a comment.Oct 1 2025, 1:15 PM

Change #1192884 had a related patch set uploaded (by D3r1ck01; author: Derick Alangi):

[mediawiki/core@wmf/1.45.0-wmf.20] session: Handle an edge-case in MultiBackendSessionStore::set()

https://gerrit.wikimedia.org/r/1192884

gerritbot added a comment.Oct 1 2025, 1:15 PM

Change #1192885 had a related patch set uploaded (by D3r1ck01; author: Derick Alangi):

[mediawiki/core@wmf/1.45.0-wmf.21] session: Handle an edge-case in MultiBackendSessionStore::set()

https://gerrit.wikimedia.org/r/1192885

ReleaseTaggerBot edited projects, added MW-1.45-notes (1.45.0-wmf.22; 2025-10-07); removed MW-1.45-notes (1.45.0-wmf.19; 2025-09-16).Oct 1 2025, 2:00 PM
gerritbot added a comment.Oct 1 2025, 3:09 PM

Change #1192585 merged by jenkins-bot:

[mediawiki/core@master] session: Lookup authenticated store first before anon store

https://gerrit.wikimedia.org/r/1192585

gerritbot added a comment.Oct 1 2025, 8:18 PM

Change #1192884 merged by jenkins-bot:

[mediawiki/core@wmf/1.45.0-wmf.20] session: Handle an edge-case in MultiBackendSessionStore::set()

https://gerrit.wikimedia.org/r/1192884

gerritbot added a comment.Oct 1 2025, 8:20 PM

Change #1192885 merged by jenkins-bot:

[mediawiki/core@wmf/1.45.0-wmf.21] session: Handle an edge-case in MultiBackendSessionStore::set()

https://gerrit.wikimedia.org/r/1192885

Stashbot added a comment.Oct 1 2025, 8:21 PM

Mentioned in SAL (#wikimedia-operations) [2025-10-01T20:21:31Z] <derick@deploy2002> Started scap sync-world: Backport for [[gerrit:1192884|session: Handle an edge-case in MultiBackendSessionStore::set() (T402808)]], [[gerrit:1192885|session: Handle an edge-case in MultiBackendSessionStore::set() (T402808)]]

Stashbot added a comment.Oct 1 2025, 8:27 PM

Mentioned in SAL (#wikimedia-operations) [2025-10-01T20:27:48Z] <derick@deploy2002> derick, d3r1ck01: Backport for [[gerrit:1192884|session: Handle an edge-case in MultiBackendSessionStore::set() (T402808)]], [[gerrit:1192885|session: Handle an edge-case in MultiBackendSessionStore::set() (T402808)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Stashbot added a comment.Oct 1 2025, 8:34 PM

Mentioned in SAL (#wikimedia-operations) [2025-10-01T20:34:28Z] <derick@deploy2002> Finished scap sync-world: Backport for [[gerrit:1192884|session: Handle an edge-case in MultiBackendSessionStore::set() (T402808)]], [[gerrit:1192885|session: Handle an edge-case in MultiBackendSessionStore::set() (T402808)]] (duration: 12m 57s)

ReleaseTaggerBot edited projects, added MW-1.45-notes (1.45.0-wmf.21; 2025-09-30); removed MW-1.45-notes (1.45.0-wmf.22; 2025-10-07).Oct 1 2025, 9:00 PM
gerritbot added a comment.Oct 2 2025, 9:42 AM

Change #1193067 had a related patch set uploaded (by D3r1ck01; author: Derick Alangi):

[mediawiki/core@wmf/1.45.0-wmf.21] session: Lookup authenticated store first before anon store

https://gerrit.wikimedia.org/r/1193067

gerritbot added a comment.Oct 2 2025, 9:52 AM

Change #1193069 had a related patch set uploaded (by D3r1ck01; author: Derick Alangi):

[mediawiki/core@wmf/1.45.0-wmf.20] session: Lookup authenticated store first before anon store

https://gerrit.wikimedia.org/r/1193069

Tgr added a comment.Oct 2 2025, 10:46 AM

Just so we have a written track record of IRC discussions here:

  1. The group 1 rollout cause a serious increase in Cassandra reads and Logstash Duplicate get(): "{key}" fetched {count} times logs (and had to be rolled back temporarily). That's because SessionManager::generateSessionId() warms the cache (marks the newly generated ID as not having any data to avoid a lookup) but due to the way MultiBackendSessionStore checks both stores when unsure which one to use, the session ID generation resulted in a real (uncached) store lookup. generateSessionId() is used a lot, even when the user doesn't actually have a session, so even for group1, session store load went way up.
Screenshot Capture - 2025-10-02 - 12-35-25.png (440×872 px, 60 KB)
Screenshot Capture - 2025-10-02 - 12-37-21.png (452×672 px, 49 KB)
BagOStuff/MWSession logsKask logs

Fixed by rMWc44e3ed7499c: session: Handle an edge-case in MultiBackendSessionStore::set().

  1. There's a smaller but still quite significant increase in duplicate gets:
Screenshot Capture - 2025-10-02 - 12-44-04.png (530×1 px, 72 KB)
logstash

This is probably due to MultiBackendSessionStore checking both backends when in doubt (even if it finds data in the store that's checked first), which should be fixed by rMWf5e029dea188: session: Lookup authenticated store first before anon store.

gerritbot added a comment.Oct 2 2025, 1:20 PM

Change #1193067 merged by jenkins-bot:

[mediawiki/core@wmf/1.45.0-wmf.21] session: Lookup authenticated store first before anon store

https://gerrit.wikimedia.org/r/1193067

gerritbot added a comment.Oct 2 2025, 1:20 PM

Change #1193069 merged by jenkins-bot:

[mediawiki/core@wmf/1.45.0-wmf.20] session: Lookup authenticated store first before anon store

https://gerrit.wikimedia.org/r/1193069

Stashbot added a comment.Oct 2 2025, 1:21 PM

Mentioned in SAL (#wikimedia-operations) [2025-10-02T13:21:07Z] <lucaswerkmeister-wmde@deploy2002> Started scap sync-world: Backport for [[gerrit:1193067|session: Lookup authenticated store first before anon store (T402808)]], [[gerrit:1193069|session: Lookup authenticated store first before anon store (T402808)]]

Stashbot added a comment.Oct 2 2025, 1:27 PM

Mentioned in SAL (#wikimedia-operations) [2025-10-02T13:27:14Z] <lucaswerkmeister-wmde@deploy2002> d3r1ck01, lucaswerkmeister-wmde: Backport for [[gerrit:1193067|session: Lookup authenticated store first before anon store (T402808)]], [[gerrit:1193069|session: Lookup authenticated store first before anon store (T402808)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Stashbot added a comment.Oct 2 2025, 1:34 PM

Mentioned in SAL (#wikimedia-operations) [2025-10-02T13:34:03Z] <lucaswerkmeister-wmde@deploy2002> Finished scap sync-world: Backport for [[gerrit:1193067|session: Lookup authenticated store first before anon store (T402808)]], [[gerrit:1193069|session: Lookup authenticated store first before anon store (T402808)]] (duration: 12m 56s)

ReleaseTaggerBot edited projects, added MW-1.45-notes (1.45.0-wmf.20; 2025-09-23); removed MW-1.45-notes (1.45.0-wmf.21; 2025-09-30).Oct 2 2025, 2:00 PM
Tgr mentioned this in T373826: NetworkSessionProvider / CirrusSearch Streaming Updater causing 'session' log spam and possibly Sessionstore (Kask) problems.Oct 2 2025, 2:25 PM
Swpb mentioned this in T406242: "invalid CSRF token" from Commons in wikibase-cli.Oct 2 2025, 4:08 PM
Tgr added a subtask: T405633: Session data is authenticated, should not be an anonymous user.Oct 5 2025, 9:15 PM
Tgr added a comment.Oct 5 2025, 9:34 PM

There's 20K Authenticated data should not be in the anonymous store log entries in the last 7 days: https://logstash.wikimedia.org/goto/201c27467af11091c6ce70064ec2b682
This is the same error condition as T405633: Session data is authenticated, should not be an anonymous user (we use the authenticated store but the user ID in the data is 0) except it happens during get() rather than set(). So maybe the same issue? Maybe it happens when we are reading back those set()s? It happens about 10x less often though.

Tgr added a comment.Oct 5 2025, 9:40 PM

We have the opposite as well, Anonymous data should not be in the authenticated store, although another magnitude less: https://logstash.wikimedia.org/goto/eaf7ca64cd616f3ff220d029b62effec

DAlangi_WMF closed subtask T405824: Develop Grafana dashboard for MediaWiki session store sessions as Resolved.Oct 6 2025, 10:34 AM
Tgr added a subtask: T405634: Authenticated data should not be in the anonymous store.Oct 7 2025, 7:45 PM
Tgr added a subtask: T406560: Anonymous data should not be in the authenticated store.
gerritbot added a comment.Oct 8 2025, 3:25 PM

Change #1194663 had a related patch set uploaded (by D3r1ck01; author: Derick Alangi):

[mediawiki/core@master] session: Improve logging for MultiBackendSessionStore

https://gerrit.wikimedia.org/r/1194663

gerritbot added a comment.Oct 9 2025, 2:20 PM

Change #1194663 merged by jenkins-bot:

[mediawiki/core@master] session: Improve logging for MultiBackendSessionStore

https://gerrit.wikimedia.org/r/1194663

gerritbot added a comment.Oct 9 2025, 2:43 PM

Change #1194963 had a related patch set uploaded (by D3r1ck01; author: Derick Alangi):

[mediawiki/core@wmf/1.45.0-wmf.21] session: Improve logging for MultiBackendSessionStore

https://gerrit.wikimedia.org/r/1194963

gerritbot added a comment.Oct 9 2025, 2:43 PM

Change #1194964 had a related patch set uploaded (by D3r1ck01; author: Derick Alangi):

[mediawiki/core@wmf/1.45.0-wmf.22] session: Improve logging for MultiBackendSessionStore

https://gerrit.wikimedia.org/r/1194964

gerritbot added a comment.Oct 9 2025, 4:09 PM

Change #1194963 merged by jenkins-bot:

[mediawiki/core@wmf/1.45.0-wmf.21] session: Improve logging for MultiBackendSessionStore

https://gerrit.wikimedia.org/r/1194963

gerritbot added a comment.Oct 9 2025, 4:10 PM

Change #1194964 merged by jenkins-bot:

[mediawiki/core@wmf/1.45.0-wmf.22] session: Improve logging for MultiBackendSessionStore

https://gerrit.wikimedia.org/r/1194964

Stashbot mentioned this in T405634: Authenticated data should not be in the anonymous store.Oct 9 2025, 4:10 PM

Mentioned in SAL (#wikimedia-operations) [2025-10-09T16:10:27Z] <tgr@deploy2002> Started scap sync-world: Backport for [[gerrit:1194963|session: Improve logging for MultiBackendSessionStore (T402808 T405633 T405634)]], [[gerrit:1194964|session: Improve logging for MultiBackendSessionStore (T402808 T405633 T405634)]]

Stashbot mentioned this in T405633: Session data is authenticated, should not be an anonymous user.Oct 9 2025, 4:10 PM
Stashbot added a comment.Oct 9 2025, 4:14 PM

Mentioned in SAL (#wikimedia-operations) [2025-10-09T16:14:09Z] <tgr@deploy2002> tgr, d3r1ck01: Backport for [[gerrit:1194963|session: Improve logging for MultiBackendSessionStore (T402808 T405633 T405634)]], [[gerrit:1194964|session: Improve logging for MultiBackendSessionStore (T402808 T405633 T405634)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Stashbot added a comment.Oct 9 2025, 4:30 PM

Mentioned in SAL (#wikimedia-operations) [2025-10-09T16:30:34Z] <tgr@deploy2002> Finished scap sync-world: Backport for [[gerrit:1194963|session: Improve logging for MultiBackendSessionStore (T402808 T405633 T405634)]], [[gerrit:1194964|session: Improve logging for MultiBackendSessionStore (T402808 T405633 T405634)]] (duration: 20m 07s)

Tgr mentioned this in T406723: Duplicate get(): "{key}" fetched {count} times.Oct 9 2025, 4:34 PM
gerritbot added a comment.Oct 9 2025, 7:05 PM

Change #1195029 had a related patch set uploaded (by D3r1ck01; author: Derick Alangi):

[mediawiki/core@master] session: Avoid logging in session store if UserInfo is available

https://gerrit.wikimedia.org/r/1195029

DAlangi_WMF closed subtask T405634: Authenticated data should not be in the anonymous store as Resolved.Oct 9 2025, 7:11 PM
gerritbot added a comment.Oct 13 2025, 1:03 PM

Change #1187781 merged by jenkins-bot:

[operations/mediawiki-config@master] session: Enable MultiBackendSessionStore on `group2` wikis

https://gerrit.wikimedia.org/r/1187781

Stashbot added a comment.Oct 13 2025, 1:03 PM

Mentioned in SAL (#wikimedia-operations) [2025-10-13T13:03:45Z] <derick@deploy2002> Started scap sync-world: Backport for [[gerrit:1187781|session: Enable MultiBackendSessionStore on group2 wikis (T402808)]]

Stashbot added a comment.Oct 13 2025, 1:08 PM

Mentioned in SAL (#wikimedia-operations) [2025-10-13T13:08:03Z] <derick@deploy2002> derick, d3r1ck01: Backport for [[gerrit:1187781|session: Enable MultiBackendSessionStore on group2 wikis (T402808)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Stashbot added a comment.Oct 13 2025, 1:15 PM

Mentioned in SAL (#wikimedia-operations) [2025-10-13T13:15:23Z] <derick@deploy2002> Finished scap sync-world: Backport for [[gerrit:1187781|session: Enable MultiBackendSessionStore on group2 wikis (T402808)]] (duration: 11m 39s)

DAlangi_WMF closed this task as Resolved.Oct 13 2025, 4:35 PM
DAlangi_WMF closed subtask T406560: Anonymous data should not be in the authenticated store as Resolved.
DAlangi_WMF closed subtask T405633: Session data is authenticated, should not be an anonymous user as Resolved.

Deployed to group2 today, will file a task to monitor logs for about a week or so. Resolving this now.

gerritbot added a comment.Oct 24 2025, 12:00 PM

Change #1195029 abandoned by D3r1ck01:

[mediawiki/core@master] session: Avoid logging in session store if UserInfo is available

https://gerrit.wikimedia.org/r/1195029

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits