Improve MediaWiki session logging
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Tgr
	Apr 30 2025, 5:45 PM

Description

Logging was of limited use for {T390514} and should be improved:

all session writes should be logged, not just session creations
differentiate between writes to new keys and rewrites
gain more confidence that session creations are actually logged (seems like only 10-15% of session writes create a new session, which seems too low given how many sessions seem to be associated with scrapers)
be able to filter by session provider type
be able to filter by app, if easily doable
make sure client-identifying information (like IP/UA) are logged

Details

Related Changes in Gerrit:

Subject	Repo	Branch	Lines +/-
Improve session logging	mediawiki/core	wmf/1.44.0-wmf.28	+34 -22
Improve session logging	mediawiki/core	master	+34 -22
session: Add logging for session store writes	mediawiki/core	master	+112 -24

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
					Restricted Task
		Resolved		Tgr	T393038 Improve MediaWiki session logging

Event Timeline

Tgr created this task.Apr 30 2025, 5:45 PM

Restricted Application added a project: MediaWiki-Platform-Team. · View Herald TranscriptApr 30 2025, 5:45 PM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Tgr mentioned this in T392182: sessionstore workload observability.Apr 30 2025, 5:46 PM

Change #1140683 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[mediawiki/core@master] session: Add logging for session store writes

https://gerrit.wikimedia.org/r/1140683

gerritbot added a project: Patch-For-Review.May 2 2025, 11:59 AM

Eevans added a parent task: Restricted Task.May 2 2025, 9:15 PM

Change #1140683 merged by jenkins-bot:

[mediawiki/core@master] session: Add logging for session store writes

https://gerrit.wikimedia.org/r/1140683

Maintenance_bot removed a project: Patch-For-Review.May 4 2025, 2:30 PM

ReleaseTaggerBot added a project: MW-1.44-notes (1.44.0-wmf.28; 2025-05-06).May 4 2025, 3:00 PM

pmiazga moved this task from Inbox, needs triage to In progress (DO NOT USE) on the MediaWiki-Platform-Team board.May 5 2025, 2:45 PM

Tgr claimed this task.May 5 2025, 2:45 PM

Dashboard: https://logstash.wikimedia.org/app/dashboards#/view/174d68c0-2c45-11f0-a1ab-6f85ab09a17d
Not super informative so far. Almost all sessions are from CentralAuth, not OAuth etc. (expected, but good to verify). Anonymous sessions are about 80% of total. Login/signup are about 70% of total, autologin is 5%. About 10-15% of writes are for API requests, the rest for web. There are a few users who do something weird with bots and have tens of thousands of session writes per day, but that's still just a drop in the bucket. IP and user agent look like a normal spread.

The number of events is exactly 1080000 for every 3 hour block on the histogram, which looks very fake. Logstash might be hitting some sort of throttling there?

Change #1143959 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[mediawiki/core@master] Improve session logging

https://gerrit.wikimedia.org/r/1143959

gerritbot added a project: Patch-For-Review.May 11 2025, 1:11 PM

In T393038#10808407, @Tgr wrote:

The number of events is exactly 1080000 for every 3 hour block on the histogram

which is exactly 100 / second, so yeah. The Prometheus stats imply there are something like 800 writes per second. So I guess the dashboard should be thought of as having been downsampled by one magnitude?

Not sure if this is a limitation of the dashboard, or the log event intake per normalized message (not by channel for sure, the session channel has way more stuff).

Change #1143959 merged by jenkins-bot:

[mediawiki/core@master] Improve session logging

https://gerrit.wikimedia.org/r/1143959

Change #1144532 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[mediawiki/core@wmf/1.44.0-wmf.28] Improve session logging

https://gerrit.wikimedia.org/r/1144532

ReleaseTaggerBot added a project: MW-1.45-notes (1.45.0-wmf.1; 2025-05-13).May 12 2025, 1:00 PM

Change #1144532 merged by jenkins-bot:

[mediawiki/core@wmf/1.44.0-wmf.28] Improve session logging

https://gerrit.wikimedia.org/r/1144532

Mentioned in SAL (#wikimedia-operations) [2025-05-12T13:59:41Z] <tgr@deploy1003> Started scap sync-world: Backport for [[gerrit:1144532|Improve session logging (T393038)]]

Mentioned in SAL (#wikimedia-operations) [2025-05-12T14:04:35Z] <tgr@deploy1003> tgr: Backport for [[gerrit:1144532|Improve session logging (T393038)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2025-05-12T14:17:05Z] <tgr@deploy1003> Finished scap sync-world: Backport for [[gerrit:1144532|Improve session logging (T393038)]] (duration: 17m 24s)

Maintenance_bot removed a project: Patch-For-Review.May 12 2025, 2:31 PM

I think this is done, the current logs are good enough.

Tgr mentioned this in T392251: SessionBackend seems to store session changes too often.Jun 1 2025, 5:10 PM

Tgr mentioned this in T395899: Some subset of MediaWiki Logstash events are capped to 100/s.Jun 3 2025, 12:58 PM