Page MenuHomePhabricator

Log when a request with the same user session comes from a different IP
Closed, ResolvedPublic

Description

To have better understanding of session leakage incidents like {T264369}, we should try logging whenever the same session starts coming from a different IP. This might end up too noisy to be useful, but is worth a try. Another option is using the <wiki>mwuser-sessionId cookies (used for client-side analytics, not related to the auth system).

I think we did something similar for {T150554} in the past, but I might be misremembering.

Event Timeline

I think we did something similar for {T150554} in the past, but I might be misremembering.

In fact we did it during the SessionManager deployment. The related task is T125455: Track ip addresses associated with a session and log when anomalous.

Change 632984 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[mediawiki/core@master] Log IP/device changes within the same session

https://gerrit.wikimedia.org/r/632984

Tgr triaged this task as Unbreak Now! priority.Oct 8 2020, 9:03 PM
Tgr added a project: Platform Engineering.

Marking UBN as we want this in place before restarting the train.

This task is part of a plan @Tgr proposed at T264369#6523086 and it took a few round of discussion before identifying which ones should be prioritized and would end up as actual train blockers.

I think that the last one we need, beside patches to mediawiki-config change.

We would like to roll wmf.11 on Tuesday ( https://lists.wikimedia.org/pipermail/wikitech-l/2020-October/093944.html )

Change 632984 merged by jenkins-bot:
[mediawiki/core@master] Log IP/device changes within the same session

https://gerrit.wikimedia.org/r/632984

Change 633209 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[mediawiki/core@master] SessionManager: Always log IP/UA in session-ip

https://gerrit.wikimedia.org/r/633209

Change 633210 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[operations/mediawiki-config@master] Enable session-ip log channel on group0

https://gerrit.wikimedia.org/r/633210

Change 633209 merged by jenkins-bot:
[mediawiki/core@master] SessionManager: Always log IP/UA in session-ip

https://gerrit.wikimedia.org/r/633209

Rollout plan:

  • sync 632984 and 633209 (should be a no-op, code is behind a session flag)
  • set $wgSuspiciousIpExpiry = 600 for group 0, then group1 except Commons and Wikidata, then Commons, then Wikidata, then a largish wiki (e.g. nlwiki), then group2
  • in each step, check logstash volume on the session-ip channel and sessionstore POST volume, and stop if the increase seems significant. Maybe try a larger expiry and see if that helps.

Change 633252 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[mediawiki/core@wmf/1.36.0-wmf.10] Log IP/device changes within the same session

https://gerrit.wikimedia.org/r/633252

Change 633253 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[mediawiki/core@wmf/1.36.0-wmf.11] Log IP/device changes within the same session

https://gerrit.wikimedia.org/r/633253

Change 633254 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[mediawiki/core@wmf/1.36.0-wmf.10] SessionManager: Always log IP/UA in session-ip

https://gerrit.wikimedia.org/r/633254

Change 633255 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[mediawiki/core@wmf/1.36.0-wmf.11] SessionManager: Always log IP/UA in session-ip

https://gerrit.wikimedia.org/r/633255

Change 633252 merged by jenkins-bot:
[mediawiki/core@wmf/1.36.0-wmf.10] Log IP/device changes within the same session

https://gerrit.wikimedia.org/r/633252

Change 633253 merged by jenkins-bot:
[mediawiki/core@wmf/1.36.0-wmf.11] Log IP/device changes within the same session

https://gerrit.wikimedia.org/r/633253

Change 633254 merged by jenkins-bot:
[mediawiki/core@wmf/1.36.0-wmf.10] SessionManager: Always log IP/UA in session-ip

https://gerrit.wikimedia.org/r/633254

Change 633255 merged by jenkins-bot:
[mediawiki/core@wmf/1.36.0-wmf.11] SessionManager: Always log IP/UA in session-ip

https://gerrit.wikimedia.org/r/633255

Mentioned in SAL (#wikimedia-operations) [2020-10-09T22:09:43Z] <tgr@deploy1001> Synchronized php-1.36.0-wmf.10/includes/: Backport: [[gerrit:633252|Log IP/device changes within the same session (T264799)]] & [[gerrit:633254|SessionManager: Always log IP/UA in session-ip]] (duration: 01m 06s)

Change 633210 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable session-ip log channel on group0

https://gerrit.wikimedia.org/r/633210

Mentioned in SAL (#wikimedia-operations) [2020-10-09T22:20:13Z] <tgr@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:633210|Enable session-ip log channel on group0 (T264799)]] (duration: 00m 59s)

Mentioned in SAL (#wikimedia-operations) [2020-10-09T22:23:28Z] <tgr@deploy1001> Synchronized php-1.36.0-wmf.11/includes/: Backport: [[gerrit:633252|Log IP/device changes within the same session (T264799)]] & [[gerrit:633254|SessionManager: Always log IP/UA in session-ip]] (duration: 01m 04s)

Change 633271 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[operations/mediawiki-config@master] Enable session-ip log channel on group1, except Commons/Wikidata

https://gerrit.wikimedia.org/r/633271

Change 633271 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable session-ip log channel on group1, except Commons/Wikidata

https://gerrit.wikimedia.org/r/633271

Mentioned in SAL (#wikimedia-operations) [2020-10-09T22:52:15Z] <tgr@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:633271|Enable session-ip log channel on group1, except Commons/Wikidata (T264799)]] (duration: 00m 57s)

Change 633272 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[operations/mediawiki-config@master] Enable session-ip log channel on Commons

https://gerrit.wikimedia.org/r/633272

Change 633272 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable session-ip log channel on Commons

https://gerrit.wikimedia.org/r/633272

Mentioned in SAL (#wikimedia-operations) [2020-10-09T23:25:12Z] <tgr@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:633272|Enable session-ip log channel on Commons (T264799)]] (duration: 00m 59s)

Change 633274 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[operations/mediawiki-config@master] Enable session-ip log channel on Wikidata

https://gerrit.wikimedia.org/r/633274

Change 633274 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable session-ip log channel on Wikidata

https://gerrit.wikimedia.org/r/633274

Mentioned in SAL (#wikimedia-operations) [2020-10-09T23:44:58Z] <tgr@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:633274|Enable session-ip log channel on Wikidata (T264799)]] (duration: 00m 59s)

Change 633276 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[operations/mediawiki-config@master] Enable session-ip log channel on eswiki

https://gerrit.wikimedia.org/r/633276

Change 633276 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable session-ip log channel on eswiki

https://gerrit.wikimedia.org/r/633276

Mentioned in SAL (#wikimedia-operations) [2020-10-10T00:18:51Z] <tgr@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:633276|Enable session-ip log channel on eswiki (T264799)]] (duration: 00m 55s)

Change 633277 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[operations/mediawiki-config@master] Enable session-ip log channel on all but enwiki

https://gerrit.wikimedia.org/r/633277

Change 633277 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable session-ip log channel on all but enwiki

https://gerrit.wikimedia.org/r/633277

Mentioned in SAL (#wikimedia-operations) [2020-10-10T00:54:50Z] <tgr@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:633277|Enable session-ip log channel on all but enwiki (T264799)]] (duration: 01m 01s)

Change 633281 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[operations/mediawiki-config@master] Enable session-ip log channel everywhere

https://gerrit.wikimedia.org/r/633281

Change 633281 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable session-ip log channel everywhere

https://gerrit.wikimedia.org/r/633281

Mentioned in SAL (#wikimedia-operations) [2020-10-10T01:32:07Z] <tgr@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:633281|Enable session-ip log channel everywhere (T264799)]] (duration: 00m 59s)

It seems like the assumption that the mwuser cookie never changes within a session was wrong; I see a fair amount of mwuser going from some value to null. It is a session cookie and we are never unsetting it; the affected URLs tend to be scripts with a cross-wiki referrer, so this must be some kind of third-party cookie blocking browser feature (or maybe an extension with a cookie blocklist, as I'm not sure how else it could tell the auth cookie and the mwuser cookie apart). I have a patch for filtering these out, but I'm not sure it is worth the effort - they are easy enough to filter out in Kibana, they are only something like 25% of total logs, and filtering them out would also remove some true positives (where a session has been leaked to a user with no mwuser cookie).

Tgr claimed this task.