Page MenuHomePhabricator

Enable csp-report-only mode everywhere
Closed, ResolvedPublicSecurity

Description

or at least on more wikis.

Expected consequences:

  • Uptick in amount of traffic to logstash (If problematic we could log to mwlog1001 only)
  • Uptick in reports to API

Based on mw.org, it seems very roughly like a wiki the size of mw.org gets about 30 hits/minute on average, with ocassional spikes to 150/minute

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 24 2018, 9:56 PM
faidon added subscribers: herron, fgiunchedi, faidon.

Cool! Cc'ing @herron and @fgiunchedi here for awareness and their input. Logstash may or may not be happy about the extra load, depending on how much that would be (esp. for big wikis) :)

Change 469527 had a related patch set uploaded (by Brian Wolff; owner: Brian Wolff):
[operations/mediawiki-config@master] Enable csp report only on outreachwiki

https://gerrit.wikimedia.org/r/469527

Cool! Cc'ing @herron and @fgiunchedi here for awareness and their input. Logstash may or may not be happy about the extra load, depending on how much that would be (esp. for big wikis) :)

I'm going to do this on some small wikis, but won't do anything to big wikis until I talk to @herron or @fgiunchedi

Change 469527 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable csp report only on outreachwiki

https://gerrit.wikimedia.org/r/469527

Mentioned in SAL (#wikimedia-operations) [2018-10-24T22:36:19Z] <bawolff@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Deploy csp report-only to outreachwiki T207900 (duration: 00m 54s)

Mentioned in SAL (#wikimedia-operations) [2018-10-24T22:38:13Z] <bawolff@deploy1001> Synchronized wmf-config/CommonSettings.php: Deploy csp report-only to outreachwiki T207900 (duration: 00m 54s)

Change 469538 had a related patch set uploaded (by Brian Wolff; owner: Brian Wolff):
[operations/mediawiki-config@master] Enable CSP-report-only on small wikis

https://gerrit.wikimedia.org/r/469538

Change 469538 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable CSP-report-only on small wikis

https://gerrit.wikimedia.org/r/469538

Mentioned in SAL (#wikimedia-operations) [2018-10-24T23:08:33Z] <bawolff@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Deploy csp report-only to small.dblist wikis T207900 (duration: 00m 56s)

[Just for context, i did small wikis, but I'll wait until talking to logstash folks before doing big wikis]

Cool! Cc'ing @herron and @fgiunchedi here for awareness and their input. Logstash may or may not be happy about the extra load, depending on how much that would be (esp. for big wikis) :)

I'm going to do this on some small wikis, but won't do anything to big wikis until I talk to @herron or @fgiunchedi

Thanks! Is it possible to get a sense of how many additional logs will csp reports be when fully on ? Is it potentially one per request or gets deduplicated on some level? Also I'm assuming we're talking about a single channel csp-report-only ?
I'd say lets go ahead and do group0 + group1 and see where we're at. Also we have icinga alerts for logstash dropping packets now, if sth goes obviously sideways those alerts will fire.

Tgr added a subscriber: Tgr.Oct 25 2018, 8:12 PM

Based on mw.org, it seems very roughly like a wiki the size of mw.org gets about 30 hits/minute on average, with ocassional spikes to 150/minute

mediawiki.org is ~0.05% of total traffic so that would be 1000 hit/s if enabled everywhere. IIRC that's ~25% of current action API traffic and roughly equal to current logstash traffic. So for enabling everywhere, maybe the analytics infrastructure would be a better match instead of API + logstash?

OTOH there is relatively little value for CSP reports of logged-out requests - security-wise they are not much of a surface, privacy-wise you probably get the same insights from logged-in requests as well. Limiting CSP headers to logged-in requests would reduce load by three magnitudes or so, making it pretty tame for the action API and logstash, and unlike random sampling it works well with Varnish.

(The assumptions here are that 1) load scales linearly with wiki size and 2) report frequency is the same for logged-in users and non-logged in users. Those are true if the reports come from random things not related to us, such as Chrome plugins injecting their own scripts into the page. They are probably not true if the reports mainly come from our gadgets. mediawiki.org gets about 200 pageviews per minute currently, so if 15% of that results in CSP reports, I think it's a reasonable assumption that they mostly are from anons and thus mostly aren't from gadgets.)

I'd say lets go ahead and do group0 + group1 and see where we're at. Also we have icinga alerts for logstash dropping packets now, if sth goes obviously sideways those alerts will fire.

Awesome.

As tgr said, further discussion suggested we should do this for logged in users only, as the primary group we care about and much less traffic.

I'd kind of like to deploy this in stages (for logged in users. I'd like to keep on for logged out on wikis where we have it set so right now):

enwikiquote->all wikiquote->group1->medium.dblist->arwiki-><other stuff>

Change 469800 had a related patch set uploaded (by Brian Wolff; owner: Brian Wolff):
[operations/mediawiki-config@master] Enable CSP-report-only for logged in/session having users on enwikiquote

https://gerrit.wikimedia.org/r/469800

Change 469800 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable CSP-report-only for logged in/session having users on enwikiquote

https://gerrit.wikimedia.org/r/469800

Mentioned in SAL (#wikimedia-operations) [2018-10-26T01:19:56Z] <bawolff@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T207900 - deploy CSP to people with session on enwikiquote (duration: 00m 55s)

Mentioned in SAL (#wikimedia-operations) [2018-10-26T01:21:18Z] <bawolff@deploy1001> Synchronized wmf-config/CommonSettings.php: T207900 - deploy CSP to people with session on enwikiquote (duration: 00m 54s)

Change 469815 had a related patch set uploaded (by Brian Wolff; owner: Brian Wolff):
[operations/mediawiki-config@master] Enable CSP report only for session users on group1 wikis.

https://gerrit.wikimedia.org/r/469815

Change 469815 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable CSP report only for session users on group1 wikis.

https://gerrit.wikimedia.org/r/469815

Mentioned in SAL (#wikimedia-operations) [2018-10-26T01:41:16Z] <bawolff@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T207900 (b74911f6201) enable csp users with session all group1 wikis (duration: 00m 55s)

Change 469816 had a related patch set uploaded (by Brian Wolff; owner: Brian Wolff):
[operations/mediawiki-config@master] Enable CSP report only with session on medium sized wikis.

https://gerrit.wikimedia.org/r/469816

Change 469816 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable CSP report only with session on medium sized wikis.

https://gerrit.wikimedia.org/r/469816

Mentioned in SAL (#wikimedia-operations) [2018-10-26T02:05:41Z] <bawolff@deploy1001> Synchronized wmf-config/InitialiseSettings.php: bd55034d122 - T207900 - enable CSP report only for users w/session on medium wikis (duration: 00m 55s)

Change 469817 had a related patch set uploaded (by Brian Wolff; owner: Brian Wolff):
[operations/mediawiki-config@master] Enable csp report only for users w/session on arwiki

https://gerrit.wikimedia.org/r/469817

Change 469817 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable csp report only for users w/session on arwiki

https://gerrit.wikimedia.org/r/469817

Mentioned in SAL (#wikimedia-operations) [2018-10-26T02:22:27Z] <bawolff@deploy1001> Synchronized wmf-config/InitialiseSettings.php: d743db261 - T207900 - enable CSP report only for users w/session arwiki (duration: 00m 54s)

Change 469819 had a related patch set uploaded (by Brian Wolff; owner: Brian Wolff):
[operations/mediawiki-config@master] Enable csp report only for users w/session on a bunch of big wikis

https://gerrit.wikimedia.org/r/469819

Change 469819 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable csp report only for users w/session on a bunch of big wikis

https://gerrit.wikimedia.org/r/469819

Mentioned in SAL (#wikimedia-operations) [2018-10-26T02:43:50Z] <bawolff@deploy1001> Synchronized wmf-config/InitialiseSettings.php: a8aa9d6aae - T207900 - enable CSP report only for users w/session fawiki, frwiki, svwiki, eswiki, ruwiki, zhwiki, dewiki (duration: 00m 56s)

Change 469820 had a related patch set uploaded (by Brian Wolff; owner: Brian Wolff):
[operations/mediawiki-config@master] Enable csp report only for users w/session on enwiki

https://gerrit.wikimedia.org/r/469820

Change 469820 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable csp report only for users w/session on enwiki

https://gerrit.wikimedia.org/r/469820

Nirmos added a subscriber: Nirmos.Oct 26 2018, 2:53 AM

Mentioned in SAL (#wikimedia-operations) [2018-10-26T02:54:48Z] <bawolff@deploy1001> Synchronized wmf-config/InitialiseSettings.php: 745d0b61 - T207900 - enable CSP report only for users w/session enwiki (duration: 00m 53s)

Mentioned in SAL (#wikimedia-operations) [2018-10-26T03:01:06Z] <bawolff@deploy1001> Synchronized wmf-config/CommonSettings.php: T207900 - Add wikimedia.org (no subdomain) to allow list for math (duration: 00m 53s)

Change 469821 had a related patch set uploaded (by Brian Wolff; owner: Brian Wolff):
[operations/mediawiki-config@master] Enable csp report only for people with session everywhere.

https://gerrit.wikimedia.org/r/469821

Change 469821 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable csp report only for people with session everywhere.

https://gerrit.wikimedia.org/r/469821

Mentioned in SAL (#wikimedia-operations) [2018-10-26T03:10:24Z] <bawolff@deploy1001> Synchronized wmf-config/InitialiseSettings.php: 745d0b61 - T207900 - enable CSP report only for users w/session enwiki (duration: 00m 55s)

Mentioned in SAL (#wikimedia-operations) [2018-10-26T03:19:52Z] <bawolff@deploy1001> Synchronized wmf-config/InitialiseSettings.php: bc9b863e - T207900 - enable CSP report only for users w/session everywhere (duration: 00m 55s)

chasemp triaged this task as Normal priority.Oct 26 2018, 6:41 PM
chasemp added a project: Restricted Project.
chasemp added a subscriber: BBlack.
chasemp assigned this task to Bawolff.Oct 26 2018, 6:59 PM
chasemp added a subscriber: chasemp.

@Bawolff I have you as point person here from the relevant meeting so I'm going to go ahead and assign

chasemp moved this task from Restricted Project Column to Restricted Project Column on the Restricted Project board.Oct 26 2018, 7:06 PM
chasemp changed the subtype of this task from "Task" to "Security Issue".Oct 26 2018, 7:13 PM

In response to this being unexpectedly enabled on all wikis for non-script fetches, I felt obligated to disable most Toolforge-related features in scripts and gadget I maintain for the community. For example:

https://meta.wikimedia.org/w/index.php?title=User:Krinkle/Scripts/CVNSimpleOverlay_wiki.js&diff=prev&oldid=18523572
https://meta.wikimedia.org/w/index.php?title=User:Krinkle/Scripts/Intuition.js&diff=prev&oldid=18523589

Bawolff closed this task as Resolved.Nov 1 2018, 8:30 PM

Ok, so this is done for now. If we eventually enable this report only for anons, we can use a separate ticket.

chasemp moved this task from Restricted Project Column to Restricted Project Column on the Restricted Project board.Nov 2 2018, 3:43 PM
chasemp raised the priority of this task from Normal to High.

I've seen a few people not understanding [Report Only] Refused to connect to blah, thinking it is an error. I can only point to http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-tech/20181206.txt at 09:52 and https://phabricator.wikimedia.org/T208188#4795662 but I'm sure there's been more.

Please do considering rephrasing. As you tell me to "Report only", where do you want me to report only this? Plus the output says "Refused to connect". Hence people think it refused to connect. Because it says it refused to connect.

Good points @Aklapper. I am not sure if this wording is ours or default. I am making a note to discuss with Security-Team. One question, I have done some testing of triggering our CSP policy and I don't see this language surface in the UI. Where are people seeing this?

Where are people seeing this?

Console of the web browser's Developer Tools

I'm pretty sure this message is defined by the browser. For me in a German Firefox it is:

Content Security Policy: Die Einstellungen der Seite haben das Laden einer Ressource auf https://viaf.org/viaf/AutoSuggest?callback=jQuery33103339132210135509_1544170637358&query=nevanlinna%2C%20rolf&_=1544170637359 festgestellt ("script-src"). Ein CSP-Bericht wird gesendet.

which is pretty clear about that no blocking happened, and only a report was sent. So I assume it's Chrome's fault to print misleading warnings in the console.

Since additionally an event is fired (https://developer.mozilla.org/en-US/docs/Web/API/SecurityPolicyViolationEvent), one could log another message to the console to clarify the warning and give additional hints.

Talked about this briefly today in the weekly for Security-Team and @Bawolff will respond

Change 481206 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@master] api: Support origin/path matching in ApiCSPReport for false positives

https://gerrit.wikimedia.org/r/481206

For toolforge/labs: [..] i think its important to be keeping track of any of the violations that arent obvious weird spyware junk as we are going to need to be communicating with those tool authors as we transition into enforce mode.

I'm writing this as a volunteer. The below will mostly summarise the #wikimedia-security IRC conversation between @sbassett and myself from last week (you were offline). I've raised my concern a few times, and I'll try my best to summarise once more over here and then leave the decision to you. I will respect whichever decision you make.

The problem I'm trying to solve is relating to privacy, not security. And (due to my refusing to accept this inspection) it means we're also talking about (some of) Wikipedia's ability to defend itself against vandalism (given I maintain RTRC and CVN gadgets etc). This is an area where the foundation has historically invested very little, and as a result the tooling around counter-vandalism is third-party developed and utilised by fairly small group of users. I hope one day (sooner rather than later) our platform will feature quality control as a first-class citizen with great user-experience, and users widely invited encouraged and enabled to participate in such workflows. However, that day is not today. Today, our content quality process is unsustainable - it depends on small group of overworked users.

Back on topic. As I understand it, there is no intention of blocking cross-domain API requests, only script requests. That is, we will allow user scripts and gadgets to fetch data from a third-party API (e.g. hosted in Cloud VPS or Toolforge; opt-in, on a per-origin/path basis). Yet, a report mode was put into production that is enabled equally for all cross-origin requests (both script and non-script). And this report mode tracks a ton of personal information.

Today, any user of a WMF wiki with at least one affected gadget, script or browser extension enabled is having all their page view traffic recorded in the fullest identifiable form imaginable, and send to Logstash:

  • User name and browser agent string in plain text.
  • Full url showing the wikis and articles they read.
  • Full url showing the actions they were performing (e.g. looking at someone else's contributions).
  • Time of day they are active and how much.
  • What they're doing on the page (e.g. the text they selected for Google Translate, or which CVN queries they run before considering to investigate or block a person), or which assistive software they might use at home.

This data is available to all 235 users with ldap groups "nda", "ops" or "wmf" (WMF engineering staff, contractors, and various members and volunteers of other organisations). This is offering a level of privacy invasion that even the restricted analytics-privatedata users wouldn't have access to in Hadoop.

Moreover, this data is presented in the same stream that reports software problems (Logstash' primary purpose). And is visible without looking for it. I think we could trivially omit a whole lot of information and be equally useful for security analysis. For example:

  • the blocked url could be limited to just the domain (that means who I'm investigating or what text I'm translating is no longer displayed in plain text).
  • the referer could be limited to just the wiki domain with maybe the wiki namespace name (that means what I'm reading or doing on-wiki is no longer tracked in full detail).
  • the user could be referenced by user ID instead (id-to-name mapping is public, but this way requires an intentional action to know who it is).

All to say, I don't want anyone to open Logstash looking for software problems and instead be shown that person X you might know is reading about Y, or perusing user Z's edits. Or that person X is using assistive tool Y in their browser at home, or was considering to do Z.

Anyway, I'm not here to argue about what we track. It won't change that I find it unacceptable and have as such proactively disabled all my affected tooling - that was two months ago. Naturally, I've been receiving a fair amount of complaining. I'd very much like to restore these tools as soon as possible. What can we do?

Az1568 added a subscriber: Az1568.Jan 25 2019, 1:14 AM

Any Update?

Change 481206 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@master] api: Support origin/path matching in ApiCSPReport for false positives

https://gerrit.wikimedia.org/r/481206

Change 504474 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[operations/mediawiki-config@master] Add entries to wgCSPFalsePositiveUrls

https://gerrit.wikimedia.org/r/504474

Change 504477 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@master] ApiCSPReport: Log user ID instead of name, and limit urls to origin

https://gerrit.wikimedia.org/r/504477

sbassett moved this task from Backlog to Done on the Security-Team board.Tue, Jun 11, 6:27 PM