or at least on more wikis.
Expected consequences:
Based on mw.org, it seems very roughly like a wiki the size of mw.org gets about 30 hits/minute on average, with ocassional spikes to 150/minute
Cool! Cc'ing @herron and @fgiunchedi here for awareness and their input. Logstash may or may not be happy about the extra load, depending on how much that would be (esp. for big wikis) :)
Change 469527 had a related patch set uploaded (by Brian Wolff; owner: Brian Wolff):
[operations/mediawiki-config@master] Enable csp report only on outreachwiki
I'm going to do this on some small wikis, but won't do anything to big wikis until I talk to @herron or @fgiunchedi
Change 469527 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable csp report only on outreachwiki
Mentioned in SAL (#wikimedia-operations) [2018-10-24T22:36:19Z] <bawolff@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Deploy csp report-only to outreachwiki T207900 (duration: 00m 54s)
Mentioned in SAL (#wikimedia-operations) [2018-10-24T22:38:13Z] <bawolff@deploy1001> Synchronized wmf-config/CommonSettings.php: Deploy csp report-only to outreachwiki T207900 (duration: 00m 54s)
Change 469538 had a related patch set uploaded (by Brian Wolff; owner: Brian Wolff):
[operations/mediawiki-config@master] Enable CSP-report-only on small wikis
Change 469538 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable CSP-report-only on small wikis
Mentioned in SAL (#wikimedia-operations) [2018-10-24T23:08:33Z] <bawolff@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Deploy csp report-only to small.dblist wikis T207900 (duration: 00m 56s)
[Just for context, i did small wikis, but I'll wait until talking to logstash folks before doing big wikis]
Thanks! Is it possible to get a sense of how many additional logs will csp reports be when fully on ? Is it potentially one per request or gets deduplicated on some level? Also I'm assuming we're talking about a single channel csp-report-only ?
I'd say lets go ahead and do group0 + group1 and see where we're at. Also we have icinga alerts for logstash dropping packets now, if sth goes obviously sideways those alerts will fire.
Based on mw.org, it seems very roughly like a wiki the size of mw.org gets about 30 hits/minute on average, with ocassional spikes to 150/minute
mediawiki.org is ~0.05% of total traffic so that would be 1000 hit/s if enabled everywhere. IIRC that's ~25% of current action API traffic and roughly equal to current logstash traffic. So for enabling everywhere, maybe the analytics infrastructure would be a better match instead of API + logstash?
OTOH there is relatively little value for CSP reports of logged-out requests - security-wise they are not much of a surface, privacy-wise you probably get the same insights from logged-in requests as well. Limiting CSP headers to logged-in requests would reduce load by three magnitudes or so, making it pretty tame for the action API and logstash, and unlike random sampling it works well with Varnish.
(The assumptions here are that 1) load scales linearly with wiki size and 2) report frequency is the same for logged-in users and non-logged in users. Those are true if the reports come from random things not related to us, such as Chrome plugins injecting their own scripts into the page. They are probably not true if the reports mainly come from our gadgets. mediawiki.org gets about 200 pageviews per minute currently, so if 15% of that results in CSP reports, I think it's a reasonable assumption that they mostly are from anons and thus mostly aren't from gadgets.)
I'd say lets go ahead and do group0 + group1 and see where we're at. Also we have icinga alerts for logstash dropping packets now, if sth goes obviously sideways those alerts will fire.
Awesome.
As tgr said, further discussion suggested we should do this for logged in users only, as the primary group we care about and much less traffic.
I'd kind of like to deploy this in stages (for logged in users. I'd like to keep on for logged out on wikis where we have it set so right now):
enwikiquote->all wikiquote->group1->medium.dblist->arwiki-><other stuff>
Change 469800 had a related patch set uploaded (by Brian Wolff; owner: Brian Wolff):
[operations/mediawiki-config@master] Enable CSP-report-only for logged in/session having users on enwikiquote
Change 469800 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable CSP-report-only for logged in/session having users on enwikiquote
Mentioned in SAL (#wikimedia-operations) [2018-10-26T01:19:56Z] <bawolff@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T207900 - deploy CSP to people with session on enwikiquote (duration: 00m 55s)
Mentioned in SAL (#wikimedia-operations) [2018-10-26T01:21:18Z] <bawolff@deploy1001> Synchronized wmf-config/CommonSettings.php: T207900 - deploy CSP to people with session on enwikiquote (duration: 00m 54s)
Change 469815 had a related patch set uploaded (by Brian Wolff; owner: Brian Wolff):
[operations/mediawiki-config@master] Enable CSP report only for session users on group1 wikis.
Change 469815 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable CSP report only for session users on group1 wikis.
Mentioned in SAL (#wikimedia-operations) [2018-10-26T01:41:16Z] <bawolff@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T207900 (b74911f6201) enable csp users with session all group1 wikis (duration: 00m 55s)
Change 469816 had a related patch set uploaded (by Brian Wolff; owner: Brian Wolff):
[operations/mediawiki-config@master] Enable CSP report only with session on medium sized wikis.
Change 469816 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable CSP report only with session on medium sized wikis.
Mentioned in SAL (#wikimedia-operations) [2018-10-26T02:05:41Z] <bawolff@deploy1001> Synchronized wmf-config/InitialiseSettings.php: bd55034d122 - T207900 - enable CSP report only for users w/session on medium wikis (duration: 00m 55s)
Change 469817 had a related patch set uploaded (by Brian Wolff; owner: Brian Wolff):
[operations/mediawiki-config@master] Enable csp report only for users w/session on arwiki
Change 469817 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable csp report only for users w/session on arwiki
Mentioned in SAL (#wikimedia-operations) [2018-10-26T02:22:27Z] <bawolff@deploy1001> Synchronized wmf-config/InitialiseSettings.php: d743db261 - T207900 - enable CSP report only for users w/session arwiki (duration: 00m 54s)
Change 469819 had a related patch set uploaded (by Brian Wolff; owner: Brian Wolff):
[operations/mediawiki-config@master] Enable csp report only for users w/session on a bunch of big wikis
Change 469819 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable csp report only for users w/session on a bunch of big wikis
Mentioned in SAL (#wikimedia-operations) [2018-10-26T02:43:50Z] <bawolff@deploy1001> Synchronized wmf-config/InitialiseSettings.php: a8aa9d6aae - T207900 - enable CSP report only for users w/session fawiki, frwiki, svwiki, eswiki, ruwiki, zhwiki, dewiki (duration: 00m 56s)
Change 469820 had a related patch set uploaded (by Brian Wolff; owner: Brian Wolff):
[operations/mediawiki-config@master] Enable csp report only for users w/session on enwiki
Change 469820 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable csp report only for users w/session on enwiki
Mentioned in SAL (#wikimedia-operations) [2018-10-26T02:54:48Z] <bawolff@deploy1001> Synchronized wmf-config/InitialiseSettings.php: 745d0b61 - T207900 - enable CSP report only for users w/session enwiki (duration: 00m 53s)
Mentioned in SAL (#wikimedia-operations) [2018-10-26T03:01:06Z] <bawolff@deploy1001> Synchronized wmf-config/CommonSettings.php: T207900 - Add wikimedia.org (no subdomain) to allow list for math (duration: 00m 53s)
Change 469821 had a related patch set uploaded (by Brian Wolff; owner: Brian Wolff):
[operations/mediawiki-config@master] Enable csp report only for people with session everywhere.
Change 469821 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable csp report only for people with session everywhere.
Mentioned in SAL (#wikimedia-operations) [2018-10-26T03:10:24Z] <bawolff@deploy1001> Synchronized wmf-config/InitialiseSettings.php: 745d0b61 - T207900 - enable CSP report only for users w/session enwiki (duration: 00m 55s)
Mentioned in SAL (#wikimedia-operations) [2018-10-26T03:19:52Z] <bawolff@deploy1001> Synchronized wmf-config/InitialiseSettings.php: bc9b863e - T207900 - enable CSP report only for users w/session everywhere (duration: 00m 55s)
@Bawolff I have you as point person here from the relevant meeting so I'm going to go ahead and assign
In response to this being unexpectedly enabled on all wikis for non-script fetches, I felt obligated to disable most Toolforge-related features in scripts and gadget I maintain for the community. For example:
https://meta.wikimedia.org/w/index.php?title=User:Krinkle/Scripts/CVNSimpleOverlay_wiki.js&diff=prev&oldid=18523572
https://meta.wikimedia.org/w/index.php?title=User:Krinkle/Scripts/Intuition.js&diff=prev&oldid=18523589
Ok, so this is done for now. If we eventually enable this report only for anons, we can use a separate ticket.
I've seen a few people not understanding [Report Only] Refused to connect to blah, thinking it is an error. I can only point to http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-tech/20181206.txt at 09:52 and https://phabricator.wikimedia.org/T208188#4795662 but I'm sure there's been more.
Please do considering rephrasing. As you tell me to "Report only", where do you want me to report only this? Plus the output says "Refused to connect". Hence people think it refused to connect. Because it says it refused to connect.
Good points @Aklapper. I am not sure if this wording is ours or default. I am making a note to discuss with Security-Team. One question, I have done some testing of triggering our CSP policy and I don't see this language surface in the UI. Where are people seeing this?
I'm pretty sure this message is defined by the browser. For me in a German Firefox it is:
Content Security Policy: Die Einstellungen der Seite haben das Laden einer Ressource auf https://viaf.org/viaf/AutoSuggest?callback=jQuery33103339132210135509_1544170637358&query=nevanlinna%2C%20rolf&_=1544170637359 festgestellt ("script-src"). Ein CSP-Bericht wird gesendet.which is pretty clear about that no blocking happened, and only a report was sent. So I assume it's Chrome's fault to print misleading warnings in the console.
Since additionally an event is fired (https://developer.mozilla.org/en-US/docs/Web/API/SecurityPolicyViolationEvent), one could log another message to the console to clarify the warning and give additional hints.
Talked about this briefly today in the weekly for Security-Team and @Bawolff will respond
Change 481206 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@master] api: Support origin/path matching in ApiCSPReport for false positives
I'm writing this as a volunteer. The below will mostly summarise the #wikimedia-security IRC conversation between @sbassett and myself from last week (you were offline). I've raised my concern a few times, and I'll try my best to summarise once more over here and then leave the decision to you. I will respect whichever decision you make.
The problem I'm trying to solve is relating to privacy, not security. And (due to my refusing to accept this inspection) it means we're also talking about (some of) Wikipedia's ability to defend itself against vandalism (given I maintain RTRC and CVN gadgets etc). This is an area where the foundation has historically invested very little, and as a result the tooling around counter-vandalism is third-party developed and utilised by fairly small group of users. I hope one day (sooner rather than later) our platform will feature quality control as a first-class citizen with great user-experience, and users widely invited encouraged and enabled to participate in such workflows. However, that day is not today. Today, our content quality process is unsustainable - it depends on small group of overworked users.
Back on topic. As I understand it, there is no intention of blocking cross-domain API requests, only script requests. That is, we will allow user scripts and gadgets to fetch data from a third-party API (e.g. hosted in Cloud VPS or Toolforge; opt-in, on a per-origin/path basis). Yet, a report mode was put into production that is enabled equally for all cross-origin requests (both script and non-script). And this report mode tracks a ton of personal information.
Today, any user of a WMF wiki with at least one affected gadget, script or browser extension enabled is having all their page view traffic recorded in the fullest identifiable form imaginable, and send to Logstash:
This data is available to all 235 users with ldap groups "nda", "ops" or "wmf" (WMF engineering staff, contractors, and various members and volunteers of other organisations). This is offering a level of privacy invasion that even the restricted analytics-privatedata users wouldn't have access to in Hadoop.
Moreover, this data is presented in the same stream that reports software problems (Logstash' primary purpose). And is visible without looking for it. I think we could trivially omit a whole lot of information and be equally useful for security analysis. For example:
All to say, I don't want anyone to open Logstash looking for software problems and instead be shown that person X you might know is reading about Y, or perusing user Z's edits. Or that person X is using assistive tool Y in their browser at home, or was considering to do Z.
Anyway, I'm not here to argue about what we track. It won't change that I find it unacceptable and have as such proactively disabled all my affected tooling - that was two months ago. Naturally, I've been receiving a fair amount of complaining. I'd very much like to restore these tools as soon as possible. What can we do?
Change 481206 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@master] api: Support origin/path matching in ApiCSPReport for false positives
Change 504474 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[operations/mediawiki-config@master] Add entries to wgCSPFalsePositiveUrls
Change 504477 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@master] ApiCSPReport: Log user ID instead of name, and limit urls to origin
Change 504477 merged by jenkins-bot:
[mediawiki/core@master] ApiCSPReport: Log user ID instead of name, and limit urls to origin
Change 481206 merged by jenkins-bot:
[mediawiki/core@master] ApiCSPReport: Support origin/path matching for false positives
Change 504474 merged by jenkins-bot:
[operations/mediawiki-config@master] Add entries to wgCSPFalsePositiveUrls
I apologize, I appear to have missed this or just didn't respond at the time, I'm not sure. In any case, I appreciate this has been frustrating and that I failed to respond properly around this issue. Thank you for taking the time to submit patches related to making the logging be less privacy encroaching.
Progress on CSP did stop for quite a while and has generally been slow, but is going again. Code is finished on the user-opt out system, but is still pending code review. I believe that logging will be useful during the transition period in order to proactively reach out to script authors. Is the logging, as currently implemented with your changes, address your concerns? Or are you still unhappy with the situation?
Thanks,
Change 701094 had a related patch set uploaded (by Krinkle; author: Krinkle):
[operations/mediawiki-config@master] CommonSettings: Restore wgCSPFalsePositiveUrls for intuition.toolforge.org
Change 701094 merged by jenkins-bot:
[operations/mediawiki-config@master] CommonSettings: Restore wgCSPFalsePositiveUrls for intuition.toolforge.org