21:01:08 <wm-labs-meetbot`> Meeting started Wed Jun 1 21:01:08 2016 UTC and is due to finish in 60 minutes. The chair is robla. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:01:08 <wm-labs-meetbot`> The meeting name has been set to 'archcom_security_rfc_meeting_https___phabricator_wikimedia_org_e198'
21:01:09 <brion> just a note -- phab folks are actually pretty responsive to small feature reqs if we pay them. ;)
21:01:24 <brion> we may or may not have used up our existing contracting credits, check with qgil :)
21:01:36 <robla> #topic Please note: Channel is logged and publicly posted (DO NOT REMOVE THIS NOTE) | Logs: http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-office/
21:01:51 <robla> hi folks!
21:01:53 <Matthew_> brion: Noted. I may open a task later asking for exactly that.
21:01:57 <brion> +1
21:02:51 <jzerebecki> will we do T123753 first?
21:02:52 <stashbot> T123753: Establish retrospective reports for #security and #performance incidents - https://phabricator.wikimedia.org/T123753
21:03:14 <robla> jzerebecki: that'd be wonderful....
21:03:56 <robla> to be clear, we're not planning a consensus-based meeting on it, but Krinkle believes we can pull that off at next week's meeting
21:04:31 <jzerebecki> these days https://coreos.com/blog/security-brief-coreos-linux-alpha-remote-ssh-issue.html has gone around
21:04:45 <jzerebecki> a retrospective on a grave security bug
21:04:50 <robla> gwicke felt like the first couple of steps of this RFC are really clear, but believes subsequent steps deserve more discussion (gwicke, please correct me if I have that right)
21:05:03 * robla looks at jzerebecki's link
21:05:25 <jzerebecki> "The issue went undetected during pre-merge review. To avoid situations like this in the future, we are concentrating on development of more comprehensive automated testing. Our verification tests now perform a series of additional security checks,"
21:05:43 <jzerebecki> " We have also taken the opportunity to introduce stronger image validation during the system image build process, automatically flagging packages with reported security issues. We will also ensure that security-related changes are accompanied by appropriate tests."
21:06:17 <gwicke> the first steps of the CSP RFC are low consequence preparations / information gathering, which I think are pretty uncontroversial
21:06:40 <robla> jzerebecki: oops, I only just figured out you were talking about postmortems. Excellent, thank you! :-) I thought you were talking about the CSP one, and I suspect gwicke is commenting on that.
21:07:30 <jzerebecki> ah yes that CSP seems like a worthwhile thing on first look is pretty uncontroversial
21:07:31 <TimStarling> where should the reports go?
21:07:37 * robla gets his 6-digit numbers confused
21:07:56 <bawolff> TimStarling: The CSP violation reports?
21:08:16 <TimStarling> sorry, I am one RFC behind, the retrospective reports for security incidents
21:08:18 <robla> TimStarling: I'm not sure. I could be convinced of either wikitech.wikimedia.org or mediawiki.org
21:08:29 <bd808> TimStarling: I think that's a good question. I'm a bit concerned that the current logging pipeline may melt with them being processed by an action api endpoint.
21:08:47 * bd808 is on the wrng topic
21:08:58 <TimStarling> yeah, I'm sure it was a good comment for any RFC
21:08:58 * robla fails at chairing
21:09:11 <robla> #topic T123753
21:09:11 <stashbot> T123753: Establish retrospective reports for #security and #performance incidents - https://phabricator.wikimedia.org/T123753
21:09:34 <brion> :)
21:09:37 <bawolff> I actually have a response to that question, but I'll wait until we get to that rfc
21:09:45 <robla> (we'll spend no more than 10-15 minutes on this one, and then move to the CSP one)
21:09:53 <brion> ok do we need things like: where do the reports go ;), how long before they get made, etc
21:10:24 <robla> #action robla propose a location for where reports go
21:10:24 <Platonides> I think wikitech
21:10:25 <brion> and if a report falls behind, do we need a fallback path?
21:10:46 <Platonides> some would be suited for mediawiki too, but others will be wmf-specific
21:10:47 <brion> eg who gets poked until it gets done ;)
21:10:55 <brion> or who does the poking, alternately
21:11:11 <jzerebecki> I think the most controversial thing on security incidents or even incidents reports in general is how to ensure that the actionables are done, as in being funded.
21:11:11 <robla> brion: I think it's sort of a percentage score thing. Some reports may never get done, and that's ok
21:11:43 <bawolff> What sort of actionables do you have in mind?
21:11:47 <brion> jzerebecki: ah for 'next steps to prevent this crap from getting worse' vs just 'and here's what we did to fix it so far'?
21:12:06 <jzerebecki> brion: yes
21:12:09 <bawolff> There's a big difference between - introduce automated testing for this type of security issue, vs fix the XSS in particular
21:12:16 <bawolff> *this particular xss
21:12:19 <bawolff> or whatever the issue is
21:12:47 <robla> I think postmortems are still useful even if we don't have anyone slavishly enforcing "strict adherance" to the process
21:13:04 <gwicke> the thing I keep wondering about when I look at this RFC is how security and performance post-mortems should differ from regular outage / incident post-mortems
21:13:30 <robla> gwicke: they should probably be more same than different
21:13:33 <Scott_WUaS> (@jzerebecki and security-oriented Wikidatans - what planning is occurring in terms of MIT-informed bitcoin and blockchain and in all countries' main and official languages - and re code security ... as well as, to re-construe the word "security" a kind of financial security for WMF and Wikdiata, for example?)
21:14:05 <bawolff> what?
21:14:13 <gwicke> robla: would it make sense to rephrase it as a refinement on post-mortem policies in general?
21:14:43 <jzerebecki> bawolff: robla i agree that postmortems are useful anyway
21:14:45 <gwicke> what works well / what doesn't, proposed changes etc
21:14:53 <robla> I think we've really handled as much of this topic as we should. Let's take further discussion back to Phab on T123753, and discuss CSP
21:14:53 <stashbot> T123753: Establish retrospective reports for #security and #performance incidents - https://phabricator.wikimedia.org/T123753
21:15:05 * robla goes to find the CSP task num
21:15:11 <robla> T135963
21:15:12 <stashbot> T135963: Add support for Content-Security-Policy (CSP) headers in MediaWiki - https://phabricator.wikimedia.org/T135963
21:15:17 <robla> #topic T135963
21:15:17 <stashbot> T135963: Add support for Content-Security-Policy (CSP) headers in MediaWiki - https://phabricator.wikimedia.org/T135963
21:15:18 <Scott_WUaS> (@bawolff - Is there any planning with the WMF Foundation for possible engagement with MIT's Bitcoin and Blockchain - and re security?)
21:15:36 <robla> Scott_WUaS: probably not a great topic for this meeting
21:15:39 <SMalyshev> re CSP, is this supposed to be configured somehow in wiki settings?
21:16:06 <Scott_WUaS> (@robla - thanks)
21:16:15 <bawolff> To answer TimStarling's question, I imagine on initial deployment, we might only show the report-only header to say like 0.01% of users to prevent servers from melting wiht initial reports, until things start to become fixed
21:16:23 <SMalyshev> if so, how and who gets to define them (i.e. wiki admins, ops, etc.)?
21:16:26 <bawolff> SMalyshev: yes
21:17:04 <bawolff> SMalyshev: I imagine that all Wikimedia would basically have a consistent setting, where wikimedia sites are allowed and others aren't
21:17:14 <brion> presumably ops defines them, in consultation with the community as we progress through the steps outlined in the rfc
21:17:25 <gwicke> I would expect there to eventually be defaults that just workTM, but the first steps are partly about figuring out what those defaults might be; admins / ops shouldn't need to worry about configuring CSP headers manually
21:17:32 <Platonides> 0.01% of users → most of them will be given to users unable to understand/fix it
21:18:01 <SMalyshev> ok, got it - but given the variety of weird code - including local and per-user JS - I wonder how much breakage this can cause
21:18:09 <brion> but yes, i would not expect different settings per wiki or anything
21:18:11 <bawolff> Platonides: I mean in so-called "report only" mode, where the web browser sends a json thingy to the api, but otherwise does not inform the user of the issue (unless they look in js console)
21:18:23 <TimStarling> I suppose existing uses of inline scripts would be migrated to use the nonce feature?
21:18:30 <brion> SMalyshev: have you read the rfc? there's some specific issues outlined, and a recommendation to use reporting mode to gather more info
21:18:56 <bawolff> TimStarling: The existing inline scripts RL uses would be. Stuff extensions do would be changed to use standard RL functions
21:18:59 <brion> some gadgets and user js will break that uses offsite resources, and will need to be updated.
21:19:14 <bawolff> I've already implemented a change for CharInsert extension to make it work with this
21:19:18 <brion> \o/
21:19:29 <bawolff> Any inliney stuff that user scripts do will break
21:19:30 <MaxSem> I think that we should eventually forbid cross-wiki script loading at all, this would drastically shrink the proposed header
21:19:50 <gwicke> I would propose to defer the discussion on the later stages for now, as it would benefit from having more information gathered in the first stages
21:19:51 <MaxSem> this would require Gadgets 2.0
21:20:12 <DanielK_WMDE__> would it make sense to test this on logged in users first? they know better how to communicate issues
21:20:18 <DanielK_WMDE__> (and are used to suffering...)
21:20:36 <brion> DanielK_WMDE__: logged in users are most likely to use gadgets or scripts that would break
21:20:43 <brion> logged out users by definition have no gadgets or user scripts
21:20:45 <gwicke> to me, anything up to including CSP on upload (images, SVG etc) seems fairly low-risk
21:20:56 <brion> upload is super safe yeah
21:21:07 <DanielK_WMDE__> brion: there are default gadgets, but yea. that's kind of the point: test the cases that are most likely to break.
21:21:07 <brion> i have plans to make svg more flexible but i think this shouldn't break that
21:21:13 <TimStarling> mashup gadgets might not always be able to change the domain name they use for resources
21:21:22 <brion> DanielK_WMDE__: *nod* sensible
21:21:33 <brion> that reminds me --
21:21:36 <TimStarling> we would have to either break them or whitelist them
21:21:37 <SMalyshev> yeah makes sense for logged-in users...
21:21:55 <DanielK_WMDE__> TimStarling: or proxy them
21:22:00 <brion> is there a way to opt in to a break through the policy, perhaps isolated in iframes, under specific circumstances?
21:22:04 <robla> #topic Please note: Channel is logged and publicly posted (DO NOT REMOVE THIS NOTE) | Logs: http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-office/
21:22:10 <DanielK_WMDE__> not that i like that option. but it's an option
21:22:14 <TimStarling> I'm thinking of closed-source scripts that are only available in minified form, that have domain names for secondary resources hard-coded within them
21:22:18 <bawolff> DanielK_WMDE__: Ideally we put it in test mode first, so by the time its actually turned on in "real" mode, everything for anons (since they all have the same set of js) is fixed
21:22:24 <TimStarling> then you can't even proxy
21:22:29 <gwicke> stage 5 (enforce CSP on testwiki) looks like the first one that might be controversial
21:22:44 <TimStarling> unless you intercept document modifications or something
21:22:54 <bawolff> gwicke: I agree, that's when things start to become hairy
21:22:59 <brion> i've looked into DOM proxying it looks like hell ;)
21:23:13 <DanielK_WMDE__> TimStarling: ick. do we even want them to work?...
21:23:20 <bawolff> brion: Stuff in an iframe is mostly controlled by that documents CSP
21:23:33 <gwicke> does anyone have general concerns about stages 0 - 4?
21:23:35 <bawolff> except for the part of csp that limits what iframe's to use
21:23:59 <brion> bawolff: *nod* that'd be ideal for mashups that 'need' that sort of thing... but we don't want a generic attacker to inject iframes any more than we want them to inject tracker pixels do we?
21:24:08 <brion> so we'd need an opt-in of some kind?
21:24:36 <bawolff> brion: yep, but that's really the tail end of this proposal
21:24:41 <brion> i suppose we could tweak the iframe permissions in the header per-user after a confirmation checkin
21:24:49 <brion> yeah i'll ponder that and propose an amendmend for late stages
21:25:13 <gwicke> if there are no concerns about 0-4, then we could record that fact & start moving on those
21:25:28 <brion> (CSP also fixes the remaining hole in my crazy ideas for sandboxed JS widgets, so this is all very interesting to me!)
21:25:30 <brion> +1
21:26:06 * robla doesn't know how to communicate partial approval of RFCs
21:26:11 <gwicke> this is reporting & enforcing for upload
21:26:26 <DanielK_WMDE__> robla: "no objections raised"?
21:26:34 <bawolff> stage 4 is actually reporting on testwiki
21:27:03 <TimStarling> it would suck to have to include *.wikimedia.org, maybe better to just move all those wikis?
21:27:18 <TimStarling> find a new 2LD for them?
21:27:24 <gwicke> bawolff: yeah, I had a different associativity in mind ;)
21:27:52 <DanielK_WMDE__> TimStarling: give commons a new domain? uuuuhhhhhh....
21:27:53 <TimStarling> for the less important ones anyway, commons can probably stay where it is
21:28:15 <robla> #info <gwicke> if there are no concerns about 0-4, then we could record that fact & start moving on those (no objections raised in response)
21:28:23 <TimStarling> I'm searchcom, for example, which IIRC is the search committee for ED selection which chose Sue Gardner
21:28:26 <bawolff> TimStarling: That would actually improve security generally, there's ways to use an XSS on foo.wikimedia.org to attack bar.wikimedia.org
21:28:33 <qgil> brion, I just saw your ping. I don't know what is the problem with time conversion, but if it is related to Calendar, bug reports / feature requests are welcome (there are many filed already)
21:28:39 <TimStarling> *I'm thinking
21:29:03 <bawolff> maybe instead of moving wikis off *.wikimedia.org we could move everything else to *.wikimedia-etc.org or something
21:29:20 <brion> qgil: thx!
21:29:41 <TimStarling> don't forget there is *.wiki now
21:29:59 <DanielK_WMDE__> bawolff: like wikimedia-files.org, wikimedia-bits.org, etc?
21:30:10 <bawolff> Well bits is dead now (I think)
21:30:15 <bawolff> but yeah
21:30:21 <TimStarling> could have searchcom.wmf.wiki
21:30:28 <SMalyshev> there will be w.wiki for url shortener IIRC
21:30:29 <DanielK_WMDE__> TimStarling: do we have any .wiki domains?
21:30:35 <bawolff> e.g. ganglia.wikimedia.org being on not *.wikimedia.org
21:30:38 <Platonides> DanielK_WMDE__: better to use wikimedia.bit
21:30:48 <TimStarling> DanielK_WMDE__: we have some I think
21:30:49 <DanielK_WMDE__> heh
21:30:55 <mutante> DanielK_WMDE__: yes, we have each language code in .wiki
21:30:59 <mutante> until recently
21:31:29 <DanielK_WMDE__> mutante: but not pedia.wiki? shame...
21:31:31 <mutante> maybe they got dropped now and we just keep w.wiki, that is for sure for the url shortener
21:31:31 <SMalyshev> w.wiki actually works :)
21:31:32 <bawolff> (I have some more specific reasons why I think it'd be cool to move stuff off *.wikimedia.org, but they're written on secret https://phabricator.wikimedia.org/T113790#1676633 )
21:32:26 <mutante> SMalyshev: yep, but it needs the rules to becoma shortner
21:32:55 <bawolff> So umm, this is kind of getting off topic...
21:33:41 <bawolff> Can we move to discussing stages 4-6 of rfc?
21:33:49 <gwicke> bawolff: do you think it would make sense to go ahead with phases 0..4, and then revisit based on the information gathered in the process?
21:34:13 <bawolff> I think so, I'd like to know if anyone see's any major show stoppers though
21:34:56 <brion> bawolff: nothing scaring me beyond what's already mentioned
21:34:58 <robla> #info <bawolff> Can we move to discussing stages 4-6 of rfc? [...] I'd like to know if anyone see's any major show stoppers
21:35:10 <bawolff> Also, for having the support in MW, is there agreement enough that its a cool idea, to merge the CSP header generation code into mw, even if its not yet decided for sure how/when we'll enable
21:35:11 <brion> we'll have to work with folks on updating gadgets and scripts though
21:35:16 <bawolff> yes
21:35:21 <brion> and/or provide an opt-in compatibility mode that reduces the restrictions
21:35:39 <bawolff> I expect it will be a long haul for getting this fully enabled
21:35:49 <brion> i say yes definitely merge the code in
21:35:55 <brion> the better to test it in real-ish places
21:36:15 <gwicke> I'm also not too worried in principle; we'll have to work out the exact CSP headers, but I'm optimistic that we can find something that strikes a good balance between functionality for gadgets etc & security
21:36:46 <bawolff> Ideally I think I'd like to see this sort of thing happen wiki by wiki (Oh it appears that there are no more violation reports coming from frwiki, lets turn it on there, etc)
21:36:59 <gwicke> and that balance might change gradually over time, in response to alternatives becoming available etc
21:38:03 <robla> #info bawolff asks if we can merge CSP code into MediaWiki, even if it isn't enabled. no objection seemed to be given
21:38:46 <SMalyshev> CSP code meaning generating code or reporting code or both? just curious
21:38:53 * bawolff meant both
21:39:07 <SMalyshev> ah, ok.
21:39:23 <SMalyshev> RFC says some CSP will be in Varnish, so I wasn't sure
21:39:44 <gwicke> a reporting end point would be useful for RB CSP reports as well; currently, we only enforce, but don't report
21:39:56 <bawolff> RB?
21:40:01 <robla> #info <SMalyshev> CSP code meaning generating code or reporting code or both? <bawolff> meant both
21:40:03 <gwicke> RESTBase / the REST API
21:40:50 <bawolff> ah, should have been able to guess that acronym
21:41:08 <gwicke> for example, all the SVG math images are served with CSP headers disallowing any kind of JS already
21:41:21 <gwicke> same with HTML
21:41:25 <brion> nice
21:41:26 <bawolff> RB can certainly use the endpoint if it wants to
21:43:21 <TimStarling> I will have some comments on the "Initial support for CSP" change in gerrit
21:43:57 <bawolff> \o/
21:44:35 <robla> bawolff: dapatrick : would you like to keep discussing this issue, or do you think there are other security issues we should discuss here?
21:44:49 <bawolff> dpatrick is not in channel afaik
21:44:58 <dapatrick> I'm in the channel.
21:45:01 <dapatrick> Lurking.
21:45:04 <robla> :-)
21:45:10 <bawolff> ah, I didn't see your username on the list
21:45:32 <dapatrick> I defer to bawollf.
21:45:41 * bawolff thinks the current topic is pretty much discussed out
21:45:41 <dapatrick> *bawolff
21:46:26 <robla> I'd be pretty happy if y'all set the agenda for the next few ArchCom meetings, to be honest :-)
21:46:56 <TimStarling> I assume we're not going to have CSP next week now
21:47:16 * robla gives dapatrick and bawolff license to be extra pushy in https://phabricator.wikimedia.org/Z425
21:47:21 <bawolff> On the topic of retrospectives we were discussing earlier, for teams who have a specific extension they are responsible for, I think it would be nice to clarify their responsibilities for doing security releases of that extension (assuming its a non-bundled extension)
21:48:29 <robla> #info <bawolff> On the topic of retrospectives we were discussing earlier, for teams who have a specific extension they are responsible for, I think it would be nice to clarify their responsibilities for doing security releases of that extension (assuming its a non-bundled extension)
21:48:56 <dapatrick> bawolff Can you give an example of such an extension?
21:48:56 <gwicke> often, the question is exactly about the "who is responsible" bit
21:49:14 <bawolff> I'm thinking specificly about recent issue in mobile frontend
21:49:29 <bawolff> where there was a lot of unclarity over who has responsibility, and what best practises are
21:51:33 <gwicke> at some point, we need to establish clearer minimum maintenance / ownership requirements for anything in production / semi-production
21:51:40 <robla> gwicke: are you proposing that the Security team should own services? ;-)
21:51:44 <gwicke> and enforce those
21:52:40 <jzerebecki> gwicke: suggestion for the point?
21:52:52 <gwicke> robla: the RFC is pointing out the problem of needing clear ownership, and *not* pushing it to some catch-all team like services or security
21:53:40 <bawolff> Anyways, in the event of a security incident in an extension that's used widely (but not bundled) and has a specific owner, I'd like to see the owner be responsible for doing official announcements of the security issue
21:53:46 <robla> gwicke: sure, I'm just ribbing you :-) I think the catch is figuring out ownership of the hundreds of things already deployed
21:54:29 <gwicke> robla: yup, we are familiar with that discussion (ex: OCG)
21:54:31 <SMalyshev> last committer owns it ;)
21:54:49 <dapatrick> bawolff I fully agree with the need to figure this out.
21:54:50 <robla> we should probably start winding up though, since we have potential to wind into another long discussion
21:55:06 <bawolff> gwicke: Maybe its difficult for the past, but it'd certainly be nice for that to be a requirement before deploying (although I was sort of under the impression that was already the case)