What is the relationship between this and https://wikitech.wikimedia.org/wiki/Incident_documentation ? The line between "really bad performance" and "outage" is not always clear.

In T123753#2161839, @Mattflaschen wrote:

What is the relationship between this and https://wikitech.wikimedia.org/wiki/Incident_documentation ? The line between "really bad performance" and "outage" is not always clear.

Thanks for showing up for E152 and asking that question! My abstract summary of our discussion: Security-Team and Performance-Team should make recommendations for what should have a retrospective, and then the parties responsible for the root cause would hopefully often do them. The log of our discussion is saved here at 21:46 (line 99):

In T123753#2161839, @Mattflaschen wrote:

What is the relationship between this and https://wikitech.wikimedia.org/wiki/Incident_documentation ? The line between "really bad performance" and "outage" is not always clear.

For code security issues, the two are pretty distinct. I would like to see retrospectives of any UBN or High Priority issues (https://www.mediawiki.org/wiki/Wikimedia_Security_Team/Prioritization_of_bugs) that affect a project/code that a WMF team is responsible for. If there vulnerability was found without it being exploited, there will not be an incident documented on wikitech.

For operational security issues, the documentation on Incident_documentation (or similar page on office wiki) is probably enough.

In T123753#2346080, @RobLa-WMF wrote:

I'd like to institute this as a practice for security issues by the end of FY16-17q1. There's potentially some management overhead associated with bootstrapping this process, which I'll take on (assuming @dpatrick agrees and wants the help).

I agree with instituting this by the end of FY16-17 Q1. And I concur with @csteipp's comment from April 4th regarding UBN and High Priority issues. We should discuss if/how the output resulting from security-centric retrospective differs from what's posted at https://wikitech.wikimedia.org/wiki/Incident_documentation currently. I would like to say that I really like what's already there and I'm looking forward to this process. And ideally, we could keep security retrospectives in the same location.

Wikimedia ArchCom can sign off on a revocation for technical or social reasons

I smell recursion in two different ways.

One anti-pattern we should avoid: dumping this on WMF specialist teams.

This is the second way.

In T123753#2346080, @RobLa-WMF wrote:

I'd like to institute this as a practice for security issues

I do not have the time currently to word a sufficient response to this. However I'd like your help with Incident_documentation and their long term mitigation for things with user impact.

We should discuss if/how the output resulting from security-centric retrospective differs from what's posted at https://wikitech.wikimedia.org/wiki/Incident_documentation currently.

Yes that is probably still not sufficiently documented. Also we might need a discussion about responsible .

And ideally, we could keep security retrospectives in the same location.

I agree.

reinforcing appropriate incentives

I honestly don't see how the ArchComm can be the responsible body for defining our security, performance or other operational processes. Calling retrospective reports of code deployments, "architecture" is a stretch by any definition of those two terms, IMO.

In T123753#2353602, @faidon wrote:

I honestly don't see how the ArchComm can be the responsible body for defining our security, performance or other operational processes.

See https://www.mediawiki.org/wiki/+2#Revocation .

In T123753#2354393, @RobLa-WMF wrote:

In T123753#2353602, @faidon wrote:

I honestly don't see how the ArchComm can be the responsible body for defining our security, performance or other operational processes.

See https://www.mediawiki.org/wiki/+2#Revocation .

Yikes, let's not go there. I think a better way to frame it is to ask whether our community stands to benefit from having ArchComm deliberate over these processes and issue recommendations, and I think the answer to that is "yes". The question of whether ArchComm can back these recommendations with coercive authority is both unpleasant and unimportant. We should simply avoid any situation that would require settling that.

I definitely do not agree that this discussion is in ArchComm's scope and that it thus has any authority on the matter. This policy/process-making power does not stem from anywhere or have any legitimacy — the committee's name (architecture) is pretty unambiguous and clear evidence of that.

The ability to formulate our operational processes seems fairly orthogonal to the right to revoke +2 rights — which, by the way, has always been awkwardly placed with the ArchComm, and it's a pretty much unenforceable/a meaningless power these days, IMHO. Even that page itself mentions on its very beginning "[t]his page documents a MediaWiki development policy, crafted over time by developer consensus (or sometimes by proclamation from a lead developer)". (emphasis mine).

All that said, like @ori, I think opening up a discussion about setting up our processes for postmortems to security and performance issues would be useful and fruitful. In fact, it might make sense to discuss improvements to our existing TechOps/RelEng-driven postmortems as part of that same discussion — they are conceptually similar and the lines between these three sources of issues aren't always very clear, anyway.

I'd be personally be happy to participate in such an informal consensus-driven group between all the relevant stakeholders. The alternative of letting the ArchComm facilitate, deliberate and rubber-stamp such processes does not make any sense to me, both conceptually (not its mandate) and in practice (not the right people to decide).

In T123753#2354790, @ori wrote:

In T123753#2354393, @RobLa-WMF wrote:

In T123753#2353602, @faidon wrote:

I honestly don't see how the ArchComm can be the responsible body for defining our security, performance or other operational processes.

See https://www.mediawiki.org/wiki/+2#Revocation .

Yikes, let's not go there. [...] The question of whether ArchComm can back these recommendations with coercive authority is both unpleasant and unimportant. We should simply avoid any situation that would require settling that.

That's fair. The initial comment got under my skin, and I led with that in my haste to respond. My apologies for the haste.

Furthermore, I agree with you that getting into a spat here about ArchCom's authority (or non-authority) is unpleasant and hopefully isn't necessary. I would like to discuss this based on whether it's a good idea, and not veer into an authority debate.

I think a better way to frame it is to ask whether our community stands to benefit from having ArchComm deliberate over these processes and issue recommendations, and I think the answer to that is "yes".

Well put; thank you! I believe that ArchCom has a responsibility to lead with respect to software development norms. While both Security and Performance have expressed support for this, @dpatrick and @Bawolff particularly will need a lot of software developers to step up if they are going to keep up. This seems like a good tool to help all of us learn how to secure our software better, and I appreciate @dpatrick's willingness to work together on this in the coming quarter.

After doing some thinking about the comments above and discussing this with folks, I've removed the following section:

TechCom's role would be as an enforcement and appeal board should committers in certain areas of the code demonstrate a longstanding lack of discipline, per the mw:+2 policy (as of 2016-03-31):

Anyone can propose a revocation discussion, the Wikimedia ArchCom can sign off on a revocation for technical or social reasons, and anyone authorized by Wikimedia Foundation's Board of Trustees (e.g. WMF's Director of Technical Operations) can sign off on a revocation for emergency security matters or obvious policy breaches.

"revocation for technical or social reasons" is basically another way of saying "this committer isn't trusted anymore". Why that might be:

[Commit privileges are] a big deal. Your merge could cause Wikipedia or other sites to fail. It could create a security vulnerability that allows attackers to delete or corrupt data, or to gain access to private information. And in the more common case, it could cause technical debt to increase if the code doesn't have tests, is poorly implemented or poorly socialized. You're therefore required to read this entire document and carefully review all the relevant links in it before using +2.

Still in my task backlog: having a corresponding wiki page to flesh out this proposal.

From a MediaWiki perspective, operations/puppet seems to be the opposite of the direction we should be heading in. It won't improve code review, it will make some privileged people (yes, including me) able to bypass it, and it will work against us getting code from people who are not already privileged accepted.

(That said the general retrospectives idea in this task seems like a good idea.)

In T123753#2373240, @Krenair wrote:

From a MediaWiki perspective, operations/puppet seems to be the opposite of the direction we should be heading in. It won't improve code review, it will make some privileged people (yes, including me) able to bypass it, and it will work against us getting code from people who are not already privileged accepted.

I've replied over at T114419#2373286 (to keep this task from moving offtopic)

@greg, thanks for pinging this task in T141287! The process you outline in T140207#2493211 looks really solid. Though nitpicking is possible, it seems a solid base to iteratively improve.

If we implement the proposal described at T140207#2493211, should we have a "Needs retrospective" column on the #Wikimedia-Incident workboard?

In T123753#2493307, @RobLa-WMF wrote:

@greg, thanks for pinging this task in T141287! The process you outline in T140207#2493211 looks really solid. Though nitpicking is possible, it seems a solid base to iteratively improve.

Thanks for the feedback! I'm still in the post-epiphany high phase :)

If we implement the proposal described at T140207#2493211, should we have a "Needs retrospective" column on the #Wikimedia-Incident workboard?

Good question. I'm I'm understanding the intent of "needs retrospective" correctly ("to have a meeting with the relevant people to review what worked, what didn't, what's still confusing, what's next") I'm thinking there should be a "Needs follow-up" column after "Active Emergency". This indicates two things:

1. Someone (me-ish) with the right permissions needs to create the milestone for the incident to collect follow-up tasks, and
2. Someone (the first responders to the incident or someone they delegate to) needs to start working on the incident report

"Needs follow-up" is not a column a task should be in for very long (just until the milestone is created, mostly). It (the column) would be just to indicate that the immediate issue is fixed.

If it needs a retrospective in addition to the incident report then that should probably be a task (eg: "Perform retro for 20160725-Whatever incident") in the milestone for that incident. That task would be informed/follow the process that comes out of this task, is my guess.

In T123753#2493360, @greg wrote:

In T123753#2493307, @RobLa-WMF wrote:

If we implement the proposal described at T140207#2493211, should we have a "Needs retrospective" column on the #Wikimedia-Incident workboard?

Good question. I'm I'm understanding the intent of "needs retrospective" correctly ("to have a meeting with the relevant people to review what worked, what didn't, what's still confusing, what's next")

Thanks for checking. I realize now that the vocabulary is causing problems in this discussion. Let me try to offer definitions for each of these as I understand them:

• "Incident report" - this is the longstanding process that Ops has after downtime or other problematic event on the Wikimedia cluster
• "Retrospective" - a meeting often used by teams involved in some form of agile software development, typically held at the end of a project, a sprint, or an iteration of some variety.
• "Postmortem" - frequently used by way of analogy, as of this writing, this word redirects to "autopsy" on enwiki (as does "Post-mortem"). However, enwiki:Post-mortem_(disambiguation) offers somewhat more helpful content, but I've usually heard "postmortem" used to describe "let's analyze what happened after something went very wrong" (like an "incident report") rather than generically "an after project assessment".
• "Hotwash" - a nicer analogy than postmortem, and a term that is also in pretty widespread use. Much of my approach to running retrospective meetings comes from someone who prefers to call them "hot washes".

I've been avoiding the postmortem analogy because ... ewww. ;-) I've also been guilty of using "retrospective" and "incident report" interchangeably.

Even more confusing, there's a matrix here:

So....back to your question, what I meant by "needs retrospective" was "needs a report". Whether a meeting is helpful for arriving at the report is up to the person writing the report.

I wikified T123753#2494495 at https://www.mediawiki.org/wiki/User:RobLa-WMF/Retrospectives . Feel free to reuse.