While this task is not about a site outage, I think https://wikitech.wikimedia.org/wiki/Incident_documentation provides a decent template for this type of report. Broadly, the template is three sections: a timeline of events, a list of places where we failed, and a list of action items for the future to prevent repeat incidents. The list of action items is almost always a list of Phabricator tasks.

Hi everyone. We are still in the process of investigating. Once we have finished our investigation, we will make a public report/analysis.

Hello all. For now, it appears that the attacker has ceased attempting to compromise accounts. We are still investigating prior instances and are not yet ready to provide a report. Once we are ready, that report will likely be in a format similar to what @MZMcBride described above. We thank everyone who has worked to remediate the issues occurring over the past few days and we will update you as soon as we can.

"Maury Markowitz" was compromised on enwiki within the last hour or so, the drama continues. See https://en.wikipedia.org/wiki/Special:Contributions/Maury_Markowitz

Oh so it seems that we now have to presume every one is compromised?

In T150605#2798239, @Paladox wrote:

Oh so it seems that we now have to presume every one is compromised?

I would suggest administrators should assume their own account(s) have been compromised and take appropriate action, starting by changing passwords on their WMF user account and the associated e-mail account set in preferences.

Do not re-use a previously used password, regardless of whether it was used on their WMF account or elsewhere.

Two factor authentication should be enabled, at the very least for their WMF account and ideally for the linked e-mail account.

A committed identity of some description is very much worth considering so that your user account may be returned to you if it is compromised and locked.

Maybe we could add ip tracking support. So if someone is logged into your account else where it could show the ip and country and location. That should only be viewable by the account holder but it will allow us to know weather we have been compromised since the hackers may have logged into an account but not use it.

Tim published an update at https://lists.wikimedia.org/pipermail/wikitech-l/2016-November/087008.html .

We really need to make it mandatory now to change your passwords on Wikipedia otherwise some users wont change there password allowing the hackers to hack in.

According to http://archive.is/1AIMB they have access to all users accounts.

In T150605#2798854, @Paladox wrote:

According to http://archive.is/1AIMB they have access to all users accounts.

That's nice.

This task is about publishing an analysis of the OurMine hack.
For anything else (like potential measures), please stick to mailing lists or dedicated tasks.

In T150605#2798854, @Paladox wrote:

According to http://archive.is/1AIMB they have access to all users accounts.

Just because they are claiming something doesnt make it true.

At this time it appears they are using passwords retrieved from other sites on the internet. They have not broken into all accounts in general, and are only suceeding at about 1 out of every 10 accounts they try to comprimise. Furthermore it appears they are targeting specific accounts they believe they know the password for, not everybody.

Please ensure that you are using a unique password on your wiki account. At this time it would be a good idea to change your password. This is especially true if you have ever used the password on another website or you have used the same password for a very long time. If either of those criteria apply to you, change your password immediately. Even if they dont, probably a good idea to still change your password.

If your account had advanced rights of any sort please immediatelt enable two factor auth by going to Special:OATH

Nudge It's coming up to 2 weeks since the OurMine hack became public knowledge. Please at least issue an interim analysis, I'm sure there is a good understanding of what happened and how. In the absence of any official analysis, volunteers are working on assumptions right now, such as whether administrators using longer passwords is sufficient protection and whether BotPasswords is okay to use on bot accounts with sysop rights, rather than OAuth.

Interim analyses were already published at https://blog.wikimedia.org/2016/11/16/compromised-accounts-response/ and https://lists.wikimedia.org/pipermail/wikitech-l/2016-November/087008.html.

Thanks, I was unsure if those were the response. I do not see any moves to ensure the advice to administrators is enforced. I doubt there will be any new policies until the analysis itself.

We are working on a more complete interim analysis/post mortem. Please keep in mind that today is a holiday in the United States which might delay the report somewhat.

Thank you for your patience.

In T150605#2820785, @Fae wrote:

whether BotPasswords is okay to use on bot accounts with sysop rights, rather than OAuth.

Both of those generate unique, long, random secrets so they are entirely safe as far as the current attack is concerned.

I mentioned BotPasswords as my understanding of https://www.mediawiki.org/wiki/Manual:Pywikibot/BotPasswords, was that it's just a password like any other, hence only as secure as the conventional login. However OAuth uses access tokens which provides an additional level of security. If the WMF is recommending that sysop accounts use 2FA, then my presumption would be that BotPasswords should be avoided on bot accounts with sysop rights for the same reasons.

In T150605#2824301, @Fae wrote:

I mentioned BotPasswords as my understanding of https://www.mediawiki.org/wiki/Manual:Pywikibot/BotPasswords, was that it's just a password like any other, hence only as secure as the conventional login. However OAuth uses access tokens which provides an additional level of security. If the WMF is recommending that sysop accounts use 2FA, then my presumption would be that BotPasswords should be avoided on bot accounts with sysop rights for the same reasons.

You can't login with BotPasswords through the web interface; it's for API interactions only. So by default, against the casual attack it's stronger; they've got to interact via the API (using their own script, or some other prebuilt bot).

Plus, the generated bot passwords are over 30 characters long, whereas users can easily use short ones if they desire

Benefits of owner-only OAuth over bot passwords are roughly that 1) you can limit the grant to a single wiki, 2) the credentials are never submitted over the internet so they cannot be stolen by someone eavesdropping on your connection (WMF wikis force a HTTPS connections, use strict transport security and probably will use public key pinning in the future, which gives a well-behaved client extremely strong security guarantees against eavesdropping; modern browsers are well-behaved but bots not necessarily so). So there is some difference but it's not big. Pulling of a man-in-the-middle attack is pretty hard even against a bot which has been misconfigured not to validate HTTPS security certificates; chances are an attacker will try to guess weak or reused passwords (against which both OAuth and bot passwords offer perfect defense, since they use long random strings as credentials, and they cannot be reused elsewhere), or infect your machine with malware and steal your credentials (against which pretty much any kind of bot authentication scheme is vulnerable). Infecting your machine is much harder than guessing passwords, but still easier than pulling off a DNS poisoning attack which is necessary to set up a man-in-the-middle scenario.

So if the difficulty level of OAuth is ten stars, then bot passwords is nine and half, and the passwords compromised by OurMine are maybe two or three :) Bot passwords are not the thing admins should be worried about right now.

Nudge. Why has this taken over 10 weeks?

Likely people were busy with more important/urgent things (vacation, developer summits, all-hands, security reviews of real software)...

Looking at captcha figures in T152219 it would seem that they ended up being captcha rate limited..

The green peak is dead on the dates when they were doing their attempts...

Nudge - it has now been 1 year, 5 months, 21 days since this request for publication was raised. We are not asking MI6 about attempts on the Prime Minister's life, it should be possible to explain what happened and the basics of how it will be prevented in the future without creating a hacker's guide to breaking Wikimedia.

Wikimedia's strategy brags about being committed to transparency and openness, let's make that a meaningful reality please.

https://phabricator.wikimedia.org/phame/post/view/114/additional_details_on_ourmine/

Okay, let's consider the factual "envelope":

• The blog post effectively closes this task
• The Phabricator blog post of September 2018 appears to phone it in, as it says as little as possible and if anything has less detail than the 2016 Wikimedia blog post of November 2016
• Posting at the Phabricator blog means hardly anyone will read it, presumably because the November 2016 post was PR driven to close down the public perception of a problem, but everyone but a few die hards have forgotten about this by now
• It took two years to write this minimal blog post, yet that is not considered a problem by any Wikimedia management, at least nobody has said this delay was unacceptable, or abnormal, or that such delays would be avoided in future
• The 2018 post stated We will continue to monitor the projects and stop these attacks, and will be implementing additional security measures to prevent another similar incident. As far as anyone outside of the security team can tell, there are no other security measures that were taken in the past two years that would prevent a similar incident. The blog post does not even obliquely state that any action has been taken apart from some (unspecified) better reporting and some better guidelines to users, which is itself unspecific enough to be the advice given in the original 2016 blog post

It would be jolly nice if someone within the WMF recognized that the response here was not optimal and the management team should do better at communications and responding to public security incidents, or have I missed something?

The standard for our incident documentation is pretty well outlined in the numerous examples on https://wikitech.wikimedia.org/wiki/Incident_documentation. The linked blog post doesn't even come close to what I would expect. The blog post doesn't mention any of the activity around 2fa development that went on, or the work in progress to stop usage of common passwords, or the password policy RfC. Many of the proposed action items and security hardening measures are public (e.g. T150647), so there's no reason to not mention them. For those that are still private, we can at least mention the Phabricator ticket number to demonstrate that there are other things that are being planned/dealt with privately.

I think someone recently described that https://wikitech.wikimedia.org/wiki/Incident_documentation/20180711-kafka-eqiad is the gold standard for an incident report, and I would hope something similar to that would be produced for this.

FWIW I don't really find blame with the security team for taking so long to deal with the issue, I think it's more of an organizational failure that the security team is clearly understaffed, but no other teams stepped up (or were asked to) help with the incident response.

Our process around security incident response is evolving and something the security team is working to improve. We're not there yet but we are definatley making improvements.

For a variety of reasons we can't disclose our countermeasures. What we can talk about are process and policy changes that support our countermeasures. An example is the post addressing the dictionary attack earlier in the summer where we mention security awareness, password and policy changes. In terms of tone, style and content the post on the dictionary attack is typical of what to expect going forward.

Since 2016 and the timing original blog post a lot has changed and there has been some turn over in the security organization and other parts of the foundation. This was before my time as Director of Security so I added what I knew to be true and what I felt was appropriate.

@Jcross: As ytou removed Security-Team, does that mean that this task is de-facto declined? Or who else is supposed to potentially work on this task?

Hi @Aklapper - apologies for any confusion. In a team meeting where we were working on cleaning up our workboard, we made the (apparently erroneous) assumption that this ticket had already been addressed and was no longer being worked on. If you could please let me know what additional actions are needed on this ticket I would be happy to keep things moving forward.

@Jcross: The status of this task is "open" (see upper left corner) so I'm not sure how to assume that this task had already been addressed?

@Aklapper - as I said in my comment above, we were clearly mistaken. We are happy to perform any work that is needed.