Page MenuHomePhabricator

Publish an analysis of the OurMine hack
Closed, DeclinedPublic

Subscribers
Tokens
"Barnstar" token, awarded by Davey2010."Barnstar" token, awarded by Poyekhali."Barnstar" token, awarded by Checkingfax."Like" token, awarded by Fae."Like" token, awarded by TomT0m."Like" token, awarded by Thibaut120094.
Assigned To
Authored By
Fae, Nov 13 2016

Description

This is a request for a report of the analysis of the OurMine hack to be published. It is understood that a non-public investigation is necessary, but it also makes sense to be transparent about events and as quickly as possible. This will provide an 'official' public assurance of the steps being taken by the WMF to make the systems more secure. Volunteers have rapidly responded by promoting two-factor authentication, as well as working collegiately on guidance for volunteers. A report of the behind the scenes analysis would aid these efforts and ensure that if wider changes of passwords or the roll-out of 2FA to non-sysop accounts makes sense, that these can be discussed within the community in a positive way. It is likely that volunteer discussions will continue and this will be reported in the Signpost next week, so timing a report in the next few days would be helpful in ensuring factual reporting.

Background
On 11th November it was confirmed that a number of "high profile" accounts had their passwords hacked, including Jimmy Wales' account. Edits made were to promote the OurMine organisation. Less than a day later, two-factor authentication was made available for all accounts with sysop status on any of the Wikimedia projects, though the WMF made no official statement as to whether it was in direct response to the hack. Since then other accounts have been shown to be compromised, being blocked until the account holders could have their passwords reset. The OurMine blog has confirmed there was a brute force attack on Jimmy Wales' account, and they claim to have copied the enwiki database; it is unclear whether this includes information not available in the public version of enwiki.

Links

  1. http://whois.domaintools.com/ourmine.org
  2. https://en.wikipedia.org/wiki/User_talk:Jimbo_Wales#Compromised
  3. https://commons.wikimedia.org/wiki/Commons:Administrators%27_noticeboard#high-profile_accounts_.28not_on_Commons.29_were_hacked
  4. https://lists.wikimedia.org/pipermail/wikimedia-l/2016-November/085456.html
  5. https://lists.wikimedia.org/pipermail/wikitech-l/2016-November/087008.html

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Stryn added a subscriber: Stryn.Nov 13 2016, 6:01 PM

While this task is not about a site outage, I think https://wikitech.wikimedia.org/wiki/Incident_documentation provides a decent template for this type of report. Broadly, the template is three sections: a timeline of events, a list of places where we failed, and a list of action items for the future to prevent repeat incidents. The list of action items is almost always a list of Phabricator tasks.

JJMC89 added a subscriber: JJMC89.Nov 13 2016, 6:33 PM

Hi everyone. We are still in the process of investigating. Once we have finished our investigation, we will make a public report/analysis.

jrbs added a subscriber: jrbs.Nov 13 2016, 11:18 PM
Poyekhali added a subscriber: Poyekhali.
Elitre added a subscriber: Elitre.Nov 14 2016, 2:09 PM
debt added a subscriber: debt.Nov 14 2016, 5:03 PM

Hello all. For now, it appears that the attacker has ceased attempting to compromise accounts. We are still investigating prior instances and are not yet ready to provide a report. Once we are ready, that report will likely be in a format similar to what @MZMcBride described above. We thank everyone who has worked to remediate the issues occurring over the past few days and we will update you as soon as we can.

"Maury Markowitz" was compromised on enwiki within the last hour or so, the drama continues. See https://en.wikipedia.org/wiki/Special:Contributions/Maury_Markowitz

MER-C added a subscriber: MER-C.Nov 16 2016, 7:55 AM

Oh so it seems that we now have to presume every one is compromised?

Nick added a subscriber: Nick.Nov 16 2016, 10:47 AM

Oh so it seems that we now have to presume every one is compromised?

I would suggest administrators should assume their own account(s) have been compromised and take appropriate action, starting by changing passwords on their WMF user account and the associated e-mail account set in preferences.

Do not re-use a previously used password, regardless of whether it was used on their WMF account or elsewhere.

Two factor authentication should be enabled, at the very least for their WMF account and ideally for the linked e-mail account.

A committed identity of some description is very much worth considering so that your user account may be returned to you if it is compromised and locked.

Maybe we could add ip tracking support. So if someone is logged into your account else where it could show the ip and country and location. That should only be viewable by the account holder but it will allow us to know weather we have been compromised since the hackers may have logged into an account but not use it.

We really need to make it mandatory now to change your passwords on Wikipedia otherwise some users wont change there password allowing the hackers to hack in.

Samtar added a subscriber: Samtar.Nov 16 2016, 12:46 PM
K6ka added a subscriber: K6ka.Nov 16 2016, 12:51 PM

According to http://archive.is/1AIMB they have access to all users accounts.

Reedy added a subscriber: Reedy.Nov 16 2016, 1:40 PM

According to http://archive.is/1AIMB they have access to all users accounts.

That's nice.

This task is about publishing an analysis of the OurMine hack.
For anything else (like potential measures), please stick to mailing lists or dedicated tasks.

Bawolff added a comment.EditedNov 16 2016, 1:47 PM

According to http://archive.is/1AIMB they have access to all users accounts.

Just because they are claiming something doesnt make it true.

At this time it appears they are using passwords retrieved from other sites on the internet. They have not broken into all accounts in general, and are only suceeding at about 1 out of every 10 accounts they try to comprimise. Furthermore it appears they are targeting specific accounts they believe they know the password for, not everybody.

Please ensure that you are using a unique password on your wiki account. At this time it would be a good idea to change your password. This is especially true if you have ever used the password on another website or you have used the same password for a very long time. If either of those criteria apply to you, change your password immediately. Even if they dont, probably a good idea to still change your password.

If your account had advanced rights of any sort please immediatelt enable two factor auth by going to Special:OATH

Tgr added a subscriber: Tgr.Nov 16 2016, 8:55 PM
Wdwd added a subscriber: Wdwd.Nov 21 2016, 4:14 PM
dpatrick triaged this task as High priority.
dpatrick added a parent task: Restricted Task.
Fae added a comment.Nov 24 2016, 12:21 PM

Nudge It's coming up to 2 weeks since the OurMine hack became public knowledge. Please at least issue an interim analysis, I'm sure there is a good understanding of what happened and how. In the absence of any official analysis, volunteers are working on assumptions right now, such as whether administrators using longer passwords is sufficient protection and whether BotPasswords is okay to use on bot accounts with sysop rights, rather than OAuth.

Fae added a comment.Nov 24 2016, 3:06 PM

Thanks, I was unsure if those were the response. I do not see any moves to ensure the advice to administrators is enforced. I doubt there will be any new policies until the analysis itself.

We are working on a more complete interim analysis/post mortem. Please keep in mind that today is a holiday in the United States which might delay the report somewhat.

Thank you for your patience.

Tgr added a comment.Nov 24 2016, 9:14 PM

whether BotPasswords is okay to use on bot accounts with sysop rights, rather than OAuth.

Both of those generate unique, long, random secrets so they are entirely safe as far as the current attack is concerned.

Fae added a comment.Nov 26 2016, 11:58 AM

I mentioned BotPasswords as my understanding of https://www.mediawiki.org/wiki/Manual:Pywikibot/BotPasswords, was that it's just a password like any other, hence only as secure as the conventional login. However OAuth uses access tokens which provides an additional level of security. If the WMF is recommending that sysop accounts use 2FA, then my presumption would be that BotPasswords should be avoided on bot accounts with sysop rights for the same reasons.

I mentioned BotPasswords as my understanding of https://www.mediawiki.org/wiki/Manual:Pywikibot/BotPasswords, was that it's just a password like any other, hence only as secure as the conventional login. However OAuth uses access tokens which provides an additional level of security. If the WMF is recommending that sysop accounts use 2FA, then my presumption would be that BotPasswords should be avoided on bot accounts with sysop rights for the same reasons.

You can't login with BotPasswords through the web interface; it's for API interactions only. So by default, against the casual attack it's stronger; they've got to interact via the API (using their own script, or some other prebuilt bot).

Plus, the generated bot passwords are over 30 characters long, whereas users can easily use short ones if they desire

Tgr added a comment.Nov 26 2016, 10:49 PM

Benefits of owner-only OAuth over bot passwords are roughly that 1) you can limit the grant to a single wiki, 2) the credentials are never submitted over the internet so they cannot be stolen by someone eavesdropping on your connection (WMF wikis force a HTTPS connections, use strict transport security and probably will use public key pinning in the future, which gives a well-behaved client extremely strong security guarantees against eavesdropping; modern browsers are well-behaved but bots not necessarily so). So there is some difference but it's not big. Pulling of a man-in-the-middle attack is pretty hard even against a bot which has been misconfigured not to validate HTTPS security certificates; chances are an attacker will try to guess weak or reused passwords (against which both OAuth and bot passwords offer perfect defense, since they use long random strings as credentials, and they cannot be reused elsewhere), or infect your machine with malware and steal your credentials (against which pretty much any kind of bot authentication scheme is vulnerable). Infecting your machine is much harder than guessing passwords, but still easier than pulling off a DNS poisoning attack which is necessary to set up a man-in-the-middle scenario.

So if the difficulty level of OAuth is ten stars, then bot passwords is nine and half, and the passwords compromised by OurMine are maybe two or three :) Bot passwords are not the thing admins should be worried about right now.

Fae added a comment.Jan 30 2017, 2:36 PM

Nudge. Why has this taken over 10 weeks?

Likely people were busy with more important/urgent things (vacation, developer summits, all-hands, security reviews of real software)...

Reedy added a comment.Feb 9 2017, 8:44 PM

Looking at captcha figures in T152219 it would seem that they ended up being captcha rate limited..

The green peak is dead on the dates when they were doing their attempts...

Aklapper removed dpatrick as the assignee of this task.Feb 19 2018, 11:20 AM
TomT0m removed a subscriber: TomT0m.Feb 19 2018, 8:28 PM
Yann added a subscriber: Yann.May 4 2018, 12:10 PM
Fae added a comment.May 4 2018, 1:50 PM

Nudge - it has now been 1 year, 5 months, 21 days since this request for publication was raised. We are not asking MI6 about attempts on the Prime Minister's life, it should be possible to explain what happened and the basics of how it will be prevented in the future without creating a hacker's guide to breaking Wikimedia.

Wikimedia's strategy brags about being committed to transparency and openness, let's make that a meaningful reality please.

Huji added a subscriber: Huji.May 4 2018, 5:39 PM
Ed7789 added a subscriber: Ed7789.May 6 2018, 1:16 PM
Bawolff moved this task from Backlog to To Follow Up on the Security-Team board.Sep 4 2018, 4:28 PM
Fae added a comment.Sep 12 2018, 6:39 AM

Okay, let's consider the factual "envelope":

  • The blog post effectively closes this task
  • The Phabricator blog post of September 2018 appears to phone it in, as it says as little as possible and if anything has less detail than the 2016 Wikimedia blog post of November 2016
  • Posting at the Phabricator blog means hardly anyone will read it, presumably because the November 2016 post was PR driven to close down the public perception of a problem, but everyone but a few die hards have forgotten about this by now
  • It took two years to write this minimal blog post, yet that is not considered a problem by any Wikimedia management, at least nobody has said this delay was unacceptable, or abnormal, or that such delays would be avoided in future
  • The 2018 post stated We will continue to monitor the projects and stop these attacks, and will be implementing additional security measures to prevent another similar incident. As far as anyone outside of the security team can tell, there are no other security measures that were taken in the past two years that would prevent a similar incident. The blog post does not even obliquely state that any action has been taken apart from some (unspecified) better reporting and some better guidelines to users, which is itself unspecific enough to be the advice given in the original 2016 blog post

It would be jolly nice if someone within the WMF recognized that the response here was not optimal and the management team should do better at communications and responding to public security incidents, or have I missed something?

The standard for our incident documentation is pretty well outlined in the numerous examples on https://wikitech.wikimedia.org/wiki/Incident_documentation. The linked blog post doesn't even come close to what I would expect. The blog post doesn't mention any of the activity around 2fa development that went on, or the work in progress to stop usage of common passwords, or the password policy RfC. Many of the proposed action items and security hardening measures are public (e.g. T150647), so there's no reason to not mention them. For those that are still private, we can at least mention the Phabricator ticket number to demonstrate that there are other things that are being planned/dealt with privately.

I think someone recently described that https://wikitech.wikimedia.org/wiki/Incident_documentation/20180711-kafka-eqiad is the gold standard for an incident report, and I would hope something similar to that would be produced for this.

FWIW I don't really find blame with the security team for taking so long to deal with the issue, I think it's more of an organizational failure that the security team is clearly understaffed, but no other teams stepped up (or were asked to) help with the incident response.

Legoktm updated the task description. (Show Details)Sep 12 2018, 6:07 PM

Our process around security incident response is evolving and something the security team is working to improve. We're not there yet but we are definatley making improvements.

For a variety of reasons we can't disclose our countermeasures. What we can talk about are process and policy changes that support our countermeasures. An example is the post addressing the dictionary attack earlier in the summer where we mention security awareness, password and policy changes. In terms of tone, style and content the post on the dictionary attack is typical of what to expect going forward.

Since 2016 and the timing original blog post a lot has changed and there has been some turn over in the security organization and other parts of the foundation. This was before my time as Director of Security so I added what I knew to be true and what I felt was appropriate.

Jcross moved this task from To Follow Up to Waiting on the Security-Team board.Sep 23 2019, 4:12 PM

@Jcross: As ytou removed Security-Team, does that mean that this task is de-facto declined? Or who else is supposed to potentially work on this task?

Hi @Aklapper - apologies for any confusion. In a team meeting where we were working on cleaning up our workboard, we made the (apparently erroneous) assumption that this ticket had already been addressed and was no longer being worked on. If you could please let me know what additional actions are needed on this ticket I would be happy to keep things moving forward.

@Jcross: The status of this task is "open" (see upper left corner) so I'm not sure how to assume that this task had already been addressed?

@Aklapper - as I said in my comment above, we were clearly mistaken. We are happy to perform any work that is needed.

Jcross closed this task as Resolved.Sep 24 2019, 3:07 PM
Jcross claimed this task.
Jcross added a project: Security-Team.

Due to the explanation in T150605#4580720, the Security Team is resolving this task.

sbassett moved this task from Waiting to Done on the Security-Team board.Sep 24 2019, 3:30 PM
Elitre removed a subscriber: Elitre.Sep 24 2019, 3:58 PM
Aklapper changed the task status from Resolved to Declined.Sep 24 2019, 4:42 PM

Thanks! That sounds more like a declined than a resolved to me, hence I'm boldly changing the task status.