This is a public placeholder task for the Oct 1. security issue related to user authentication. Investigation is ongoing, details will be shared later. So far we have not seen any indication that the issue would be intentional or widespread. Out of an abundance of caution, we have logged out all users (at 21:48 UTC).
Description
Description
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | None | T264370 User authentication security issue (Oct 1, 2020) | |||
| Restricted Task |
Event Timeline
There are a very large number of changes, so older changes are hidden. Show Older Changes
Comment Actions
Were people really logged out at 7:43 UTC? Judging from comments on en-wiki the log out was from 21:43 UTC
Comment Actions
I use 2fA for Wikimedia sites and logged in again successfully. Is recommended to change the password anyways?
Comment Actions
See https://lists.wikimedia.org/pipermail/wikitech-l/2020-October/093922.html which does not say that people should change passwords - thanks.
Comment Actions
Was the release mentioned rolled back before the logout was done?
I mean, if the problem happens when people log in, then to force *everyone* to re-login is to call for more instances of the problem to happen…
I see T264363, but it is quite too technical for me to follow.
This comment was removed by Urbanecm.
Comment Actions
Isn't this the third or fourth time everyone has been forcibly logged out in the past year? How is this acceptable? Why does this keep happening and who's taking responsibility to ensure that it stops happening?
Comment Actions
The undeniable truth of software engineering and computer security is that bugs happen. While techniques like automated testing and code review are used to try to prevent bugs from entering production, and deployment stages are used to minimize the impact of new bugs, those techniques are not capable of preventing every bug (and thus, every security issue) from hitting production servers. This is because all code is written by humans and humans are imperfect.
Trying to eliminate all security problems is a noble goal, but it is also a futile one. Instead, it is much more important to ensure that there is an effective response to those security problems once they are discovered. The forced logouts are an example of that plan in action. The only ways to prevent them would be to expect perfection from human developers, to stop developing and improving MediaWiki, or to stop adequately responding to security incidents. Those are all worse options.
We don't yet know what caused this specific incident. Only once that stage of the investigation is complete can we begin to discuss how similar issues might be prevented in the future. All indications at the moment, however, are that it is unrelated to the previous session caching issues in June.
Comment Actions
This task is the public facing task for the user authentication issue. It is blocking the train and is thus an unbreak now priority. I am marking it stalled pending resolution of the issue which is tracked internally in the private task T264369.
Comment Actions
These are very nice words, but none of them answer the questions I asked. When the "emergency door" is being used regularly, with widespread user-facing impact, it warrants a serious and thorough investigation into why we're repeatedly having issues that require such drastic preventative action.
I woke up on Friday morning to a text from a Wikipedia administrator who couldn't remember his password. Because he was on a school IP address that had been blocked, he was also offered no option to reset his password via e-mail. Wikimedia wikis already face significant challenges attracting and retaining contributors; forcibly logging all of them out is disruptive and a big deal.
We don't yet know what caused this specific incident. Only once that stage of the investigation is complete can we begin to discuss how similar issues might be prevented in the future. All indications at the moment, however, are that it is unrelated to the previous session caching issues in June.
Who specifically is leading this investigation you reference?
Comment Actions
Hi, the issue is related to user authentication which is an highly sensible matter and unfortunately can not be discussed publicly. Rest assured it is being investigated by multiple persons since that involves multiple layers of the stack.
For your other questions, they are more general and would be better asked on another venue (mailing list or another task). Thanks!
Comment Actions
Lowering priority as new logging is in place and this is no longer a blocker of 1.36.0-wmf.11—the latter has been re-deployed for all wikis.
Comment Actions
The task description states Investigation is ongoing, details will be shared later.
I see nothing in https://wikitech.wikimedia.org/wiki/Incident_documentation yet. If this issue has been sufficiently mitigated so that the relevant tasks can be marked resolved, now would be the time to begin publicly sharing information about the issue.
Comment Actions
Yes indeed that is usually the process we follow for security issues. The public task (this T264370) is usually just a placeholder when the real work and investigation is captured in a different private task (since it potentially hold personal information, might be a threat to the infrastructure or leak counter measures to the attacker(s)).
The same occurs for the incident documentation. Along the Phabricator private task we also open a private document which is nicer to format and such a document has been created and polished. Unfortunately for the same reason as the private task, we can not share it.
What I can say is that the actual root cause has not been identified. We went through a lot of history and could not find any other use case. As a result of the investigation we have enabled massive logging on the server side infrastructure which, if the issue occurs again, would give us ample details and should let us pinpoint the actual root cause of the issue.
We haven't found the cause, it was a one time occurrence and we have added massive logged on the infrastructure to help analysis in the future if it occurs again. There is unfortunately not much more to say.
Comment Actions
Just to (belatedly) close the loop here, the likely cause was found and fixed two months later. Details are in {T274514} (restricted task).
Comment Actions
Ping @Dsharpe / WMF-Legal for advice on this. T274514 likely cannot ever be made public since it's currently PermanentlyPrivate, and for good reason, but we may be able to add NDA'd or other trusted Phabricator users to the task.
Comment Actions
What about just sharing here a technical description of the issue, without private or personal information?
Comment Actions
I don't know what information from T274514 can be shared, but the two public patches associated with that task will probably answer most of your questions.