Page MenuHomePhabricator

Develop an easy way for wikis to create a blackout protest
Open, Needs TriagePublic

Description

With the blackout set up by itwiki in the last days, we learned that it could be handy to have a tool which would allow sysops to enable a serverside blackout, instead of a clientside one. For this purposal, we currently have Extension:Blackout. However, an extension like that isn't handy: it requires a patch to be planned for a SWAT window, and in an emergency situation this could take way too much time. Discussing about it, we came to the conclusion that it could be useful to have a dedicated page in MediaWiki namespace (e.g. Mediawiki:Blackout) containing a JSON configuration for enabling a serverside blackout without devs' help. A very basic configuration could be this one:

"enabled": false,
"displayedContent": "Mediawiki:page_with_the_text_to_display (like a transclusion)",
"whitelistedPages": [
    "page 1",
    "Help: page 2",
    "MediaWiki: page 3"
],
"whitelistedGroups": [
    "sysop"
]

I'd be glad to work on it, but I'm not sure if that needs some kind of approval, both from a technical and ethical POV (although sysops can already effectively break the site). Please note that this could also be implemented in Extension:Blackout, which at the current state needs a new skin for every blackout.

Event Timeline

Daimona created this task.Jul 5 2018, 5:12 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 5 2018, 5:12 PM

Yeah lets not, any disgruntled sysop (or hacked accounts) could activate it. In valid emergencies situations the ops teams can deploy outside the SWAT windows.

Krenair added a subscriber: Krenair.Jul 5 2018, 9:15 PM

You can work on it without needing anyone's permission but I don't think it would get deployed to prod per @Peachey88

@Krenair, @Peachey88 Of course I wasn't looking for a permission, but didn't want to work on something that can't be deployed :) I'm not so convinced though: admins can already block editing on the whole site with one line of titleblacklist, and much worse they can add to common.js evil code which will be executed for every user (I mean in the case of hacked accounts or similar). Maybe making sysops exempt by default would make it better?

The titleblacklist case probably bears looking at, but JS at least isn't going to be the case once T190015: Create separate user group for editing sitewide CSS/JavaScript that does not include administrators by default is resolved.

@Dinoguy1000 Premising that I didn't read the whole other task. If its contents are what the title and the initial message say, it would just shift the problem of a compromised account to a littler group, but it would still be there. Maybe for this one we may further restrict it to bureaucrats? BTW, maybe my previous message wasn't clear, but the last sentence should be read as "make it so that sysops will always be exempted, whatever the settings are".

Yes, compromised accounts are always going to be a potential issue, but there are potential techniques to mitigate that risk, such as requiring all accounts with these rights to have two-factor authentication enabled. And I don't think it would be appropriate for bureaucrats to be given this right either; my understanding of the bureaucrat group is that it was always intended purely for granting/revoking other userrights (normally per community consensus), so adding a technical responsibility to the group would be inappropriate.

@Dinoguy good point. Maybe the same group as JS editors, once it'll be implemented? Could possibly be fine if such a group would be for "technical users".

Nevertheless, we cannot depend on third parties to take such important decisions as when to start and end a blackout. It is crucial for us to listen to many opinions as possible in order to proceed even at the very last time. Obviously, over the past few days that could not have been possible if we had to open a task and coordinate with someone else. It is fine to add restrictions, but communities must have the certainty to be able to act freely. Once the extension will be ready, before it gets installed we can open a discussion on it.wiki to decide whether to give the right to sysops or to a new, more trusted group.

@Sakretsu Well, there are a few of compromised high-privileged accounts and as WMF take care about server security, WMF will be the one to decide who should be able to perform those changes on-wiki if somebody. Even Jimbo's accounts was compromised (verify) and Jimbo used to have full privileges everywhere (magic userrights right) and still have very sensitive privileges such as being able to view all oversighted revisions and definitely is enough trusted to be able to do blackout.

On the other side, access to the server cluster isn't verified by password but by SSH keys (locally, such keys are encrypted by a password, but this password doesn't mean anything if alone/without the key). SSH key is something that you cannot guess and the only one mechanism how to circumvent the authentication is to find a bug in the protocol itself (or, to get the SSH key a sysadmin uses and decrypt it by guessing a password they used for decryption). It is really impossible to use this mechanism to authenticate on-wiki users (not because of technical limitation but "logical" limitations; majority of people do not know what SSH keys are and how to generate one) and because of that, authentication of a sysadmin is more reliable than authentication of any on wiki user.

Also, it is questionable if blackouts should be performed without discussions with WMF. IMHO it can be considered as using Wikipedia for political campaign and such using should be IMHO communicated with WMF before it is done.

Deskana added a subscriber: Deskana.

The proposed tool seems like it would create more problems that it would solve. Input from the experts in Security would be nice.

Since security problems are a big deal with this, here's a different approach: keep the blackout config as I proposed in the task description (+ auto exempt sysops), but with a difference: from such page, the blackout can only be disabled, and NOT enabled. Then, after refactoring Extension:Blackout where necessary, deploy it to all WMF wikis, so that if it needs to be activated the process will be much shorter. In such a case, people will still need a deployment to activate it, while being able to manage other things on their own. IMHO, in this case, the destructive potential is almost zero.

This sounds like possible solution.

Maybe the text, image etc. shown on the blackout page can be a system messages (so defining the purpose of blackout can be done by the community), a deployer can then just run a maintenance script enabling the blackout. Any sysop can then click a button "disable blackout". Yup, that sounds nice.

The idea is to directly transclude the whole page, whatever it is. BTW, the exact timeline I'm thinking of would be this one:

  1. The community, with the approval of WMF, decides to blackout
  2. A patch is created and SWATted as soon as possible. We should adapt the $wgBlackout variable to be used for these cases, so that the patch would only consist of $wgBlackout['Enable'] = true.
  3. The blackout is now enabled, and sysops can change config (landing page, whitelisted groups, almost whatever we want to add)
  4. As the community decides that the blackout should end, the on-wiki config is edited adding "enabled": false, and another patch for serverside deletion is planned for SWAT. This patch is deployed, and the blackout officially ends.

So, the idea is to create an approval period, and during that period give sysadmins the ability to manage blackout. Am I right?,

Yes, more or less. The period may be relatively short, and may also happen without explicit bilateral communication between communities and WMF (like in this case). In any case, the blackout would be enabled by a sysadmin.

What emergency situation(s) would require a blackout of a whole wiki?

@TerraCodes The ones we've seen for the recent blackout, as I was saying. Itwiki had more ore less three days to decide for a blackout, and there has been some uncertainties until one hour before the effective blackout. Other wikis have had even less time, and in order not to be late they opened short polls (we're talking of a matter of hours, not days), for instance cawiki. Some wikis actually ended up with nothing done because of not enough time left to vote, for instance elwiki. In such a situation, where time is really crucial and should be spent in reaching consensus, communities should have a quick way to enforce their decision. At any rate, this is the classical situation that you can't actually foresee, and for which we need to be ready, no matter how rare such situation could be.

TerraCodes added a comment.EditedJul 7 2018, 8:42 AM

@TerraCodes The ones we've seen for the recent blackout, as I was saying. Itwiki had more ore less three days to decide for a blackout, and there has been some uncertainties until one hour before the effective blackout. Other wikis have had even less time, and in order not to be late they opened short polls (we're talking of a matter of hours, not days), for instance cawiki. Some wikis actually ended up with nothing done because of not enough time left to vote, for instance elwiki. In such a situation, where time is really crucial and should be spent in reaching consensus, communities should have a quick way to enforce their decision. At any rate, this is the classical situation that you can't actually foresee, and for which we need to be ready, no matter how rare such situation could be.

Ah, I was thinking it meant a different type of emergency. Thanks for explaining~

Change 444363 had a related patch set uploaded (by Daimona Eaytoy; owner: Daimona Eaytoy):
[mediawiki/extensions/Blackout@master] [WIP] Refactor: add on-wiki configuration

https://gerrit.wikimedia.org/r/444363

Daimona claimed this task.Aug 31 2018, 4:25 PM
Krinkle added a subscriber: Krinkle.EditedSep 1 2018, 3:08 AM

In response to T203236 and https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/Blackout/+/444363/12 (Patch Set 12)

Review and questions
  • Read the configuration from a wg-configuration variable, not a MediaWiki JSON page. Config changes are now easy to make and deploy via Git. Using a JSON configuration page is in my opinion not worth the risk and complexity. The enablement of a Blackout must involve developers and infrastructure owners every time and should not happen without notification at least a few days ahead of time. With daily deployments and lots of people around to deploy at other times if needed, I think that removes any benefit of it being on-wiki, and adds the benefit of automated linting and testing, last-minute verification via a debug server, and technical code review.
  • Where does the idea of a server-side redirect come from? The previous blackouts at WMF were performed using JavaScript gadgets. I agree that we should change this. But I think we need very extensive testing and research on the next method, because the most recent method caused problems (T199252). What research led to the choice of an HTTP redirect?
    • Why was HTTP 50x chosen?
    • Is it valid for an HTTP 50x response to be a redirect?
    • Is it really "good for crawlers"? – citation needed :)
    • The use of HTTP 50x requires approval from WMF Traffic operators. I believe the current WMF Traffic configuration (Varnish) would cause an overload and crash the MediaWiki servers, given that 50x responses are not cacheable. – leading to a different kind of "blackout"!
  • The current approach will not work correctly because it depends on a PHP-generated response from MediaWiki servers, which means it only applies to logged-in users. For logged-out users, the protest would only be visible on pages have been edited at least once after the protest started – which can't happen given the proposed patch disables editing. For pages that are not edited after the protest starts (currently, all) the protest would only become around 4 to 7 days later when the cache is automatically purged (this used to be 30 days, but has recently been shorted to under 10 days).
Proposal for requirements

As mentioned, it requires research and testing to decide how the technical method should work. I do not have a proposal right now. But, here are some ideas for requirements and restrictions to help think in the right direction:

  • The switch must be controlled by a MediaWiki configuration variable.
  • When the switch is enabled, the protest must automatically be visible on all page views within 5 minutes.
  • When the switch is disabled, the protest must automatically disappear from all page views within 5 minutes.
  • The HTTP responses from MediaWiki during the blackout must be equally cacheable as normal. – Otherwise there will be an overload/crash.
  • Access to content for data consumers must be unchanged (e.g. queries to api.php, views via Parsoid, information for Google and Apple etc.)

At first, I can think of only two ideas that meet the above requirements:

  1. use JavaScript to redirect to a protest page when the article is loaded.
  2. use CSS or JavaScript to show a full-screen protest banner on top of the article.

Number 1 is what it.wikipedia.org did last month. Number 2 is en.wikipedia.org did in 2012 for the SOPA blackout.

From a user perspective, both of these two ideas are similar to what the Blackout extension does in PHP. So what is the difference? For a test website on the local machine without real users, they are almost identical. For Wikipedia scale, the difference is very big, and depends on a small technical detail.

The Blackout extension changes the HTML that is associated with url /wiki/Page_name. This means when users go to /wiki/Page_name they have to be connected with MediaWiki PHP code to see the change. For Wikipedia, this is not the case normally because the HTML is cached.

Every second, there are around 150,000 requests incoming to WMF wikis (that is 8 million per minute). Only around 50 requests per second go to MediaWiki. The others are cache matches (or "cache hits").

The it.wikipedia.org method changed MediaWiki:Common.js. Editing common-js does not change the HTML of URL /wiki/Some_topic. All wiki pages, every today, load a script from /w/load.php?modules=site. And that response of that URL is changed when editing common-js.

The en.wikipedia.org method enabled a CentralNotice banner. Doing that, does not change the HTML of URL`/wiki/Some_topic`. All wiki pages, every today, load a script from meta.wikimedia.org/w/load.php to ask for a banner and display it. That is only what changed, not the article view.

For both of these methods, when the browser loads the page, something happens in JavaScript that changes the page. But the original page was not changed. The Blackout extension, technically, changes the original page. That is a problem.

Krinkle renamed this task from Provide administrators a tool to enable a serverside blackout on a wiki to Design an easy way for wikis to create a blackout protest.Sep 1 2018, 3:09 AM
Krinkle added a project: Performance-Team.
Krinkle renamed this task from Design an easy way for wikis to create a blackout protest to Develop an easy way for wikis to create a blackout protest.Sep 1 2018, 3:13 AM
Bawolff added a subscriber: Bawolff.Sep 1 2018, 3:57 AM

I think we should think carefully around the social & technical requirements of blackouts first, and then figure out what sort of solution we want once we have enumerated the requirements. Given the large number of stakeholders this is probably something that should go through the rfc process.

@Krinkle thanks for the deep review; here are my thoughts/replies:

  • Reading from a wg variable is what this extension used to do. The idea is to let privileged wiki user refine the blackout settings on their own, with no need to bother people for deployment; although as you said it could take few time to get the needed changes deployed, the JSON-way theoretically takes no time, and can be changed as much as needed on the fly. However, as previously discussed on this task, enabling the blackout still requires changing a wg variable, so it needs to be approved, and tested as necessary. Customizable stuff should not make any difference for the backend, and is made so that sysop group will always be whitelisted (as well as the JSON page) to avoid unexpected, full locks.
  • We definitely need to make some research about blackout methods, and avoiding things like T199252 is crucial; that's the reason why the fifth point in the parent task is in place. Also, having a server-side blackout tool doesn't mean that we'll always want to use it. Still, it could be used if a JS blackout will be considered to be too weak for a certain situations.
    • The 503 was choosed after hearing a couple of thoughts on itwiki after the blackout; it seems that such code shouldn't create too many problems with google crawler, although this needs to be verified. Actually, I didn't check if a 503 response can be a redirect. A better solution can probably come from Traffic as you said.
  • I also didn't know about the difference this makes for logged-in and logged-out users. I guess this is the bigger trouble here.

I agree with the points of your proposal, but I'd still like to have some things customizable on-wiki (not the blackout activation itself). In the patch above, these things would be: landing page, whitelisted user groups, whitelisted pages, and blackout deactivation. The latter is intended to be temporary and to fill the gap between the actual, planned end of the blackout and the change in the wg variable. Ideally, when planning to cease the blackout at a precise time this shouldn't be used, but still it's better to have it in place for any inconvenience.
Proposals 1. and 2. are the current way of blacking out, and surely need to be improved (again, T199252 is the example here). The difference here would be to have a server-side tool instead of a client-side one, with the main difference being that the former is not circumventable. Whether this would be good for a given blackout event or not is something that cannot be foreseen. For sure, already having such a tool could make the difference.

Peter added a subscriber: Peter.Sep 2 2018, 7:13 PM