Page MenuHomePhabricator

[RFC] Use `npm install mediawiki-express` as basis for all-in-one install of MediaWiki+services
Closed, DeclinedPublic

Description

Architecturally it may be desirable to factor our codebase into multiple independent services with clear APIs, but small wikis would clearly like a "single server" installation with all of the services running under one roof, as it were. Some options previously proposed have involved VM containers that bundle PHP, Node, MediaWiki and all required services into a preconfigured full system image (T87774).

This summit topic proposes an alternative: tightly integrating PHP/HHVM with a persistent server process running under node.js. The central service bundles together multiple independent services, written in either PHP or JavaScript, and coordinates their configurations. Running a wiki-with-services can be done on a shared node.js host like Heroku.

This is not intended as a production configuration for large wikis -- in those cases having separate server farms for PHP, PHP services, and JavaScript services is best: that independence is indeed the reason why refactoring into services is desirable. But integrating the services into a single process allows for hassle-free configuration and maintenance of small wikis.

A proof-of-concept has been built. The node package php-embed embeds PHP 5.6.14 into a node.js (>= 2.4.0) process, with bidirectional property and method access between PHP and node. The package mediawiki-express uses this to embed MediaWiki into an express.js HTTP server. (Other HTTP server frameworks could equally well be used.) A hook in the LocalSettings.php allows you to configure the mediawiki instance in JavaScript.

A bit of further hacking would allow you to install mediawiki skins or extensions via NPM and to dispatch to Parsoid (running in the same process).

SUMMIT PLAN

  • Determine whether this technology (or something similar) might be an acceptable alternative for small sites which are currently using shared hosting. See T113210: How should Wikimedia software support non-Wikimedia deployments of its software? for related discussion.
  • Identify and address technical roadblocks to deploying modular single-server wikis (see below).
  • Discuss methods for deploying complex wikitext extensions. For example, the WikiHiero extension would ideally be distributed with (a) PHP code hooking mediawiki core, (b) client-side JavaScript extending Visual Editor, and (c) server-side JavaScript extending Parsoid. Can these be distributed as a single integrated bundle?

TECHNICAL CHALLENGES

  • Certain pieces of our code are hardwired to specific directories underneath mediawiki-core code. This complicates efforts to run mediawiki from a "clean tree", and to distribute piece of mediawiki separately. In particular:
    • It would be better if the vendor directory could (optionally) live outside the core mediawiki tree, so it could be distributed separately from the main codebase, and allow for alternative package structures.
    • Extensions and skins would benefit from allowing a "path-like" list of directories, rather than a single location underneath the core mediawiki tree. Extensions/skins could be distributed as separate packages, with a simple hook to add their locations to the search path.
  • @tstarling has suggested that when running in single-server mode, some internal APIs (for example, between mediawiki and Parsoid) would be better exposed as unix sockets on the filesystem, rather than as internet domain sockets bound to localhost. For one, this would be more "secure by default" and avoid inadvertent exposure of internal service APIs.
  • It would be best to define a standardized mechanism for "services" to declare themselves & be connected & configured. This may mean standard routes on a single-server install (/w and /wiki for core, /parsoid for parsoid, /thumb for the thumbnailer service, etc), or else standard locations for unix sockets.
  • Can we leverage some of the various efforts to bridge composer and npm (for example), so we don't end up with incompatible packaging?

Event Timeline

cscott created this task.Oct 2 2015, 3:21 AM
cscott updated the task description. (Show Details)
cscott raised the priority of this task from to Needs Triage.
cscott added a project: TechCom-RFC.
cscott added a subscriber: cscott.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 2 2015, 3:21 AM
Qgil added a subscriber: Qgil.Oct 3 2015, 8:56 PM

Congratulations! This is one of the 52 proposals that made it through the first deadline of the Wikimedia-Developer-Summit-2016 selection process. Please pay attention to the next one: > By 6 Nov 2015, all Summit proposals must have active discussions and a Summit plan documented in the description. Proposals not reaching this critical mass can continue at their own path out of the Summit.

Krinkle added a subscriber: Krinkle.Oct 7 2015, 8:42 PM

https://github.com/cscott/node-php-embed is a first step towards a Proof-of-Concept here.

tstarling moved this task from Inbox to Under discussion on the TechCom-RFC board.Oct 14 2015, 8:44 PM
Qgil added a comment.Oct 28 2015, 10:41 AM

This task is still missing clear Summit goals, topics that could be discussed here before, and active discussion. It would be good to sort out these problems before the next deadline on November 6.

cscott renamed this task from [RFC] `npm install mediawiki` to [RFC] Use `npm install mediawiki-express` as basis for all-in-one install of MediaWiki+services.Oct 28 2015, 10:06 PM
cscott updated the task description. (Show Details)
cscott updated the task description. (Show Details)Oct 28 2015, 11:01 PM
cscott updated the task description. (Show Details)Nov 5 2015, 9:23 PM
cscott added a subscriber: tstarling.
Restricted Application added a subscriber: StudiesWorld. · View Herald TranscriptNov 5 2015, 9:23 PM
cscott updated the task description. (Show Details)Nov 5 2015, 9:38 PM
cscott updated the task description. (Show Details)Nov 5 2015, 9:47 PM

November 6, and this proposal doesn't seem to have much traction, it is not on track. Unless there is a sudden change, I will leave the ultimate decision of pre-scheduling it for the Wikimedia-Developer-Summit-2016 to @RobLa-WMF and the Architecture Committee.

cscott added a comment.Nov 6 2015, 3:41 PM

Discussion has taken place on the wikitech mailing list: https://lists.wikimedia.org/pipermail/wikitech-l/2015-November/083876.html
Mention has also been made on the related (but not identical) topics, eg https://phabricator.wikimedia.org/T87774#1786827 and https://phabricator.wikimedia.org/T113210#1768899.

And there are 11 subscribers to the phab task.

Unfortunately, there's only been one person who said "it's a terrible idea (because npm sucks)". Lack of opposition (and the fact that I was hacking on the PoC and not stirring up discussion) means the phab task lacks comment count.

Perhaps some of the subscribers to this task can try a npm install mediawiki-express themselves and bulk up this task with a discussion of their experience?

cscott updated the task description. (Show Details)Nov 6 2015, 6:46 PM

This proposal came up in an office hour for T113210: How should Wikimedia software support non-Wikimedia deployments of its software? yesterday. One misconception discovered there was the idea that this is "adding" node to the software stack.

I think it's more useful to think of node-mediawiki-express as *replacing* apache with node as the http server. We're using node as the SAPI instead of mod_apache.

This allows us to move more of the "HTTP server configuration" code into a place where we can tackle it with package-management tools. And in particular this seems to be useful with unbundled services communicating over HTTP, which can be installed and configured inside node-as-the-http-server, with the services managed with the standard package manger.

So: one useful way to think of this proposal is as a way to better manage the HTTP server configuration in a multi-service environment.

Wikimedia Developer Summit 2016 ended two weeks ago. This task is still open. If the session in this task took place, please make sure 1) that the session Etherpad notes are linked from this task, 2) that followup tasks for any actions identified have been created and linked from this task, 3) to change the status of this task to "resolved". If this session did not take place, change the task status to "declined". If this task itself has become a well-defined action which is not finished yet, drag and drop this task into the "Work continues after Summit" column on the project workboard. Thank you for your help!

I'm intrigued and fascinated by this solution, but believe that containers are more promising as a general-purpose distribution solution.

There is already a docker-based system for MediaWiki and a handful of services at https://github.com/wikimedia/mediawiki-containers, and production is moving towards leveraging containers longer-term as well.

GWicke changed the task status from Open to Stalled.Mar 2 2016, 11:00 PM
GWicke triaged this task as Low priority.
GWicke moved this task from (unused) to Backlog on the TechCom-RFC board.
Qgil removed a subscriber: Qgil.Mar 3 2016, 9:53 AM
Krinkle removed a subscriber: Krinkle.Mar 8 2016, 10:02 PM
RobLa-WMF mentioned this in Unknown Object (Event).May 4 2016, 7:33 PM
daniel added a subscriber: daniel.EditedJan 9 2018, 6:49 PM

This RFC seems to be stalled. If there is currently no interest in driving this further, it should for now be removed from the RFC work board.
If there is interest in continuing the RFC process, please let us (TechCom) known who will be working on this RFC, and who commits to implementing it if approved, and in what time frame.

Removing from the TechCom-RFC workboard as appears to still be stalled.

Aklapper assigned this task to cscott.Apr 23 2018, 11:38 AM

@cscott: Can you please associate an active project to this task? Also see T114457#3887526 and T114457#2082145 about status. Thanks.

No reply hence randomly guessing. Feel free to correct.

CCicalese_WMF closed this task as Declined.May 8 2018, 1:59 AM
CCicalese_WMF added a subscriber: CCicalese_WMF.

The MediaWiki Platform Team discussed this and will not pursue this approach. However, if there's another team that would like to claim and reopen this task, they should feel free to do so.