Page MenuHomePhabricator

Request creation of Wikispore VPS project
Closed, ResolvedPublic

Description

Project Name: Wikispore

Wikitech Usernames of requestors: Pharos, Tgr, Sannita, Sj

Purpose: A well-tended wiki farm for the exploration of new genres of free knowledge.

Brief description: Experimental project with use of namespaces for subprojects, with different thematic areas. Use of standard libraries and SUL would be positive, as would Wikibase/Wikidata integration, as far as is possible. We discussed this project with members of the Wikimedia Cloud Services team at Wikimania in Stockholm.

More here:
https://meta.wikimedia.org/wiki/Wikispore

How soon you are hoping this can be fulfilled: This month.

Event Timeline

Pharos created this task.Aug 23 2019, 11:26 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 23 2019, 11:26 AM
Pharos updated the task description. (Show Details)Aug 23 2019, 11:28 AM

what does standard libraries mean? SUL is for sure not directly possible, I was speculating about something like oauth against mediawiki.org prod.

Out of curiosity, would this instance be related to development of some technology / code?
Or would this instance host any content on wiki pages which is not wiki page content to test and "throw away"?

Tgr added a comment.Aug 23 2019, 5:50 PM

what does standard libraries mean?

Off the top of my head:

  • common wikitext syntax extentions: ParserFunctions, Cite, SyntaxHighlight, Math (probably in client-only mode to avoid having to set up mathoid)
  • Echo, Thanks, DismissableSitenotice, MobileFrontend, MediaViewer, Popups, ULS, CodeEditor, CodeMirror, WikiEditor for a decent user experience
  • maybe VisualEditor (nice for user experience but it means we'd have to set up Parsoid, maybe RESTBase, maybe Citoid; that tends to be a world of pain)
  • Interwiki, GlobalUserPage for nicer integration with the production cluster
  • maybe PageImages, TextExtracts for nicer sharing experience
  • maybe Gadgets for customization
  • PDFHandler, TimedMediaHandler, Graph (in client-only mode) if they seem relevant to the trialed projects
  • Scribunto, TemplateData, TemplateStyles, TemplateSandbox, TemplateWizard if we expect that some of the projects will need complex template magic

Maybe anti-abuse / security extensions (OATH, AbuseFilter, Nuke, CheckUser, LoginNotify etc) but we can probably do that when the need arises. We'll probably need ConfirmEdit at least to not get overrun by spambots. (Or just disable anonymous editing, that's probably easier.)

Ideally WikibaseClient, I have no idea how much it's possible to interface with Wikidata while not being on the same cluster.

Ideally CirrusSearch for a decent search experience. Not sure how hard it is to set up the ElasticSearch cluster it needs.

Now that I have written it down, it's a bit of a handful. Maybe we should cull it more.

SUL is for sure not directly possible, I was speculating about something like oauth against mediawiki.org prod.

Indeed, we'll want to approximate SUL with OAuthAuthentication (which is kind of unfinished, but not a ton of work).

bd808 added a subscriber: bd808.Aug 23 2019, 9:53 PM

Now that I have written it down, it's a bit of a handful. Maybe we should cull it more.

The MediaWiki and extensions bits should be relatively easy. You are right to think about how things add up though both in admin toil and in system complexity. You'll need Apache and MySQL too of course and probably Redis or Memcached. A small elasticsearch cluster should be doable using existing Puppet manifests. I think you will really want to use Puppet to manage the instances in your cluster both to reduce toil and to make it easier to scale things as the project grows. You actually might be able to get started using https://wikitech.wikimedia.org/wiki/Help:MediaWiki-Vagrant_in_Cloud_VPS and not worry about getting much fancier until you have enough wikis and usage that running it all on a single virtual machine becomes a performance issue. Building out fancy infrastructure before you find out the demand could distract you from the core problem of helping folks explore different contribution styles. (Says a person who has started many side projects and gotten lost in build and test tooling before proving that the idea was even worth continuing to code.)

Now that I have written it down, it's a bit of a handful. Maybe we should cull it more.

You actually might be able to get started using https://wikitech.wikimedia.org/wiki/Help:MediaWiki-Vagrant_in_Cloud_VPS and not worry about getting much fancier until you have enough wikis and usage that running it all on a single virtual machine becomes a performance issue. Building out fancy infrastructure before you find out the demand could distract you from the core problem of helping folks explore different contribution styles. (Says a person who has started many side projects and gotten lost in build and test tooling before proving that the idea was even worth continuing to code.)

My understanding is that MediaWiki-Vagrant may not be supported for much longer, but if it can be an easy interim solution, and it is possible to transfer things over afterward, then great. Maybe we could try it out for a couple of months there, and perhaps move onto another stage if warranted by November, when I'm hoping to share the concept at WikiConference North America.

bd808 added a comment.Tue, Aug 27, 3:50 PM

My understanding is that MediaWiki-Vagrant may not be supported for much longer, but if it can be an easy interim solution, and it is possible to transfer things over afterward, then great. Maybe we could try it out for a couple of months there, and perhaps move onto another stage if warranted by November, when I'm hoping to share the concept at WikiConference North America.

MediaWiki-Vagrant has literally never been officially supported by any Foundation or affiliate team. There have been some short term projects to add specific functionality in the past. Ori started it as a project to help his team in 2013. It has been expanded by a number of people including large contributions by myself, Dan Duvall, and Gergő. I am much less active in adding new features to it than I was when I was doing MediaWiki development work in my prior roles at the Foundation, but the project is far from "dead".

If you have time and energy to build your own MediaWiki full stack wiki farm deployment tooling that's great. If not, MediaWiki-Vagrant exists and works today and has well documented integration with Cloud VPS projects.

If you have time and energy to build your own MediaWiki full stack wiki farm deployment tooling that's great. If not, MediaWiki-Vagrant exists and works today and has well documented integration with Cloud VPS projects.

Sounds great to me, let's do it!

bd808 claimed this task.Tue, Aug 27, 4:41 PM

Approved in 2019-09-27 WMCS team meeting

Tgr added a comment.EditedTue, Aug 27, 7:33 PM

We'll have to forbid using Vagrant roles and come up with our own redacted versions of them because almost all are completely unsuitable for a live content wiki (they set up test users, test wikis etc).
That's not too hard though and I think baseline Vagrant is not too bad in that regard. And setting up CirrusSearch, Logstash or VisualEditor manually is not something I'd look forward to.

bd808 added a comment.Tue, Aug 27, 9:23 PM

Meza might be another option for managing the farm?

Tgr added a comment.Tue, Aug 27, 9:32 PM

Meza would be a good fit in theory, but as long as no one familiar with it is involved, doesn't seem too practical.

bd808 added a comment.Wed, Aug 28, 5:44 PM

I could not figure out @Sannita's Developer account. Maybe they do not have one yet? I have created the project and added @Tgr, @Sj, and @Pharos as admins. They can add @Sannita once someone figures out their Developer account name.

$ openstack --os-region-name eqiad1-r project create --enable --description "wiki farm for the exploration of new genres of free knowledge" wikispore
+-------------+---------------------------------------------------------------+
| Field       | Value                                                         |
+-------------+---------------------------------------------------------------+
| description | wiki farm for the exploration of new genres of free knowledge |
| domain_id   | default                                                       |
| enabled     | True                                                          |
| id          | wikispore                                                     |
| is_domain   | False                                                         |
| name        | wikispore                                                     |
| parent_id   | default                                                       |
+-------------+---------------------------------------------------------------+

$ for u in tgr sj pharos; do openstack role add --user $u --project wikispore projectadmin; openstack role add --user $u --project wikispore user; done
$ openstack role assignment list --project wikispore --names
+--------------+----------------------+-------+-------------------+--------+-----------+
| Role         | User                 | Group | Project           | Domain | Inherited |
+--------------+----------------------+-------+-------------------+--------+-----------+
| projectadmin | Novaadmin@Default    |       | wikispore@Default |        | False     |
| user         | Novaadmin@Default    |       | wikispore@Default |        | False     |
| observer     | Novaobserver@Default |       | wikispore@Default |        | False     |
| projectadmin | Pharos@Default       |       | wikispore@Default |        | False     |
| user         | Pharos@Default       |       | wikispore@Default |        | False     |
| projectadmin | Sj@Default           |       | wikispore@Default |        | False     |
| user         | Sj@Default           |       | wikispore@Default |        | False     |
| projectadmin | Gergő Tisza@Default  |       | wikispore@Default |        | False     |
| user         | Gergő Tisza@Default  |       | wikispore@Default |        | False     |
+--------------+----------------------+-------+-------------------+--------+-----------+

Mentioned in SAL (#wikimedia-cloud) [2019-08-28T17:45:28Z] <bd808> Project created (T231072)

bd808 closed this task as Resolved.Wed, Aug 28, 5:45 PM
Tgr moved this task from Backlog to Done on the Wikispore board.Tue, Sep 3, 4:13 PM