Page MenuHomePhabricator

Provision test instance for "Templates" projects
Closed, ResolvedPublic13 Estimated Story Points

Description

(compare with e.g. http://de.wikipedia.org/wiki/Special:Version):
Extensions

  • CentralAuth
  • CharInsert
  • Cite (used by many templates!)
  • CodeEditor
  • CodeMirror
  • FlaggedRevs
  • Gadgets
  • InviteSignup
  • MobileFrontend
  • ParserFunctions
  • Parsoid (only the new repository, mediawiki/services/parsoid)
  • Scribunto & full Lua scripting language
  • SyntaxHighlight
  • TemplateData
  • TemplateStyles
  • TemplateWizard
  • TimedMediaHandler
  • WikibaseClient
  • WikiEditor
  • VisualEditor (custom role, not the built-in mw-vagrant role for Parsoid/JS)

Skins

  • MonoBook
  • Timeless
  • Vector

Gadgets

https://wikitech.wikimedia.org/wiki/Help:MediaWiki-Vagrant_in_Cloud_VPS

Steps to provision
Set up puppet roles, for any instance with a name starting mediawiki-:

role::labs::mediawiki_vagrant
role::labs::lvm::srv

Create instance with 2 cores. Add the default security group, and a new web security group allowing access to TCP port 8080.

Change wiki admin password.

Apply our custom branches:

git fetch "https://gerrit.wikimedia.org/r/mediawiki/vagrant" refs/changes/21/616521/3 && git cherry-pick FETCH_HEAD

git fetch "https://gerrit.wikimedia.org/r/mediawiki/vagrant" refs/changes/07/613207/7 && git cherry-pick FETCH_HEAD

Create the vagrant box with no roles at first,

mwvagrant up

Enable vagrant roles,

mwvagrant roles enable \
    betafeatures \
    centralauth \
    charinsert \
    cite \
    codeeditor \
    codemirror \
    flaggedrevs \
    gadgets \
    invitesignup \
    mobilefrontend \
    monobook \
    parserfunctions \
    parsoid_php \
    scribunto \
    syntaxhighlight \
    templatedata \
    templatestyles \
    templatewizard \
    timeless \
    titleblacklist \
    wikidata \
    wikieditor \
    wmde_templates

Update code and database:

mwvagrant git-update

Convert Admin user to CentralAuth

mwvagrant ssh
...
mwscript /vagrant/mediawiki/extensions/CentralAuth/maintenance/migratePass0.php --wiki=wiki
mwscript /vagrant/mediawiki/extensions/CentralAuth/maintenance/migratePass1.php --wiki=wiki

Configured outbound mail according to these manuals. The settings are already bundled with the custom patch above.
https://wikitech.wikimedia.org/wiki/Help:Email_in_Cloud_VPS
https://www.mediawiki.org/wiki/Manual:$wgSMTP

Create web proxy subdomains
https://dev-wmde-templates-alpha.wmcloud.org/
https://de-wmde-templates-alpha.wmcloud.org/
https://en-wmde-templates-alpha.wmcloud.org/
https://fa-wmde-templates-alpha.wmcloud.org/
https://tr-wmde-templates-alpha.wmcloud.org/
https://login-wmde-templates-alpha.wmcloud.org/
https://mobile-wmde-templates-alpha.wmcloud.org/
https://wikidata-wmde-templates-alpha.wmcloud.org/

Remaining work

  • Wikidata role isn't acting right. This patch was suddenly required for unknown reasons. MediaWiki databases did not self-update.
  • Subdomains should be accessible through "eswiki-wmde-templates-alpha.wmcloud.org", "wikidatawiki-wmde-templates-alpha.wmcloud.org", but these proxies don't seem to be routed successfully by Vagrant's Apache. The vagrant+cloud wikifarm docs aren't quite working for us. Maybe the hostnames are something other than db name?
  • Parsoid/PHP and VisualEditor configuration.
  • Using vagrant reload or vagrant halt may break the Vagrant instance (attempting to visit any wiki will lead to a "No wiki found..." page). Workaround: mwvagrant ssh and sudo service apache2 restart

Related tickets:

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Lena_WMDE renamed this task from Set up test instance for "Templates" projects to Provision test instance for "Templates" projects.Jul 8 2020, 10:19 AM
Lena_WMDE updated the task description. (Show Details)
Lena_WMDE updated the task description. (Show Details)
Lena_WMDE set the point value for this task to 5.Jul 8 2020, 10:26 AM

Current scope: set up machine, have MediaWiki with all extension installed, set all team members up as admins.

We may want to include gadgets and extensions related to syntax highlighting (though we do not plan to touch them, we do want to make sure we aren't breaking them somehow):

Extensions
SyntaxHighlighter
SyntaxHighlight

Gadgets
Syntax highlighter
wikEd

Lena_WMDE updated the task description. (Show Details)
Lena_WMDE updated the task description. (Show Details)
Lena_WMDE updated the task description. (Show Details)

I've been having trouble creating our instance. Puppet roles are set for the project so that any instance named "mediawiki*" will get the MediaWiki-vagrant role, with spare disk mounted at /srv. However, vagrant up fails for both buster and stretch host images, with an NFS error. This kind of thing,

mount.nfs: mount to NFS server '10.0.3.1:/srv/mediawiki-vagrant' failed: RPC Error: Unable to receive

Word on the street is that it will "eventually" work, maybe after 5 reboots and a vegan poultry sacrifice.

Meanwhile, here are some breadcrumbs,
All team devs are project admins: https://openstack-browser.toolforge.org/project/wmde-templates-alpha
Project admin log: https://wikitech.wikimedia.org/wiki/Nova_Resource:Wmde-templates-alpha/SAL
Current instance: ssh mediawiki1003.wmde-templates-alpha.eqiad.wmflabs
Web proxy: http://wmde-templates.wmflabs.org

Most of the extensions are supplied by built-in roles,

vagrant roles enable \
    cite \
    codeeditor \
    codemirror \
    flaggedrevs \
    mobilefrontend \
    parserfunctions \
    scribunto \
    syntaxhighlight \
    templatedata \
    templatestyles \
    templatewizard \
    visualeditor \
    wikieditor

Wikidata requires special steps, see these instructions.

Extension:CharInsert doesn't have a vagrant role yet, but I'll create one.

Extension:SyntaxHighliter is not in use on production wikis, and the last source code update was 5 years ago. I suggest we remove it from our compatibility requirements.

What is "TemplateDialog"? I can't find any reference to an extension by this name and nothing shows up on dewiki Special:Version.

Change 612547 had a related patch set uploaded (by Awight; owner: Awight):
[mediawiki/vagrant@master] New role for Extension:CharInsert

https://gerrit.wikimedia.org/r/612547

@awight If that's the case, then go ahead and remove Extension:SyntaxHighlighter as a requirement. I guess we can assume not many people are using it, so leaving it out of our consideration seems fine to me.

Ya I don't think there is a TemplateDialog extension. It's referencing this dialog: https://www.mediawiki.org/wiki/Help:VisualEditor/User_guide#Editing_templates - I'm not sure what needs to pulled in for the code. I have a feeling it's just part of Visual Editor and nothing extra needs to be done.

I've moved MobileFrontend and FlaggedRevs into the main requirements, it's effortless to include them. We can review and exclude again later if needed.

This is up for grabs, please see "Remaining work" for some next steps.

We might be the only ones trying to run VisualEditor/Parsoid under Cloud VPS at the moment. Hoping to find a role model for working config, however try this prototype server for the VE team, https://visualeditor-prototype.wmflabs.org/wiki/Main_Page?action=edit . It's broken.

Many things are wrong with visualeditor role on our instance, it doesn't take the hostname into account, Parsoid ports don't match, both PHP and JS parsoid are enabled but neither seems to work...

Trying a new web proxy subdomain scheme, to align with existing apache routes:
https://dev-wmde-templates-alpha.wmcloud.org/

This should make it simple to reach other hosts, for example:
https://wikidata-wmde-templates-alpha.wmcloud.org/

Change 613207 had a related patch set uploaded (by Andrew-WMDE; owner: Andrew-WMDE):
[mediawiki/vagrant@master] [DNM] Test instance configuration for "Templates" projects

https://gerrit.wikimedia.org/r/613207

Change 612547 merged by jenkins-bot:
[mediawiki/vagrant@master] New role for Extension:CharInsert

https://gerrit.wikimedia.org/r/612547

This seems relevant: https://gerrit.wikimedia.org/r/c/integration/quibble/+/612937
services/parsoid provides the v3 API, the one expected by current VisualEditor. mediawiki-core's built-in Parsoid/PHP seems to only have the v1 API.

I'm not sure if this is covered already, but I just realized we should make sure that anything needed to support Lua should be included. In particular, I know that there is a separate source code editor for programming languages. Is this already there?

In T257322#6325740, @ecohen wrote:

I'm not sure if this is covered already, but I just realized we should make sure that anything needed to support Lua should be included. In particular, I know that there is a separate source code editor for programming languages. Is this already there?

+1 This is provided by the "Scribunto" extension, which has been installed.

awight updated the task description. (Show Details)
awight changed the point value for this task from 5 to 13.Jul 28 2020, 11:07 AM
awight moved this task from Doing to Demo on the WMDE-QWERTY-Sprint-2020-07-22 board.

Raised the estimate.

But we're done, hooray!

Not sure how we ended up pulling in wikidata, but it came almost for free...

This comment was removed by awight.

Unfortunately found something that was missing from the initial requirements: beta feature "New wikitext mode" which gives you the source editor in VE: https://www.mediawiki.org/wiki/2017_wikitext_editor

Unfortunately found something that was missing from the initial requirements: beta feature "New wikitext mode" which gives you the source editor in VE: https://www.mediawiki.org/wiki/2017_wikitext_editor

  • Verified resolved :-)

I've been having trouble creating our instance. Puppet roles are set for the project so that any instance named "mediawiki*" will get the MediaWiki-vagrant role, with spare disk mounted at /srv. However, vagrant up fails for both buster and stretch host images, with an NFS error. This kind of thing,

mount.nfs: mount to NFS server '10.0.3.1:/srv/mediawiki-vagrant' failed: RPC Error: Unable to receive

Word on the street is that it will "eventually" work, maybe after 5 reboots and a vegan poultry sacrifice.

I ran into the same error message, filed T259431: labs-vagrant broken due to NFS issues about it.

thiemowmde added a subscriber: thiemowmde.

I don't know what's going on, but none of the URLs mentioned in this ticket work at the moment. Most (e.g. https://de-wmde-templates-alpha.wmcloud.org) bring up a MediaWiki error message that says "No wiki found […] Please specify a valid Host header." https://wiki-wmde-templates-alpha.wmcloud.org can not even be resolved.

I don't know what's going on, but none of the URLs mentioned in this ticket work at the moment. Most (e.g. https://de-wmde-templates-alpha.wmcloud.org) bring up a MediaWiki error message that says "No wiki found […] Please specify a valid Host header." https://wiki-wmde-templates-alpha.wmcloud.org can not even be resolved.

Here's the workaround. We're not totally sure what causes the problem, but it sometimes manifests after a vagrant reload.

cd /srv/mediawiki-vagrant
mwvagrant ssh
...
sudo service apache2 restart

https://wiki-wmde-templates-alpha.wmcloud.org can not even be resolved.

Thanks, this subdomain is outdated, I've removed it from the list.

Lena_WMDE claimed this task.