Page MenuHomePhabricator

Support for Recipes to set up dev environment
Open, MediumPublic

Assigned To
None
Authored By
santhosh
Feb 27 2023, 4:20 AM
Referenced Files
None
Tokens
"Like" token, awarded by abi_."Like" token, awarded by KartikMistry."Mountain of Wealth" token, awarded by Addshore.

Description

mwcli currently provides quick and easy creation of wikis and related database, monitoring tools. Once wiki is created, a developer need to install extensions using the usual methods and edit LocalSettings.php. It would be nice to support a higher level abstraction for this common developer tasks.

For example https://www.mediawiki.org/wiki/MediaWiki-Docker/Configuration_recipes and https://www.mediawiki.org/wiki/MediaWiki-Docker lists a number of recipes a developer need to follow to set up a dev environment. They are often setting up multiple dependent extensions, sensible defaults for configuration settings, nodejs based services, user roles and so on.
Sometimes, a working environement would need multiple wikis in different languages like https://www.mediawiki.org/wiki/User:Santhosh.thottingal/WikiFamily.

Currently all these recipes are kind of tutorials. A developer need to read them, execute the steps one by one to get a usuable development environemnt. If they can be abstracted and exposed under a known, listed, auto-updatable, dependency chainable 'recipes', we can improve our developer experience.

@Addshore described it as:

In a nutshell, define something in a yaml file that specifies sites to install, maybe configs for those sites, and also services to run
Have that either locally, or point to a URL, and tada you have a dev environment that is defined per the spec.

Event Timeline

How does something like this look?

type: mwcli/recipe
version: 0.0.1
services:
  - mediawiki
  - mysql
  - cxserver
motd: |
  Welcome to the wiki family with multiple mediawiki instances with different languages
mediawiki:
  defaults:
    dbtype: mysql
    localsettings: |
      $wgLanguageCode = $wgDBname;
      wfLoadExtension( 'UniversalLanguageSelector' );
      wfLoadExtension( 'BetaFeatures' );
      wfLoadExtension( 'Cite' );
      wfLoadExtension( 'EventLogging' );
      wfLoadExtension( 'MobileFrontend' );
      
      wfLoadExtension( 'VisualEditor' );
      $wgDefaultUserOptions['visualeditor-enable'] = 1;
      // Parsoid/PHP required configuration
      $wgEnableRestAPI = true;
      wfLoadExtension( 'Parsoid', 'vendor/wikimedia/parsoid/extension.json' );
      $wgVirtualRestConfig['modules']['parsoid'] = [];
      
      wfLoadExtension( 'ContentTranslation' );
      $GLOBALS['wgContentTranslationDatabase'] = 'shared';
      $GLOBALS['wgContentTranslationAsBetaFeature'] = true;
      $wgContentTranslationVersion = 2;
      $wgContentTranslationEventLogging = true;
      $wgContentTranslationEnableMT = true;
      $wgContentTranslationEnableSectionTranslation = true;
      $wgContentTranslationEnableAnonSectionTranslation = true;
      $wgContentTranslationTranslateInTarget = true;
      $wgContentTranslationSiteTemplates = [
      	"view" => "//$1.mediawiki.mwdd.localhost:8080/wiki/$2",
      	"action" => "//$1.mediawiki.mwdd.localhost:8080/w/index.php?title=$2",
      	"api" => "//$1.wikipedia.org/w/api.php",
      	"cx" => "http://cxserver.mwdd.localhost:8080/v1",
      	"cookieDomain" => "mediawiki.mwdd.localhost",
      	"restbase" => "https://$1.wikipedia.org/api/rest_v1"
      ];
      $GLOBALS['wgContentTranslationCXServerAuth'] = [
      	"algorithm" => "HS256",
      	"key" => "secret",
      	"age" => "3600"
      ];
      $wgContentTranslationContentImportForSectionTranslation = true;
      $wgSectionTranslationTargetLanguages = ['bn',"ml", "is", "ig"];
  extensions:
    - name: Cite
    - name: ContentTranslation
    - name: UniversalLanguageSelector
    - name: VisualEditor
    - name: BetaFeatures
    - name: GlobalPreferences
    - name: MobileFrontend
    - name: EventLogging
  skins:
    - name: Vector
  install:
    - dbname: en
    - dbname: ml
    - dbname: ta
    - dbname: fr

https://codebeautify.org/yaml-editor-online/y23b69992

Addshore triaged this task as Medium priority.Mar 15 2023, 6:14 PM

Do you want these to be combinable, or strictly one at a time? E.g. if I want to test a patch that's about some interaction between ContentTranslation and Echo, can I enable the ContentTranslation and Echo recipes, or would I need to create my own recipe based on those two?
If you want them to be combinable, you need some level of deduplication (wfLoadExtension calls at the very least).

Do you want to support some level of templating/encapsulation? For Vagrant roles, the ability to extract often needed, repetitive functionality (like setting up a systemd service) into a helper method has been quite nice for maintainability.

I would suggest to try to hide the complexity of setup of localsettings from the user, creating specific options, but leave the option to inject some piece of PHP into it by pointing to a local file to include in localsettings.

So for example, say I want content translation. That should be one of the "features" offered, like

mediawiki:
  content-translation:
     enable: true
     languages: [...]
  local_settings_file: "src/AdditionalLocalSettings.php"

Also: ideally all these features of "a wiki" you can set up out of mwcli should be "composable". So you can have various services up and running at the same time.

For anything that should go into LocalSettings, please use the new https://www.mediawiki.org/wiki/Manual:YAML_settings_file_format! MW has support for this build in since 1.39. It can be used from LocalSettings.php by calling $wgSettings->loadFile( 'something' ).

Maintaining the config as a YAML file (orJSON, or a PHP array if you prefer) should make it a lot easier to mix and match and manipulate configuration. This is what the new format was designed for.

I think it would be nice to re-use extension.json if possible. While I think the aim is for the extensions to provide sensible defaults in extension.json config, there are cases where a new field like "development_value" could be useful"

"GELevelingUpFeaturesEnabled":
  "description": "Feature flag to enable \"leveling up\" (phab:#growthexperiments-levelingup) functionality.",
  "value": false,
  "development_value": true
}

Also, we have the requires field in extension.json but maybe something like optionalRequires could be added:

"optionalRequires": {
		"extensions": {
			"VisualEditor": "*",
		}
                "services": [ "cxserver" ]
	},

then it is up to the local development environment tooling to decide what to do with that data.

On a bigger point, it would be ideal if we define what a role / extension needs for its local environment such that it can be accessed and reused by mwcli, Vagrant, bash scripts, etc.

I think it would be nice to re-use extension.json if possible. While I think the aim is for the extensions to provide sensible defaults in extension.json config, there are cases where a new field like "development_value" could be useful"

We were looking at extension.json when we designed the structure for settings files. The main difference is that the "config" key in extension.json is designed to declare configuration, not set it. So it would be the equivalent of "config-schema" in a settings file. The mechanism for setting config in extension.json is through well known top level keys such as JobClasses or APIModules.

When we designed the schema, the idea was that version 3 of extension.json would be based on the settings file structure. That would also make the logic in ExtensionRegistration a lot simpler.

Ohia all, thanks for all the discussion :)

Do you want these to be combinable, or strictly one at a time?

For a first version I think one at a time would make sense.

Note that mwcli docker has a --context flag, so you can have multiple development environments, possibly with totally different setups, and even different code and services (or versions of services) running at a time.

I would suggest to try to hide the complexity of setup of localsettings from the user, creating specific options, but leave the option to inject some piece of PHP into it by pointing to a local file to include in localsettings.

This is also similar to what is currently done.
Quite a lot of local settings magic is hidden from the user (they can of course look if they want to.
See https://gitlab.wikimedia.org/repos/releng/cli/-/blob/main/internal/mwdd/files/embed/mediawiki/MwddSettings.php
Disgetable examples would be

And of course a user can override any of this

Generally, It'd be nice if extensions just came with most of this configuration, and it could be enabled with some sort of configuration mode.
So for example you could enable "BetaFeatures" in a "wikimedia" mode, and the config would be similar / the same as in wikimedia production, or a "standalone" mode
This would hide lots of configuration complexity.

mediawiki:
  content-translation:
     enable: true
     languages: [...]
  local_settings_file: "src/AdditionalLocalSettings.php"

Also: ideally all these features of "a wiki" you can set up out of mwcli should be "composable". So you can have various services up and running at the same time.

I like this, but perhaps also for a v2 of the feature.

For anything that should go into LocalSettings, please use the new https://www.mediawiki.org/wiki/Manual:YAML_settings_file_format! MW has support for this build in since 1.39. It can be used from LocalSettings.php by calling $wgSettings->loadFile( 'something' ).

Maintaining the config as a YAML file (orJSON, or a PHP array if you prefer) should make it a lot easier to mix and match and manipulate configuration. This is what the new format was designed for.

Looks like I need to take a look at this!