Page MenuHomePhabricator
Create Task
Maniphest T330617

Support for Recipes to set up dev environment
Open, MediumPublic
Actions

Description

a Wikimedia CLI currently provides quick and easy creation of wikis and related database, monitoring tools. Once wiki is created, a developer need to install extensions using the usual methods and edit LocalSettings.php. It would be nice to support a higher level abstraction for this common developer tasks.

For example https://www.mediawiki.org/wiki/MediaWiki-Docker/Configuration_recipes and https://www.mediawiki.org/wiki/MediaWiki-Docker lists a number of recipes a developer need to follow to set up a dev environment. They are often setting up multiple dependent extensions, sensible defaults for configuration settings, nodejs based services, user roles and so on.
Sometimes, a working environement would need multiple wikis in different languages like https://www.mediawiki.org/wiki/User:Santhosh.thottingal/WikiFamily.

Currently all these recipes are kind of tutorials. A developer need to read them, execute the steps one by one to get a usuable development environemnt. If they can be abstracted and exposed under a known, listed, auto-updatable, dependency chainable 'recipes', we can improve our developer experience.

@Addshore described it as:

In a nutshell, define something in a yaml file that specifies sites to install, maybe configs for those sites, and also services to run
Have that either locally, or point to a URL, and tada you have a dev environment that is defined per the spec.

Details

Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
Recipesrepos/releng/cli!646addshorerecipesmain
Customize query in GitLab

Related Objects

Event Timeline

santhosh created this task.Feb 27 2023, 4:20 AM
santhosh mentioned this in T294427: [mwcli] work on the ContentTranslation cxserver usecase.
Addshore awarded a token.Feb 27 2023, 11:07 AM
Addshore moved this task from Inbox to Discuss & Decide on the a Wikimedia CLI board.Feb 27 2023, 11:20 AM
Addshore added a comment.EditedFeb 27 2023, 11:32 AM

How does something like this look?

type: mwcli/recipe
version: 0.0.1
services:
  - mediawiki
  - mysql
  - cxserver
motd: |
  Welcome to the wiki family with multiple mediawiki instances with different languages
mediawiki:
  defaults:
    dbtype: mysql
    localsettings: |
      $wgLanguageCode = $wgDBname;
      wfLoadExtension( 'UniversalLanguageSelector' );
      wfLoadExtension( 'BetaFeatures' );
      wfLoadExtension( 'Cite' );
      wfLoadExtension( 'EventLogging' );
      wfLoadExtension( 'MobileFrontend' );
      
      wfLoadExtension( 'VisualEditor' );
      $wgDefaultUserOptions['visualeditor-enable'] = 1;
      // Parsoid/PHP required configuration
      $wgEnableRestAPI = true;
      wfLoadExtension( 'Parsoid', 'vendor/wikimedia/parsoid/extension.json' );
      $wgVirtualRestConfig['modules']['parsoid'] = [];
      
      wfLoadExtension( 'ContentTranslation' );
      $GLOBALS['wgContentTranslationDatabase'] = 'shared';
      $GLOBALS['wgContentTranslationAsBetaFeature'] = true;
      $wgContentTranslationVersion = 2;
      $wgContentTranslationEventLogging = true;
      $wgContentTranslationEnableMT = true;
      $wgContentTranslationEnableSectionTranslation = true;
      $wgContentTranslationEnableAnonSectionTranslation = true;
      $wgContentTranslationTranslateInTarget = true;
      $wgContentTranslationSiteTemplates = [
      	"view" => "//$1.mediawiki.mwdd.localhost:8080/wiki/$2",
      	"action" => "//$1.mediawiki.mwdd.localhost:8080/w/index.php?title=$2",
      	"api" => "//$1.wikipedia.org/w/api.php",
      	"cx" => "http://cxserver.mwdd.localhost:8080/v1",
      	"cookieDomain" => "mediawiki.mwdd.localhost",
      	"restbase" => "https://$1.wikipedia.org/api/rest_v1"
      ];
      $GLOBALS['wgContentTranslationCXServerAuth'] = [
      	"algorithm" => "HS256",
      	"key" => "secret",
      	"age" => "3600"
      ];
      $wgContentTranslationContentImportForSectionTranslation = true;
      $wgSectionTranslationTargetLanguages = ['bn',"ml", "is", "ig"];
  extensions:
    - name: Cite
    - name: ContentTranslation
    - name: UniversalLanguageSelector
    - name: VisualEditor
    - name: BetaFeatures
    - name: GlobalPreferences
    - name: MobileFrontend
    - name: EventLogging
  skins:
    - name: Vector
  install:
    - dbname: en
    - dbname: ml
    - dbname: ta
    - dbname: fr

https://codebeautify.org/yaml-editor-online/y23b69992

Addshore triaged this task as Medium priority.Mar 15 2023, 6:14 PM
KartikMistry awarded a token.Mar 16 2023, 4:59 AM
Tgr subscribed.Mar 16 2023, 5:26 AM

Do you want these to be combinable, or strictly one at a time? E.g. if I want to test a patch that's about some interaction between ContentTranslation and Echo, can I enable the ContentTranslation and Echo recipes, or would I need to create my own recipe based on those two?
If you want them to be combinable, you need some level of deduplication (wfLoadExtension calls at the very least).

Do you want to support some level of templating/encapsulation? For Vagrant roles, the ability to extract often needed, repetitive functionality (like setting up a systemd service) into a helper method has been quite nice for maintainability.

abi_ awarded a token.Mar 16 2023, 5:29 AM
Mooeypoo subscribed.Mar 16 2023, 5:40 AM
Joe subscribed.Mar 16 2023, 9:09 AM
Joe added a comment.Mar 16 2023, 9:14 AM

I would suggest to try to hide the complexity of setup of localsettings from the user, creating specific options, but leave the option to inject some piece of PHP into it by pointing to a local file to include in localsettings.

So for example, say I want content translation. That should be one of the "features" offered, like

mediawiki:
  content-translation:
     enable: true
     languages: [...]
  local_settings_file: "src/AdditionalLocalSettings.php"

Also: ideally all these features of "a wiki" you can set up out of mwcli should be "composable". So you can have various services up and running at the same time.

daniel subscribed.EditedMar 16 2023, 9:54 AM

For anything that should go into LocalSettings, please use the new https://www.mediawiki.org/wiki/Manual:YAML_settings_file_format! MW has support for this build in since 1.39. It can be used from LocalSettings.php by calling $wgSettings->loadFile( 'something' ).

Maintaining the config as a YAML file (orJSON, or a PHP array if you prefer) should make it a lot easier to mix and match and manipulate configuration. This is what the new format was designed for.

zeljkofilipin subscribed.Mar 16 2023, 11:38 AM
kostajh subscribed.EditedMar 16 2023, 11:47 AM

I think it would be nice to re-use extension.json if possible. While I think the aim is for the extensions to provide sensible defaults in extension.json config, there are cases where a new field like "development_value" could be useful"

"GELevelingUpFeaturesEnabled":
  "description": "Feature flag to enable \"leveling up\" (phab:#growthexperiments-levelingup) functionality.",
  "value": false,
  "development_value": true
}

Also, we have the requires field in extension.json but maybe something like optionalRequires could be added:

"optionalRequires": {
		"extensions": {
			"VisualEditor": "*",
		}
                "services": [ "cxserver" ]
	},

then it is up to the local development environment tooling to decide what to do with that data.

On a bigger point, it would be ideal if we define what a role / extension needs for its local environment such that it can be accessed and reused by mwcli, Vagrant, bash scripts, etc.

daniel added a comment.EditedMar 16 2023, 12:11 PM
In T330617#8701941, @kostajh wrote:

I think it would be nice to re-use extension.json if possible. While I think the aim is for the extensions to provide sensible defaults in extension.json config, there are cases where a new field like "development_value" could be useful"

We were looking at extension.json when we designed the structure for settings files. The main difference is that the "config" key in extension.json is designed to declare configuration, not set it. So it would be the equivalent of "config-schema" in a settings file. The mechanism for setting config in extension.json is through well known top level keys such as JobClasses or APIModules.

When we designed the schema, the idea was that version 3 of extension.json would be based on the settings file structure. That would also make the logic in ExtensionRegistration a lot simpler.

eamedina subscribed.Mar 16 2023, 1:33 PM
Nikerabbit subscribed.Mar 16 2023, 2:25 PM
Addshore added a comment.Mar 16 2023, 5:35 PM

Ohia all, thanks for all the discussion :)

Do you want these to be combinable, or strictly one at a time?

For a first version I think one at a time would make sense.

Note that mwcli docker has a --context flag, so you can have multiple development environments, possibly with totally different setups, and even different code and services (or versions of services) running at a time.

I would suggest to try to hide the complexity of setup of localsettings from the user, creating specific options, but leave the option to inject some piece of PHP into it by pointing to a local file to include in localsettings.

This is also similar to what is currently done.
Quite a lot of local settings magic is hidden from the user (they can of course look if they want to.
See https://gitlab.wikimedia.org/repos/releng/cli/-/blob/main/internal/mwdd/files/embed/mediawiki/MwddSettings.php
Disgetable examples would be

And of course a user can override any of this

Generally, It'd be nice if extensions just came with most of this configuration, and it could be enabled with some sort of configuration mode.
So for example you could enable "BetaFeatures" in a "wikimedia" mode, and the config would be similar / the same as in wikimedia production, or a "standalone" mode
This would hide lots of configuration complexity.

In T330617#8701483, @Joe wrote:
mediawiki:
  content-translation:
     enable: true
     languages: [...]
  local_settings_file: "src/AdditionalLocalSettings.php"

Also: ideally all these features of "a wiki" you can set up out of mwcli should be "composable". So you can have various services up and running at the same time.

I like this, but perhaps also for a v2 of the feature.

In T330617#8701559, @daniel wrote:

For anything that should go into LocalSettings, please use the new https://www.mediawiki.org/wiki/Manual:YAML_settings_file_format! MW has support for this build in since 1.39. It can be used from LocalSettings.php by calling $wgSettings->loadFile( 'something' ).

Maintaining the config as a YAML file (orJSON, or a PHP array if you prefer) should make it a lot easier to mix and match and manipulate configuration. This is what the new format was designed for.

Looks like I need to take a look at this!

Ollie.Shotton_WMDE subscribed.May 9 2025, 1:26 PM
Addshore mentioned this in T384373: Spike - try mw-cli!.May 22 2025, 8:51 AM
Addshore added a comment.May 31 2025, 10:14 AM

So, this continues being a topic that people seem to be very interested in in general.
There are already various efforts around of trying to collect this info to make it easier and automate various parts of setup for people.

MediaWiki-Docker takes the very manual approach, for example https://www.mediawiki.org/wiki/MediaWiki-Docker/Extension/MediaSearch
This page lists

  • Suggested or required additional extensions
  • Suggested additional config for said extensions that are likely needed for development
  • Additional commands or scripts that need to be run as part of the setup`updateSearchIndexConfig`

Perhaps the first part can already be inferred from extension.json (if it were populated) https://github.com/wikimedia/mediawiki-extensions-MediaSearch/blob/master/extension.json#L14-L16
However it seems like a half sensible place for it to live.

{
	"requires": {
		"MediaWiki": ">= 1.27.0",
		"extensions": {
			"FooBar": "*",
			"Baz": ">= 1.2.3"
		}
	}
}

However this begs the question, are the requirements of an extension the same as the recommended things to install while developing.
And probably the answer is no?
Also extensions can probably have multiple different developer environment setups, for example Wikibase might have a single site setup, as well as a Commons style setup, and a Wikimedia style setup multi site etc.

Nonetheless, I'd love for such information regarding recommended setups to be controlled in some way by the extension repos (so probably not on a wiki page), not tied to any specific environment (so no mwcli specific),

Essentialy this mediawiki-docker example boils down to....

skins: Vector, MinervaNeue
extensions: CirrusSearch, Cite, CommonsMetadata etc.....
services: elasticsearch
extra config.....

\$wgUseInstantCommons = true;
\$wgEnableUploads = true;
\$wgUseImageMagick = true;
\$wgScribuntoDefaultEngine = 'luastandalone';
\$wgMediaViewerEnableByDefault = true;
\$wgPFEnableStringFunctions = true;
\$wgApiFrameOptions = 'SAMEORIGIN';
\$wgCacheDirectory = "\$IP/cache";

// Cirrus Search
require_once "\$IP/extensions/CirrusSearch/tests/jenkins/FullyFeaturedConfig.php";
\$wgCirrusSearchServers = [ "elasticsearch" ];
\$wgSearchType = 'CirrusSearch';

// Wikibase
wfLoadExtension( 'WikibaseRepository', "\$IP/extensions/Wikibase/extension-repo.json" );
require_once "\$IP/extensions/Wikibase/repo/ExampleSettings.php";
wfLoadExtension( 'WikibaseClient', "\$IP/extensions/Wikibase/extension-client.json" );
require_once "\$IP/extensions/Wikibase/client/ExampleSettings.php";

\$wgEnableWikibaseRepo = true;
\$wgEnableWikibaseClient = true;

\$wgWBClientSettings['dataBridgeEnabled'] = true;
\$wgWBClientSettings['dataBridgeHrefRegExp'] = '[/=]((?:Item:)?(Q[1-9][0-9]*)).*#(P[1-9][0-9]*)$';
\$wgWBClientSettings['dataBridgeEditTags'] = [ 'Data Bridge' ];

\$wgWBRepoSettings['dataBridgeEnabled'] = true;
\$wgWBRepoSettings['taintedReferencesEnabled'] = true;
\$wgWBRepoSettings['termboxEnabled'] = true;

// WikibaseMediaInfo
\$wgMediaInfoEnableFilePageDepicts = true;
\$wgMediaInfoEnableOtherStatements = true;
\$wgMediaInfoShowQualifiers = true;
\$wgMediaInfoProperties = [
	'depicts' => 'P1',
];
\$wgDepictsQualifierProperties = [
	'depicts' => 'P1',
];

\$wgUseImageMagick = true;

// MediaSearch
\$useProductionSearchApi = true;
\$wgMediaSearchExternalSearchUri = \$useProductionSearchApi ? 'https://commons.wikimedia.org/w/api.php' : '';
\$wgMediaSearchExternalEntitySearchBaseUri = \$useProductionSearchApi ? 'https://www.wikidata.org/w/api.php' : '';

extra commands....

php extensions/CirrusSearch/maintenance/updateSearchIndexConfig.php
php extensions/CirrusSearch/maintenance/forceSearchIndex.php --skipLinks --indexOnSkip
php extensions/CirrusSearch/maintenance/forceSearchIndex.php --skipParse
php maintenance/runJobs.php
// udpate.php?
php extensions/Wikibase/lib/maintenance/populateSitesTable.php
php extensions/Wikibase/repo/maintenance/rebuildItemsPerSite.php
php extensions/Wikibase/client/maintenance/populateInterwiki.php

I'm not sure why \$wgDisableSearchUpdate = true; needs to be manually set to true initially, then changed?
Also \$wgSearchType = 'CirrusSearch'; is only added after some things are run?

As someone that just wants to facilitate people setting things up, the desire from my side would be most of the above is abstracted away behind commands that are already provided by the extensions as a composer command for development, or combined maintenance scripts?
Also ideally there should be no reason to poke LocalSettings mutliple times throughout the setup, and the various scripts and steps and config should just be optimized toward that.

Ideally....

  • clone extensions
  • add config (which ideally would just be loading the extensions)
  • add config specific to a certain setup, maybe a single additional require of require_once "\$IP/extensions/Wikibase/config/dev.php"; or a dev-multisite.php for example
  • run composer run ext-init or something as a standard thing in all extensions?
  • All done?
zeljkofilipin unsubscribed.Jun 4 2025, 10:02 AM
SDunlap subscribed.Aug 12 2025, 4:22 PM
seanleong-WMDE subscribed.May 1 2026, 1:03 PM
Addshore claimed this task.May 2 2026, 1:53 PM
Addshore moved this task from Discuss & Decide to In Progress on the a Wikimedia CLI board.
CodeReviewBot added a comment.May 2 2026, 1:56 PM

addshore updated https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/646

Draft: Recipes

Addshore added a comment.May 2 2026, 1:59 PM

So I have done an initial implementation, with a couple of included recipes.

The one liner for the wikibase client repo setup for @seanleong-WMDE is mw dev recipe --name wikibase-repoclient as this one is included as an example.
You can also provide a local --file or a remote --url

The YAML for that recipe is currently https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/646/diffs?file=2146a35e0a4d36aae322522f1c63759aac30e4ff#diff-content-2146a35e0a4d36aae322522f1c63759aac30e4ff cc @seanleong-WMDE
I also made content translation based on what I could find https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/646/diffs?file=d7eea196c1e3c6f2f131e4d107d0e9947e878b27#diff-content-d7eea196c1e3c6f2f131e4d107d0e9947e878b27 cc @santhosh

For convenance, "wikibase-repoclient":

type: mwcli.dev/recipe
version: 0.1
name: wikibase-repoclient
description: >-
  Wikibase repo+client setup. The repo wiki ("default") hosts Wikibase
  entities; the client wiki ("client") reads from the repo via entity
  sources and cross-wiki links.

services:
  - name: mediawiki
  - name: mysql

code:
  core: true
  skins:
    - name: Vector
  extensions:
    - name: Wikibase
    - name: ParserFunctions

sites:
  - dbname: default
    dbtype: mysql
  - dbname: client
    dbtype: mysql

jobRunner:
  sites:
    - default
    - client

localSettings:
  files:
    shared:
      - content: |
          wfLoadExtension( 'ParserFunctions' );

          $entitySources = [
            'local' => [
              'repoDatabase' => 'default',
              'baseUri' => 'http://default.mediawiki.local.wmftest.net:8080/entity/',
              'entityNamespaces' => [ 'item' => 120, 'property' => 122 ],
              'rdfNodeNamespacePrefix' => 'wd',
              'rdfPredicateNamespacePrefix' => 'wdt',
              'interwikiPrefix' => 'local',
              'type' => 'db',
            ],
          ];
    perWiki:
      default:
        - content: |
            wfLoadExtension( 'WikibaseRepository', "$IP/extensions/Wikibase/extension-repo.json" );
            require_once "$IP/extensions/Wikibase/repo/ExampleSettings.php";

            $wgWBRepoSettings['entitySources'] = $entitySources;
            $wgWBRepoSettings['localEntitySourceName'] = 'local';
      client:
        - content: |
            wfLoadExtension( 'WikibaseClient', "$IP/extensions/Wikibase/extension-client.json" );
            require_once "$IP/extensions/Wikibase/client/config/WikibaseClient.example.php";

            $wgWBClientSettings['repoUrl'] = 'http://default.mediawiki.local.wmftest.net:8080';
            $wgWBClientSettings['repoSiteName'] = 'default';
            $wgWBClientSettings['repoArticlePath'] = '/wiki/$1';
            $wgWBClientSettings['repoScriptPath'] = '/w';

            $wgWBClientSettings['entitySources'] = $entitySources;
            $wgWBClientSettings['itemAndPropertySourceName'] = 'local';
            $wgWBClientSettings['siteLinkGroups'] = [ 'local' ];
            $wgWBClientSettings['siteGlobalID'] = 'client';

maintenance:
  - name: rebuild-l10n-cache-default
    command: ["php", "maintenance/run.php", "rebuildLocalisationCache.php", "--wiki", "default", "--lang", "en"]
  - name: rebuild-l10n-cache-client
    command: ["php", "maintenance/run.php", "rebuildLocalisationCache.php", "--wiki", "client", "--lang", "en"]
  - name: populate-sites-table-client
    command: ["php", "maintenance/run.php", "./extensions/Wikibase/lib/maintenance/populateSitesTable.php", "--wiki", "client"]
  - name: add-site-default-on-client
    command: ["php", "maintenance/run.php", "addSite.php", "--", "default", "local", "--interwiki-id", "default", "--pagepath", "http://default.mediawiki.local.wmftest.net:8080/w/index.php?title=$1", "--language", "en", "--filepath", "http://default.mediawiki.local.wmftest.net:8080/w/$1", "--wiki", "client"]
  - name: add-site-client-on-client
    command: ["php", "maintenance/run.php", "addSite.php", "--", "client", "local", "--interwiki-id", "client", "--pagepath", "http://client.mediawiki.local.wmftest.net:8080/w/index.php?title=$1", "--language", "en", "--filepath", "http://client.mediawiki.local.wmftest.net:8080/w/$1", "--wiki", "client"]

content:
  wikibase:
    properties:
      - id: P1
        label: text
        datatype: string
    items:
      - id: Q1
        label: first
        claims:
          - property: P1
            value: hello
  pages:
    - wiki: client
      title: Main_Page
      text: "{{#property:P1|from=Q1}}"
Addshore added a comment.May 2 2026, 2:05 PM

Just noticed I hardcoded the port of 8080 in the YAML, so I'll get that removed too so it can be dynamically setup

Addshore mentioned this in T393764: Attempt to wrap mediawiki-docker in mwcli.May 3 2026, 7:01 AM
CodeReviewBot added a comment.May 3 2026, 8:45 AM

addshore merged https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/646

Recipes

Addshore moved this task from In Progress to Pending Release on the a Wikimedia CLI board.May 3 2026, 8:45 AM

It's in main now
Build artifact at https://gitlab.wikimedia.org/repos/releng/cli/-/jobs/813210/artifacts/download

You can try this dev build with

mw update --version=https://gitlab.wikimedia.org/repos/releng/cli/-/jobs/813210/artifacts/download
seanleong-WMDE awarded a token.May 3 2026, 10:02 AM
Addshore added a project: Wikimedia-Hackathon-2026.May 3 2026, 1:54 PM
Quiddity mentioned this in T422536: 🥳 Wikimedia Hackathon 2026 Project Showcase / Closing Session.May 4 2026, 1:49 AM
• Aklapper moved this task from Backlog to Proposed Projects on the Wikimedia-Hackathon-2026 board.May 4 2026, 9:00 AM
Addshore added a comment.May 5 2026, 11:07 AM

@seanleong-WMDE I wonder if you have any other feedback on this after playing around with it a bit?
Otherwise I'll try and push it out in a real release this week!

seanleong-WMDE added a comment.EditedMay 5 2026, 7:55 PM

Hi @Addshore, I played around with it for a while, and everything looks good so far! Something to take note (not related to this ticket), it seems to hit the rate limit, error: 429, while cloning Wikibase and its submodule fresh atm, shouldn't be a problem when the repo is already cloned.

Addshore added a comment.May 5 2026, 9:11 PM

@seanleong-WMDE I actually saw this once while at the hackathon, but was hoping that It was just the hackathon internet.
I believe it is from the http clones from from the phabricator git remote, currently

[submodule "view/lib/wikibase-serialization"]
	path = view/lib/wikibase-serialization
	url = https://phabricator.wikimedia.org/source/wikibase-serialization.git
[submodule "view/lib/wikibase-data-values"]
	path = view/lib/wikibase-data-values
	url = https://phabricator.wikimedia.org/source/datavalues-javascript.git
[submodule "view/lib/wikibase-data-model"]
	path = view/lib/wikibase-data-model
	url = https://phabricator.wikimedia.org/source/wikibase-data-model.git

I think T374926: [EPIC][Infra] Move Wikibase and WikibaseLexeme Git submodules to suitable Git host relates to this, but also the situation might be worse with the more recent rate limit changes...

Addshore mentioned this in T374926: [EPIC][Infra] Move Wikibase and WikibaseLexeme Git submodules to suitable Git host.May 5 2026, 9:12 PM
seanleong-WMDE added a comment.May 7 2026, 3:54 PM

@Addshore, I see. I saw this ticket while searching up as well. This happens on the other docker-dev environment as well, but I think if the submodules were to merge into the Wikibase repository, it should be fine? But I agree, with the recent rate limit changes, it might be worse until some other changes are made.

Addshore mentioned this in T413117: Draft proposal (dibb bet format) for TP/DevEx investment in mwcli for common local dev, patchdemo, and CI/quibble environment provisioning.May 8 2026, 8:40 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits