RFC: Provide mechanism for defining and utilizing configuration sets for local development and browser / API-testing tests
- Affected components: MediaWiki core, extensions / skins that want to use the mechanism(s)
- Engineer(s) or team for initial implementation: TBD.
- Code steward: TBD.
Motivation
I would like to be able to easily define configuration modes (services as well as individual configuration variables). These modes could be used for simplifying developer setups in local development, as well as for providing more flexibility for automated testing.
I'm creating this as an RFC so that various teams and contributors can collectively discuss what we want to see and what would work well.
Requirements
- The ability to define ServiceWiring overrides for a particular environment. For example when my extension is operating in the context of CI tests, I want to redefine some services that my extension provides to use static implementations instead of production implementations that rely on services which may not exist (ElasticSearch, Memcache, etc).
- The patch in "Exploration" provides the ability to define "ServiceWiringCI.php" and "ServiceWiringDevelopment.php" which extend and override the primary, production ServiceWiring.php file
- The ability to override particular Config values for a given environment. T173955 might have some overlap here, or maybe not. Ideally one would be able to execute tests in parallel with configuration specific to those tests.
- The ability to define and override arbitrary environments. "CI" might work well for some extensions, but other features are more complicated and have a variety of modes they operate under, so we should be able to define a common CI configuration and then sub configurations that could inherit from the common one.
- An easy way to determine, in code, which environment mode(s) you are in, i.e. Production, Development, CI, or whatever other custom variant you have defined. Symfony framework has some prior art we could look at here if we don't want to invent our own thing. I could also see how local developer setups and browser / api-testing tests may want to specify that the site is operating in multiple environment modes depending on what is defined by a given extension (e.g. "use GrowthExperiments remote configuration loader mode and Vector version 2 mode").
- Currently we have code which relies on $wgWikimediaJenkinsCI, whatever mechanism we set up here would replace that
- Setting the environment mode should be possible via the HTTP request (or CLI command), perhaps using a cookie, a request header, request parameter, or an environment variable, so that the Selenium or api-testing framework does not necessarily need to be running on the same filesystem as MediaWiki, and to allow for parallel tests. An environment variable would be handy for MediaWiki-Docker for less technical users to be able to easily run MediaWiki in a given mode(s).
See also:
- T230211: Ensure that GET requests in API integration tests will always see the effect of previous POST requests.
- T199939: Audit tests/selenium/LocalSettings.php file aiming at possibly deprecating the feature
- T240244: Growthexperiments - automation tests for Homepage Suggested edits / T243516: Use StaticTaskSuggester and StaticConfigurationLoader when in CI
- T267304: Introduce a production/test environment flag in GrowthExperiments
Exploration
A rough idea would be something like this:
- a config variable like $wgConfigSets in core. This config variable can be appended to by core (not sure where the logical place would be exactly) and by extensions and skins.
- each configuration set has a key for its ID (a string, e.g. "growthexperiments-static-suggested-edits-module") and then an object containing:
- path to ServiceWiring.php for the configuration set. The services defined for this configuration set will override existing services, rather than error on collision.
- path to LocalSettingsOverride.php file. This file looks like a LocalSettings.php, used for overriding values set in LocalSettings.php/DefaultSettings.php
- list of IDs of other configuration sets to use, these are evaluated before the configuration set where this list is defined. This way you can build multiple variants of an existing configuration setup without having to copy/paste a bunch of files/code.
- description of what the configuration does. It would be nice to be able to browse or search the list via php maintenance/showConfigurationSets.php
- With the configuration sets now defined, in LocalSettings.php, a developer can specify something like $wgActiveConfigSets = [ 'growthexperiments-static-suggested-edits-module' ];.
- In theory a lot of the configuration settings currently done in MediaWiki-Vagrant could be migrated to this system, and in Vagrant you would just set $wgActiveConfigSets += [ 'growthexperiments-vagrant' ].
- Similarly, instead of the "Add this to your LocalSettings.php" steps in a lot of recipes on MediaWiki-Docker's help pages we could tell developers to add a one liner to their LocalSettings.php, or if T173955: Allow additional LocalSettings to be overriden by environment variables is done some day, they could add to their docker-compose.override.yml file and not have to mess with LocalSettings.php at all
- Now, for browser and API tests we need the ability to specify which configuration set(s) to make active in the context of a Selenium test or api-testing test.
- We would define a feature flag like $wgAllowConfigSetsOverrideFromRequest with a default to false.
- If it's set to true (which we'd do in Quibble's LocalSettings override in CI), then clients can send a header like X-Active-ConfigSets with a list of config sets to enable for a given request. This should also allow for parallel test execution of the Selenium/api-testing tests with different configurations.
Decision Statement Overview Artifact
n.b. canonical source is here: Decision Statement Overview, the following is copy/pasted so those who don't want to interact with Google Docs can review / comment.
What
Write your problem statement using layperson’s terminology. In one sentence, what is the problem or opportunity?
It is more difficult than it needs to be for less-technical users, new and experienced developers, and test writers to find, share, and keep up-to-date with configuration relevant to their needs, and we lack testing infrastructure to assess different configuration sets for end-to-end tests.
What does the future look like if this is achieved?
MediaWiki and extension + skins have configuration flags. These can be mixed and matched in multiple ways. There are default settings provided by core / extensions / skins, but these are not always developer friendly nor are they guaranteed to match what is in production.
Developers who want to set up core, extension, or skin to work on a particular feature need to manually update LocalSettings.php with various configuration snippets that can become out of date over time. Managing this is time-consuming, error-prone and results in frustration for newer and experienced developers.
In our continuous integration environment, automated tests run using the default configuration settings provided by core, extensions, and skins. That means that different configuration settings, including those used in production, are not tested.
Putting the above statements together: the problem is that it is too difficult for less-technical users, new developers, experienced developers, and test writers to find, share, and keep up-to-date with configuration relevant to their needs.
The proposal is to add infrastructure in MediaWiki core and convention in our community for defining configuration modes. A “mode” consists of an ID and a set of 1) ServiceWiring files, 2) configuration value overrides, 3) list of other configuration modes to inherit from.
Developers and test writers could then specify which configuration mode(s) they want active in their local environment. Developers would add the configuration modes they want active to a variable like $wgActiveConfigurationModes to LocalSettings.php; browser tests and API tests could use a cookie or request header to specify which configuration modes should be active for the particular request.
To encourage collaboration and discoverability, a maintenance script and/or Special page listing the available modes for core + extensions/skins could be created.
What happens if we do nothing?
The status quo: it remains difficult, time-consuming, and error-prone to set up and maintain one’s local environment for features provided by various skins and extensions and testing features in various configurations in end-to-end tests is difficult and mostly not done.
Why?
Identify the value(s) this problem/opportunity provides. Add links to relevant OKRs.
- Platform Evolution
- Engineering productivity easier for engineers to get different extensions/skins set up and configured correctly over time
- Fully automated and continuous code health and deployment infrastructure automated testing for non-default (i.e. production) configuration settings. This project would also benefit the upcoming preview environments project.
- Tooling for contributors is easy to use, well-documented, and accessible to users, increasing engagement and contribution easier for less-technical users to get started and make contributions
- Thriving Movement
- We will welcome and support newcomers easier for new users to get set up and begin contributing
Why are you bringing this decision to the Technical Forum?
Every team has had to find a solution to the problem of how to set up extension configuration, how to document it, and how to utilize that in tests. Input from this shared experience would result in a stronger technical proposal, and would also help with buy-in, because if teams did not use the proposed “configuration modes” they would not be useful.