Page MenuHomePhabricator

Configuration Modes
Open, MediumPublic

Description

RFC: Provide mechanism for defining and utilizing configuration sets for local development and browser / API-testing tests

Decision Statement Overview

  • Affected components: MediaWiki core, extensions / skins that want to use the mechanism(s)
  • Engineer(s) or team for initial implementation: TBD.
  • Code steward: TBD.

Motivation

I would like to be able to easily define configuration modes (services as well as individual configuration variables). These modes could be used for simplifying developer setups in local development, as well as for providing more flexibility for automated testing.

I'm creating this as an RFC so that various teams and contributors can collectively discuss what we want to see and what would work well.

Requirements
  • The ability to define ServiceWiring overrides for a particular environment. For example when my extension is operating in the context of CI tests, I want to redefine some services that my extension provides to use static implementations instead of production implementations that rely on services which may not exist (ElasticSearch, Memcache, etc).
    • The patch in "Exploration" provides the ability to define "ServiceWiringCI.php" and "ServiceWiringDevelopment.php" which extend and override the primary, production ServiceWiring.php file
  • The ability to override particular Config values for a given environment. T173955 might have some overlap here, or maybe not. Ideally one would be able to execute tests in parallel with configuration specific to those tests.
  • The ability to define and override arbitrary environments. "CI" might work well for some extensions, but other features are more complicated and have a variety of modes they operate under, so we should be able to define a common CI configuration and then sub configurations that could inherit from the common one.
  • An easy way to determine, in code, which environment mode(s) you are in, i.e. Production, Development, CI, or whatever other custom variant you have defined. Symfony framework has some prior art we could look at here if we don't want to invent our own thing. I could also see how local developer setups and browser / api-testing tests may want to specify that the site is operating in multiple environment modes depending on what is defined by a given extension (e.g. "use GrowthExperiments remote configuration loader mode and Vector version 2 mode").
    • Currently we have code which relies on $wgWikimediaJenkinsCI, whatever mechanism we set up here would replace that
  • Setting the environment mode should be possible via the HTTP request (or CLI command), perhaps using a cookie, a request header, request parameter, or an environment variable, so that the Selenium or api-testing framework does not necessarily need to be running on the same filesystem as MediaWiki, and to allow for parallel tests. An environment variable would be handy for MediaWiki-Docker for less technical users to be able to easily run MediaWiki in a given mode(s).

See also:


Exploration

A rough idea would be something like this:

  • a config variable like $wgConfigSets in core. This config variable can be appended to by core (not sure where the logical place would be exactly) and by extensions and skins.
  • each configuration set has a key for its ID (a string, e.g. "growthexperiments-static-suggested-edits-module") and then an object containing:
    • path to ServiceWiring.php for the configuration set. The services defined for this configuration set will override existing services, rather than error on collision.
    • path to LocalSettingsOverride.php file. This file looks like a LocalSettings.php, used for overriding values set in LocalSettings.php/DefaultSettings.php
    • list of IDs of other configuration sets to use, these are evaluated before the configuration set where this list is defined. This way you can build multiple variants of an existing configuration setup without having to copy/paste a bunch of files/code.
    • description of what the configuration does. It would be nice to be able to browse or search the list via php maintenance/showConfigurationSets.php
  • With the configuration sets now defined, in LocalSettings.php, a developer can specify something like $wgActiveConfigSets = [ 'growthexperiments-static-suggested-edits-module' ];.
  • Now, for browser and API tests we need the ability to specify which configuration set(s) to make active in the context of a Selenium test or api-testing test.
  • We would define a feature flag like $wgAllowConfigSetsOverrideFromRequest with a default to false.
  • If it's set to true (which we'd do in Quibble's LocalSettings override in CI), then clients can send a header like X-Active-ConfigSets with a list of config sets to enable for a given request. This should also allow for parallel test execution of the Selenium/api-testing tests with different configurations.

Decision Statement Overview Artifact

n.b. canonical source is here: Decision Statement Overview, the following is copy/pasted so those who don't want to interact with Google Docs can review / comment.

What

Write your problem statement using layperson’s terminology. In one sentence, what is the problem or opportunity?

It is more difficult than it needs to be for less-technical users, new and experienced developers, and test writers to find, share, and keep up-to-date with configuration relevant to their needs, and we lack testing infrastructure to assess different configuration sets for end-to-end tests.

What does the future look like if this is achieved?

MediaWiki and extension + skins have configuration flags. These can be mixed and matched in multiple ways. There are default settings provided by core / extensions / skins, but these are not always developer friendly nor are they guaranteed to match what is in production.

Developers who want to set up core, extension, or skin to work on a particular feature need to manually update LocalSettings.php with various configuration snippets that can become out of date over time. Managing this is time-consuming, error-prone and results in frustration for newer and experienced developers.

In our continuous integration environment, automated tests run using the default configuration settings provided by core, extensions, and skins. That means that different configuration settings, including those used in production, are not tested.

Putting the above statements together: the problem is that it is too difficult for less-technical users, new developers, experienced developers, and test writers to find, share, and keep up-to-date with configuration relevant to their needs.

The proposal is to add infrastructure in MediaWiki core and convention in our community for defining configuration modes. A “mode” consists of an ID and a set of 1) ServiceWiring files, 2) configuration value overrides, 3) list of other configuration modes to inherit from.

Developers and test writers could then specify which configuration mode(s) they want active in their local environment. Developers would add the configuration modes they want active to a variable like $wgActiveConfigurationModes to LocalSettings.php; browser tests and API tests could use a cookie or request header to specify which configuration modes should be active for the particular request.

To encourage collaboration and discoverability, a maintenance script and/or Special page listing the available modes for core + extensions/skins could be created.

What happens if we do nothing?

The status quo: it remains difficult, time-consuming, and error-prone to set up and maintain one’s local environment for features provided by various skins and extensions and testing features in various configurations in end-to-end tests is difficult and mostly not done.

Why?

Identify the value(s) this problem/opportunity provides. Add links to relevant OKRs.

  • Platform Evolution
    • Engineering productivity easier for engineers to get different extensions/skins set up and configured correctly over time
    • Fully automated and continuous code health and deployment infrastructure automated testing for non-default (i.e. production) configuration settings. This project would also benefit the upcoming preview environments project.
    • Tooling for contributors is easy to use, well-documented, and accessible to users, increasing engagement and contribution easier for less-technical users to get started and make contributions
  • Thriving Movement
    • We will welcome and support newcomers easier for new users to get set up and begin contributing

Why are you bringing this decision to the Technical Forum?

Every team has had to find a solution to the problem of how to set up extension configuration, how to document it, and how to utilize that in tests. Input from this shared experience would result in a stronger technical proposal, and would also help with buy-in, because if teams did not use the proposed “configuration modes” they would not be useful.

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Krinkle renamed this task from RFC: Provide standard mechanism for overriding services and configuration in testing and development environments to RFC: Provide mechanism for overriding configuration for browser tests.

@Krinkle I would prefer to define this more broadly ("testing and development environments"), because I think we should implement a solution that, for example, also facilitates providing frontend engineers with a setup tuned specifically to allow them to work on a feature without requiring a bunch of additional services.

In any case, the testing aspect also applies to the api-testing framework, not just the Selenium tests.

This RFC will be discussed on IRC next Wednesday, November 25, at 22:00 UTC (2pm EST, 23:00 CET).
As always, the meeting will take place in #wikimedia-office channel on Freenode.

brennen moved this task from Backlog to Radar on the User-brennen board.
brennen added a subscriber: brennen.
kostajh renamed this task from RFC: Provide mechanism for overriding configuration for browser tests to RFC: Provide mechanism for defining and utilizing configuration sets for local development and browser / API-testing tests.Nov 23 2020, 10:19 AM
kostajh updated the task description. (Show Details)
kostajh added a subscriber: awight.

cc @awight about the parallel test execution bit in particular

Change 641156 abandoned by Kosta Harlan:
[mediawiki/core@master] [WIP] Add flag for specifying environment MediaWiki is running in

Reason:

https://gerrit.wikimedia.org/r/641156

Setting the environment mode should be possible via the HTTP request (or CLI command), perhaps using a cookie, a request header, request parameter, or an environment variable, so that the Selenium or api-testing framework does not necessarily need to be running on the same filesystem as MediaWiki, and to allow for parallel tests.

+1, and maybe we should emphasize this by phrasing as "The environment mechanism must not interfere with parallel tests or break isolation between tests. It must be possible for tests to run in separate environments."

I think there's some overlap between this proposal and T242835: RFC: Standard method for feature-management in skins/extensions, which is worthy of exploration. AIUI this task is restricted to local development and testing (CI and QA on the Beta Cluster?) but I can see how the outcome could also be added to in order to implement some of the features proposed in T242835, e.g. picking a config set based on whether the user is in a specific group or based in the last digit of their ID.

I think there's some overlap between this proposal and T242835: RFC: Standard method for feature-management in skins/extensions, which is worthy of exploration. AIUI this task is restricted to local development and testing (CI and QA on the Beta Cluster?) but I can see how the outcome could also be added to in order to implement some of the features proposed in T242835, e.g. picking a config set based on whether the user is in a specific group or based in the last digit of their ID.

Thanks @phuedx. I'll keep that in mind when adding notes to the document, which I hope to start doing this week.

Jenlenfantwright renamed this task from RFC: Provide mechanism for defining and utilizing configuration sets for local development and browser / API-testing tests to Configuration Modes.May 5 2021, 2:18 AM
Jenlenfantwright updated the task description. (Show Details)
Aklapper renamed this task from Configuration Modes to RFC: Provide mechanism for defining and utilizing configuration sets for local development and browser / API-testing tests.May 5 2021, 9:46 AM
Aklapper updated the task description. (Show Details)

@Jenlenfantwright: Hi, I've reverted your title change as it's unclear to me why it was made (and as I am not sure who you are).

@Jenlenfantwright: Hi, I've reverted your title change as it's unclear to me why it was made (and as I am not sure who you are).

@Aklapper, @Jenlenfantwright asked me if it was OK to rename the task, and I said it was fine. Jen is working as a project manager on the Tech Forum.

@Jenlenfantwright, @kostajh: Ah, I am sorry! It would be helpful if folks could self-identify in their Phabricator profile and/or user page on-wiki.

Jenlenfantwright renamed this task from RFC: Provide mechanism for defining and utilizing configuration sets for local development and browser / API-testing tests to Configuration Modes.May 5 2021, 12:43 PM
Jenlenfantwright claimed this task.
Jenlenfantwright updated the task description. (Show Details)

@Aklapper No worries! I have made the change again and added my title to my profile for future. Please let me know if you have any questions. Thanks!

How is one supposed to provide feedback on the decision statement when it's in Google Docs?

How someone not working in WMF can participate in the discussion? I work in WMDE and I don't have access to anything now while in legacy TechCom, most of the work were either in phabricator or IRC public meetings and I could participate at any RFC I wanted. How technical decision forum that explicitly stated is aiming to make the process more inclusive is going in the exact opposite direction?

How is one supposed to provide feedback on the decision statement when it's in Google Docs?

Do you mean Google Docs in general or this specific document which has restricted access? FWIW I've asked on this task and in other TechForum documents that the visibility for documents is public. It's unfortunate that the default is restricted access for work like this that should be publicly viewable & allow for comments / suggestions, IMO. The current document linked in the task description is outdated as well, since there's a new template as of a week or two ago. I've asked @Jenlenfantwright if they could link to the new one and set public access on it.

How someone not working in WMF can participate in the discussion? I work in WMDE and I don't have access to anything now while in legacy TechCom, most of the work were either in phabricator or IRC public meetings and I could participate at any RFC I wanted. How technical decision forum that explicitly stated is aiming to make the process more inclusive is going in the exact opposite direction?

https://www.mediawiki.org/wiki/Technical_Decision_Making_Process/Technical_Forum says that WMDE is supposed to have a representative (cc @WMDE-leszek) in this process.


I have my critiques & questions about the updated process too, but I'd prefer that discussion here is about the proposal in the task description, and discussion of the process should probably go into a separate phab ticket tagged with tech-decision-forum.

@kostajh @Ladsgroup The Google is doc linked to the updated completed template and has been set to public access for commenting as requested. Please let me know if you are unable to access it.

I support what @kostajh said at the end of his comment. General process related feedback please not here. @Ladsgroup I am happy to be receiver of your comments should you have any.

FWIW, WMDE has a representative in the Technical Decision Forum and does not for the time being seek any changes. There has been some misunderstanding here internally

How is one supposed to provide feedback on the decision statement when it's in Google Docs?

Do you mean Google Docs in general or this specific document which has restricted access?

Google Docs in general. I'm the sole maintainer of MediaWiki-Configuration in my limited free time and don't use any proprietary software to do so. I'm happy to provide feedback on what's written up in Phab or on-wiki, but I don't know if that's the latest or what's being proposed to the tech decision forum. In any case, I don't intend to accept any proposals/patches that are not developed openly.

Hi all, I wanted to leave some information here regarding feedback on the process, google docs, and volunteer involvement. Generally, you can find information on the process here. Forum membership here. Decisions are published here. To keep people informed, updates are posted here.

To speak to some of the specific comments questions raised in this thread:

  • For writing decisions, we standardized on GDocs and provide templates for writing proposals. We felt it was the best place to structure documents with real time collaboration feature. For publishing decisions, we standardized on publishing to MediaWiki.org as we felt it was the best place to document decisions.
  • For providing feedback on a proposal, staff members of WMF or WMDE should contact their team representative is listed on the Forum membership page linked above. Note: proposal feedback is collected on a per-team basis (not on an individual basis).
  • For general feedback on the process, we run a retrospective on every decision and make updates. Additionally, @Jenlenfantwright will be running a general retrospective with forum members later this month. If you have process feedback (like on our tooling), you should be sure to let your representative know so they can incorporate your thoughts when this retrospective is sent out.
  • For community/volunteer feedback. The Forum does not currently have representation from community members. We first set up the process for teams at WMF and WMDE and will be extending it to include community feedback. This was an intentional staging of the process. This reason was two-fold: we wanted to work out some kinks first before rolling out a community process, and we needed to unblock some important decisions within the WMF. Before launching the Forum, we had discussed this within Technology Department Management and with the Technical Engagement Team which led to the drafting of a community representation proposal. Feel free to provide feedback there. We did not schedule next steps during annual planning, but now that is wrapping up we will be prioritizing this soon. I don't have concrete dates yet, but there will be emails when it happens.

@Ladsgroup based on this and @WMDE-leszek's comment, I think we answered all questions on where and how to provide feedback. If not, lmk.

@Legoktm I hope the answers above give you clarity on why and how we are using Google docs and how to engage. As a staff member, you have access to the documents and I encourage you to participate with the process. If you still have issues here, you can elevate these concerns through your team's forum representative to propose changes to the process. As @kostajh said, discussion on this ticket is best kept to the proposal at hand. Lastly, you can participate in the community representation process when we kick that off. If these avenues aren't satisfactory, please reach out to me directly or through your manager to talk about this more.


Hope this helps clarify things for all - please reach out to me for follow up and can answer other questions.

(Adding @Keegan our other other Forum Chair to the discussion and @MNadrofsky to the thread since Platform Engineering is listed as the code steward on the maintainers page)

We are discussing the WMDE specific case internally but I want to emphasize being developed openly is a matter of principle and should not be addressed by specific cases. For example, we have volunteers who consciously don't have or use a google account (@Nemo_bis probably can tell more) and we actively excluding them by requiring Google accounts.

I added what is written in the Google Document into the task description, see the heading "Decision Statement Overview Artifact" and below.

@Legoktm I hope the answers above give you clarity on why and how we are using Google docs and how to engage. As a staff member, you have access to the documents and I encourage you to participate with the process. If you still have issues here, you can elevate these concerns through your team's forum representative to propose changes to the process. As @kostajh said, discussion on this ticket is best kept to the proposal at hand. Lastly, you can participate in the community representation process when we kick that off. If these avenues aren't satisfactory, please reach out to me directly or through your manager to talk about this more.

Well, like I said, I'm the sole maintainer of MediaWiki-Configuration in my limited free time. I'm not doing it in my staff capacity, which at least right now, has nothing to do with MW config. I'm happy to provide feedback on what's written up in Phab or on-wiki. All major changes to MediaWiki-Configuration have gone through some form of public drafting and consultation process, if not an RfC explicitly. Extension developers who aren't staff are regularly consulted and we haven't had a breaking change *at all* which is pretty neat I think. So I'm not interested in spending my free time using proprietary software and don't intend to ask anyone else to if they want to participate in MW development.

There's also the issue that Google unilaterally banned one of our trusted +2ers, excluding them from this process, but I guess that's off topic now.

I added what is written in the Google Document into the task description, see the heading "Decision Statement Overview Artifact" and below.

Thank you!

I'm curious if you could explain what the deficiencies are in https://www.mediawiki.org/wiki/Requests_for_comment/PlatformSettings.php that don't address your needs. I think it's awkward to use for non-packagers, but theoretically it has all the functionality being requested. Part of that is what/who should be responsible for maintaining and defining the configuration settings, I would think it makes sense for CI, Vagrant, MW-Docker, to do that rather than the extensions or core itself.

I'm not really sold on having different config modes for api-testing, selenium, PHPUnit, etc. I think it would get confusing pretty fast and just introduce unnecessary variance, people are then going to ask "why does this work by default in selenium but not api-testing?" I think we've done a decent amount of work in unifying the CI platform and experience and see this as potentially a step back.

That said, so far modes have been just "defined" by requiring DevelopmentSettings or PlatformSettings, I'm definitely +1 to having a formal definition of what mode MediaWiki is in.

I'm curious if you could explain what the deficiencies are in https://www.mediawiki.org/wiki/Requests_for_comment/PlatformSettings.php that don't address your needs. I think it's awkward to use for non-packagers, but theoretically it has all the functionality being requested. Part of that is what/who should be responsible for maintaining and defining the configuration settings, I would think it makes sense for CI, Vagrant, MW-Docker, to do that rather than the extensions or core itself.

PlatformSettings.php doesn't meet the needs of what I've described in the document:

  1. It doesn't stay up to date as the relevant configuration snippets for setting up an extension / skin change over time. Referencing a configuration mode ID provided by core/extension/skin would allow for maintainers to provide updates without the end user having to do anything besides git pull.
  2. because it's not in version control, it's not the kind of thing that is going to be socialized via regular code review processes.
  3. It doesn't allow for handling different requests with different configuration setups. Configuration modes would allow you to override service definitions and config value for individual or suites of Selenium/api-testing tests.
  4. Discoverability of various configuration setups is not good (e.g. copy/paste from hopefully not outdated pages on mediawiki.org)

I'm not really sold on having different config modes for api-testing, selenium, PHPUnit, etc.

That's not exactly what I'm proposing; it's more that you would allow suites of tests to use different configuration setups. For example, in https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GrowthExperiments/+/699705 I'm writing a test that assumes there is a "add link" structured task to interact with in a browser test; but in another browser test I would want to test what happens if the "add link" structured task is not defined.

I think it would get confusing pretty fast and just introduce unnecessary variance, people are then going to ask "why does this work by default in selenium but not api-testing?"

I don't see the "why does this work in selenium but not in api-testing" as a possible risk, but I do agree that it could be confusing unless there is good tooling and documentation for discovery of configuration modes that encourages reuse and makes it simple to define modes that do exactly what you want.

I think we've done a decent amount of work in unifying the CI platform and experience and see this as potentially a step back.

I agree that we have put effort into removing custom overrides (e.g. the old tests/selenium/LocalSettings.php file T199939: Audit tests/selenium/LocalSettings.php file aiming at possibly deprecating the feature) and I'm not sure if it is formal policy somewhere, but I have seen extensions aim to use configuration in extension.json as the production default. Overall, I think that's a good thing.

But, InitialiseSettings.php is still 27,502 lines long, and I'd love a way to be able to write front-end tests that can exercise the different configuration permutations that we see used on different wikis.

I just talked to @Joe about this, to better understand how the need that originally drove this (controlling config during testing) overlaps with the needs of mediawiki on kubernetes, and with how we manage configuration in production.

At the basic level, it seems to me like we want a mechanism in core that does the following:
based on one or more request headers, determine a set of configuration set to load, and merge these config sets to determine the config that is to be used for handling the request.

In production, the relevant header would be the Host header. For testing, we can invent X-MWMock or X-MWConfig that can be used to pull in the config or wiring we want for a request.

It's still not entirely clear to me where the mapping between headers and config would live. For the testing use case, this mapping would be fixed for each component (e.g. per extension). For the production use case, it might just live in LocaloSettings.php I suppose. This is (directly or indirectly) where "map host to config" is already being done in an ad hoc fashion for most wiki farms, including ours.

In ExtensionRegistry, we already have logic for merging multiple sets of config together. I think what we want is a similar mechanism, executing at a similar time, pulling in blocks of config based on request headers.

For example:

  • Host: fr.wikipedia.org could map to something like [ "$mwconfig/wikipedia.json", "$mwconfig/language/fr.json", "$mwconfig/site/frwiki.json" ];
  • X-Mock: DummyUsers could map to something like [ "tests/mock/DummyUsersWiring.php" ];
  • X-Config: PrivateWiki could map to something like [ "tests/config/PrivateWiki.php" ];

I'm curious if you could explain what the deficiencies are in https://www.mediawiki.org/wiki/Requests_for_comment/PlatformSettings.php that don't address your needs. I think it's awkward to use for non-packagers, but theoretically it has all the functionality being requested. Part of that is what/who should be responsible for maintaining and defining the configuration settings, I would think it makes sense for CI, Vagrant, MW-Docker, to do that rather than the extensions or core itself.

PlatformSettings.php doesn't meet the needs of what I've described in the document:

  1. It doesn't stay up to date as the relevant configuration snippets for setting up an extension / skin change over time. Referencing a configuration mode ID provided by core/extension/skin would allow for maintainers to provide updates without the end user having to do anything besides git pull.

How is this any different than the current status quo of configuration settings? Like, what ends up being the difference between $wgEnableFeatureFoo = true and $wgEnableConfigurationMode['foo'] = true;? If you have to set multiple $wg variables to enable a feature, shouldn't it just be refactored down to one setting?

  1. because it's not in version control, it's not the kind of thing that is going to be socialized via regular code review processes.

Sure it is? https://salsa.debian.org/mediawiki-team/mediawiki/-/blob/master/debian/PlatformSettings.php For MediaWiki-Docker/Vagrant/CI it would be in their respective Git repositories.

  1. It doesn't allow for handling different requests with different configuration setups. Configuration modes would allow you to override service definitions and config value for individual or suites of Selenium/api-testing tests.

Like I said earlier, I don't think the use case of selenium/api-testing really makes sense here. In PHPUnit and QUnit tests, you can already do $this->setMwGlobals() and mw.config.set() respectively, why not just add those options in for Selenium/api-testing?

  1. Discoverability of various configuration setups is not good (e.g. copy/paste from hopefully not outdated pages on mediawiki.org)

Agreed 100%. But IMO this is just a problem in general with our documentation ("bug 1") and it being manual instead of being drive by the code and automated (like the API docs are).

I'm not really sold on having different config modes for api-testing, selenium, PHPUnit, etc.

That's not exactly what I'm proposing; it's more that you would allow suites of tests to use different configuration setups. For example, in https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GrowthExperiments/+/699705 I'm writing a test that assumes there is a "add link" structured task to interact with in a browser test; but in another browser test I would want to test what happens if the "add link" structured task is not defined.

I looked at the patch, but I don't really follow, it just looks like it's enabling Parsoid?

I just talked to @Joe about this, to better understand how the need that originally drove this (controlling config during testing) overlaps with the needs of mediawiki on kubernetes, and with how we manage configuration in production.

At the basic level, it seems to me like we want a mechanism in core that does the following:
based on one or more request headers, determine a set of configuration set to load, and merge these config sets to determine the config that is to be used for handling the request.

In production, the relevant header would be the Host header. For testing, we can invent X-MWMock or X-MWConfig that can be used to pull in the config or wiring we want for a request.

We...already have this. It's SiteConfiguration and $wgConf, which allows you to vary configuration based on $wgDBname, which maps exactly to the Host header.

It's still not entirely clear to me where the mapping between headers and config would live. For the testing use case, this mapping would be fixed for each component (e.g. per extension). For the production use case, it might just live in LocaloSettings.php I suppose. This is (directly or indirectly) where "map host to config" is already being done in an ad hoc fashion for most wiki farms, including ours.

In ExtensionRegistry, we already have logic for merging multiple sets of config together. I think what we want is a similar mechanism, executing at a similar time, pulling in blocks of config based on request headers.

This is SiteConfiguration.

For example:

  • Host: fr.wikipedia.org could map to something like [ "$mwconfig/wikipedia.json", "$mwconfig/language/fr.json", "$mwconfig/site/frwiki.json" ];

How is this different from our current system of dblists, except that everything is in multiple files?

In production, the relevant header would be the Host header. For testing, we can invent X-MWMock or X-MWConfig that can be used to pull in the config or wiring we want for a request.

We...already have this. It's SiteConfiguration and $wgConf, which allows you to vary configuration based on $wgDBname, which maps exactly to the Host header.

The idea is to generalize this approach, so we can be used to switch between setups easily based on a different header (or an environment variable) for debugging and testing. That's the reason I personally want it, and I think it's also what Kosta has in mind. Then the idea came up that such a system would also help with containerized deployments in production. How and why exactly I still need to better understand.

For example:

  • Host: fr.wikipedia.org could map to something like [ "$mwconfig/wikipedia.json", "$mwconfig/language/fr.json", "$mwconfig/site/frwiki.json" ];

How is this different from our current system of dblists, except that everything is in multiple files?

If I understood @Joe correctly, for the production use case, it's desirable to have this in multiple files that can be mixed and matched.

For the testing use case, the difference is that these config snippets would not be specific to the site, but specific to test cases defined in extension repos.

Great to see this productive discussion! I've run against this problem more times than I care to tell, my team often wants to test both sides of a configuration from browser tests. This type of test is often needed for safely migrating major changes.

Our recent workaround is unsustainable: we've added cookie-based backdoors in production code, which overrides actual configuration to toggle logic. This is a known anti-pattern, but we've explored the options and it seems nothing better is possible given our current framework.

The outline given in the task description sounds good for my needs, thank you for sharing notes so far. I like that configuration might come down to a single variable such as MEDIAWIKI_ENV, which can be set from various layers.

I think that @Legoktm's simple "PlatformSettings.php" proposal is the perfect next step, this file could read cookies / an environment variable / etc. to override configuration, which can be customized for each use case and saves us from committing to a more sophisticated generalization of all possible needs. We can develop the necessary technology in a file injected by Quibble for example, and fine-tune the testing use cases.

In the longer term, we probably want a more structured and integrated mechanism, for example a wiki farm (such as the Wikimedia clusters) would be complex enough that it deserves to be implemented as a full extension rather than raw PHP files.