Page MenuHomePhabricator

Iteratively clean up wmf-config to be less dynamic and with smaller settings files (2022)
Closed, ResolvedPublic

Description

This is a subtask of T223602 with a less ambitious and more iterative approach to get us in a better shape and have a better sense of what's in front of us. This should leave us in a state that is essentially a cleansed and easy to maintain and comprehend version of the status quo, which the parent task can then tackle further as only focussing on data format and location, with the rest of all the edge cases and problems already sorted out.

Below is result of @Ladsgroup and @Krinkle chatting at MediaWIki Hackathon 2022:

Minimal scope
  • Keep things fast to edit.
  • Keep things fast to deploy.
  • Keep using (for now) the same underlying system for storing and loading configuration from MW core and contributor perspective.
  • Keep the same CI job and diff/preview features and confidence we get from that.
Minimal problem statement
  1. There should only one "right" place for a given setting.

Use case: Discovery, productivity, on-boarding, debugging.

  1. Make config files more machine-editable.

Use case: Automate wmf-config patch creation when creating new wikis.

  1. Remove most dynamic code and turn most such statements into pure data arrays.

Use case: Increase benefit of our CI diffConfig job. Currently it can't diff side-effects within CommonSettings.php and the other various complex PHP files.

Use case: Let CI assert there are no accidental overwrites or conflicts between the various wmf-config files that assign the same variable.

Use case: Prepare the repo such in the parent task we can figure out how we want to format and version this data for kubernetes, whether by injecting at pod launch time, or in some fast way from an image rebuild etc.

Today
  • /wmf-config/CommonSettings.php, the entry point from MW core Setup/LocalSettings, it essentially assigns $wgConf = require IS.php.
  • /wmf-config/InitialiseSettings.php, is the almost-pure-data wgConfig structured array.
  • /wmf-config/*.php, are mostly configuration but currently as executable statements. Most of these could be simple arrays in IS.php, but we keep them separate for easier browsing and editing, plus there's some wLoadExtension() statements.
  • /wmf-config/missing.php, not configuration. This is a web entry point used by multiversion (it is also referenced by dead code in CS.php, predating multiversion).
  • /wmf-config/etcd.php, defines functions, not configuration.
  • /wmf-config/import.php, defines functions, not configuration.
  • /wmf-config/profiler.php, preload init for PHP, not configuration.
  • /wmf-config/interwiki.php, configuration data assigned in CS.php.
  • /wmf-config/ProductionServices.php, configuration for Wikimedia\MWConfig\ServiceConfig, which is read in CS.php as $wmgAllServices = ServiceConfig::getInstance()->getAllServices();
Proposed outcome
  • /wmf-config/CommonSettings.php, entry point assigning $wgConf = core.php + ext-FlaggedRevs.php + IS.php.
  • /wmf-config/<component>.php, pure data in StaticSiteConfiguration format, e.g. core.php, ext-FlaggedRevs.php, etc, split out from IS.php.
  • /multiversion/missing.php, new place for missing.php.
  • /src/etc.php, new place for etc.php.
  • /src/import.php, new place for import.php.
  • /src/profiler.php, new place for profiler.php.
Steps to get there
  • Minor change to IS.php from function to pure static array, and update require in CommonSettings.php.
  • Create the first /wmf-config/<component>.php file with a handful of settings.
  • In CS.php, read assign IS.php + <component>.php.
  • Write PHPUnit test to assert there are no overlapping keys between these files.
  • Slowly move non-config from current wmf-config/*.php files to wmf-config/CommonSettings.php (e.g. wfLoadExtension calls).
  • Slowly move config from current wmf-config/*.php files to wmf-config/<component>.php.
  • Slowly move simple config from CS.php to a relevant wmf-config/component>.php file.

Details

SubjectRepoBranchLines +/-
operations/mediawiki-configmaster+3 -7
operations/mediawiki-configmaster+249 -236
operations/mediawiki-configmaster+949 -937
operations/mediawiki-configmaster+2 -0
operations/mediawiki-configmaster+0 -1 K
operations/mediawiki-configmaster+1 K -20
operations/mediawiki-configmaster+12 -0
operations/mediawiki-configmaster+149 -138
operations/mediawiki-configmaster+1 K -1 K
operations/mediawiki-configmaster+1 K -1 K
operations/mediawiki-configmaster+5 K -5 K
operations/mediawiki-configmaster+4 K -4 K
operations/mediawiki-configmaster+6 -14 K
operations/mediawiki-configmaster+321 -211
operations/mediawiki-configmaster+1 K -1 K
operations/mediawiki-configmaster+1 K -1 K
operations/mediawiki-configmaster+25 -27
operations/mediawiki-configmaster+0 -18
operations/mediawiki-configmaster+11 -4
operations/mediawiki-configmaster+9 -12
operations/mediawiki-configmaster+3 -3
operations/mediawiki-configmaster+21 -11
operations/mediawiki-configmaster+94 -101
operations/mediawiki-configmaster+262 -15
operations/mediawiki-configmaster+13 -14
operations/mediawiki-configmaster+21 -10
operations/mediawiki-configmaster+1 -12
operations/mediawiki-configmaster+1 -1
operations/mediawiki-configmaster+307 -295
operations/mediawiki-configmaster+3 -3
operations/mediawiki-configmaster+143 -243
operations/mediawiki-configmaster+29 -23
operations/mediawiki-configmaster+2 -0
Show related patches Customize query in gerrit

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Re: Tech News - What wording would you suggest as the content? I.e. What are we hoping local editors are happy to learn from within this information resource, and what kinds of actions are we expecting/encouraging them to take with it?

E.g. We could say something like

It is now possible to easily view [all? most? some? of] the configuration settings that apply to just one wiki, and to compare settings between two wikis if those settings are different. For example: All Esperanto Wikipedia settings, or settings that are different between the Spanish and Esperanto Wikipedias. Local communities may want to discuss and propose changes to their local settings. Details about each of the named settings can be found by searching MediaWiki.org.

Corrections/clarifications/overhauls welcome!

Most of configurations are visible. Private configuration (passwords, etc.) are not, some others as well are not there due to our tech debt but we will improve that.

Ok, added like so https://meta.wikimedia.org/wiki/Tech/News/2022/28 - Edits welcome within the next ~22 hours. Thanks!

Change 810147 merged by jenkins-bot:

[operations/mediawiki-config@master] multiversion: Factor out getTagsForWiki() for re-use

https://gerrit.wikimedia.org/r/810147

Change 807609 merged by jenkins-bot:

[operations/mediawiki-config@master] missing.php: Update docs and add test plan

https://gerrit.wikimedia.org/r/807609

Change 807610 merged by jenkins-bot:

[operations/mediawiki-config@master] multiversion: Move missing.php from wmf-config/ to /multiversion

https://gerrit.wikimedia.org/r/807610

Change 818648 had a related patch set uploaded (by Krinkle; author: Krinkle):

[operations/mediawiki-config@master] multiversion: Untangle MWConfigCacheGenerator from CS.php (1/2)

https://gerrit.wikimedia.org/r/818648

Change 818649 had a related patch set uploaded (by Krinkle; author: Krinkle):

[operations/mediawiki-config@master] multiversion: Untangle MWConfigCacheGenerator from CS.php (2/2)

https://gerrit.wikimedia.org/r/818649

Change 818651 had a related patch set uploaded (by Krinkle; author: Krinkle):

[operations/mediawiki-config@master] multiversion: Move labs-overrides responsibility to getStaticConfig()

https://gerrit.wikimedia.org/r/818651

Change 819203 had a related patch set uploaded (by Krinkle; author: Krinkle):

[operations/mediawiki-config@master] CirrusTest: Remove reference to 'unittest' realm

https://gerrit.wikimedia.org/r/819203

Change 819203 merged by jenkins-bot:

[operations/mediawiki-config@master] CirrusTest: Remove reference to 'unittest' realm

https://gerrit.wikimedia.org/r/819203

Change 818648 merged by jenkins-bot:

[operations/mediawiki-config@master] multiversion: Untangle MWConfigCacheGenerator from CS.php (1/2)

https://gerrit.wikimedia.org/r/818648

Change 818649 merged by jenkins-bot:

[operations/mediawiki-config@master] multiversion: Untangle MWConfigCacheGenerator from CS.php (2/2)

https://gerrit.wikimedia.org/r/818649

Change 818651 merged by jenkins-bot:

[operations/mediawiki-config@master] multiversion: Move labs-overrides responsibility to getStaticConfig()

https://gerrit.wikimedia.org/r/818651

Change 885064 had a related patch set uploaded (by Krinkle; author: Krinkle):

[operations/mediawiki-config@master] multiversion: Create dblist-manage command for easy add/delete

https://gerrit.wikimedia.org/r/885064

Change 885066 had a related patch set uploaded (by Krinkle; author: Krinkle):

[operations/mediawiki-config@master] multiversion: Remove getCachableMWConfig in favour of getConfigGlobals

https://gerrit.wikimedia.org/r/885066

Change 799272 merged by jenkins-bot:

[operations/mediawiki-config@master] Move CirrusSearch settings from IS.php to ext-CirrusSearch.php

https://gerrit.wikimedia.org/r/799272

Mentioned in SAL (#wikimedia-operations) [2023-02-01T11:14:13Z] <ladsgroup@deploy1002> Synchronized wmf-config/ext-CirrusSearch.php: Move CirrusSearch settings from IS.php to ext-CirrusSearch.php, part I (T308932) (duration: 07m 04s)

Mentioned in SAL (#wikimedia-operations) [2023-02-01T11:21:54Z] <ladsgroup@deploy1002> Synchronized multiversion/MWConfigCacheGenerator.php: Move CirrusSearch settings from IS.php to ext-CirrusSearch.php, part II (T308932) (duration: 07m 04s)

Mentioned in SAL (#wikimedia-operations) [2023-02-01T11:29:01Z] <ladsgroup@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Move CirrusSearch settings from IS.php to ext-CirrusSearch.php, part III (T308932) (duration: 06m 43s)

Next is Babel. I need to write a script to analyze IS.php and give the biggest prefixes to clean.

Change 887307 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/mediawiki-config@master] Migrate Babel config into its own file

https://gerrit.wikimedia.org/r/887307

Change 887307 merged by jenkins-bot:

[operations/mediawiki-config@master] Migrate Babel config into its own file

https://gerrit.wikimedia.org/r/887307

Mentioned in SAL (#wikimedia-operations) [2023-02-09T12:03:08Z] <ladsgroup@deploy1002> Synchronized wmf-config/ext-Babel.php: Move Babel settings from IS.php to ext-Babel.php, part I (T308932) (duration: 07m 06s)

Mentioned in SAL (#wikimedia-operations) [2023-02-09T12:10:16Z] <ladsgroup@deploy1002> Synchronized multiversion/MWConfigCacheGenerator.php: Move Babel settings from IS.php to ext-Babel.php, part II (T308932) (duration: 06m 40s)

Mentioned in SAL (#wikimedia-operations) [2023-02-09T12:17:27Z] <ladsgroup@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Move Babel settings from IS.php to ext-Babel.php, part III (T308932) (duration: 06m 47s)

Change 885064 merged by jenkins-bot:

[operations/mediawiki-config@master] multiversion: Create dblist-manage command for easy add/delete

https://gerrit.wikimedia.org/r/885064

Change 885066 merged by jenkins-bot:

[operations/mediawiki-config@master] multiversion: Remove getCachableMWConfig in favour of getConfigGlobals

https://gerrit.wikimedia.org/r/885066

Change 889557 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/mediawiki-config@master] Migrate EventLogging config into its own file

https://gerrit.wikimedia.org/r/889557

Change 889557 merged by jenkins-bot:

[operations/mediawiki-config@master] Migrate EventLogging config into its own file

https://gerrit.wikimedia.org/r/889557

Mentioned in SAL (#wikimedia-operations) [2023-02-16T16:04:10Z] <ladsgroup@deploy1002> Synchronized wmf-config/ext-EventLogging.php: Move EventLogging settings from IS.php to ext-EventLogging.php, part I (T308932) (duration: 07m 05s)

Mentioned in SAL (#wikimedia-operations) [2023-02-16T16:11:33Z] <ladsgroup@deploy1002> Synchronized multiversion/MWConfigCacheGenerator.php: Move EventLogging settings from IS.php to ext-EventLogging.php, part II (T308932) (duration: 06m 48s)

Mentioned in SAL (#wikimedia-operations) [2023-02-16T16:21:00Z] <ladsgroup@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Move EventLogging settings from IS.php to ext-EventLogging.php, part III (T308932) (duration: 06m 54s)

Change 885064 merged by jenkins-bot:

[operations/mediawiki-config@master] multiversion: Create dblist-manage command for easy add/delete

https://gerrit.wikimedia.org/r/885064

I've to admit I kind of liked the yaml files. They made it easier to bootstrap configuration for a new wiki (I copied yaml file of a similar new-ish wiki, and ran the composer script). With the new script, I have to figure out which dblists a new wiki should have and run a set of commands to get the same effect.

Can we agree on dblists a new typical wiki should have, and introduce a new comman, something like composer dblist-manage init newwiki wikipedia, which would add newwiki to dblists each Wikipedia usually is?

Change 890234 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/mediawiki-config@master] Move all of NS-related config out of IS.php to a dedicated file

https://gerrit.wikimedia.org/r/890234

Change 890234 merged by jenkins-bot:

[operations/mediawiki-config@master] Move all of NS-related config out of IS.php to a dedicated file

https://gerrit.wikimedia.org/r/890234

Change 885064 merged by jenkins-bot:

[operations/mediawiki-config@master] multiversion: Create dblist-manage command for easy add/delete

https://gerrit.wikimedia.org/r/885064

I've to admit I kind of liked the yaml files. They made it easier to bootstrap configuration for a new wiki (I copied yaml file of a similar new-ish wiki, and ran the composer script). With the new script, I have to figure out which dblists a new wiki should have and run a set of commands to get the same effect.

Oh no, I thought it would actually add more work for you because it's two steps :/

Can we agree on dblists a new typical wiki should have, and introduce a new comman, something like composer dblist-manage init newwiki wikipedia, which would add newwiki to dblists each Wikipedia usually is?

That's a great idea, I will try to do it (maybe a dedicated ticket?) if no one beats me to it.

Mentioned in SAL (#wikimedia-operations) [2023-02-20T08:54:32Z] <ladsgroup@deploy1002> Synchronized wmf-config/core-Namespaces.php: Move all of NS-related config out of IS.php to a dedicated file, part I (T308932) (duration: 16m 10s)

Mentioned in SAL (#wikimedia-operations) [2023-02-20T09:01:51Z] <ladsgroup@deploy1002> Synchronized multiversion/MWConfigCacheGenerator.php: Move all of NS-related config out of IS.php to a dedicated file, part II (T308932) (duration: 06m 47s)

Mentioned in SAL (#wikimedia-operations) [2023-02-20T09:09:26Z] <ladsgroup@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Move all of NS-related config out of IS.php to a dedicated file, part III (T308932) (duration: 06m 24s)

[...]

Can we agree on dblists a new typical wiki should have, and introduce a new comman, something like composer dblist-manage init newwiki wikipedia, which would add newwiki to dblists each Wikipedia usually is?

That's a great idea, I will try to do it (maybe a dedicated ticket?) if no one beats me to it.

Filled as T330059: Create composer manage-dblist init subcommand in operations/mediawiki-config. Thanks!

Change 890789 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/mediawiki-config@master] Move userrights related configs from IS.php to core-Permissions.php

https://gerrit.wikimedia.org/r/890789

Change 890789 merged by jenkins-bot:

[operations/mediawiki-config@master] Move userrights related configs from IS.php to core-Permissions.php

https://gerrit.wikimedia.org/r/890789

Mentioned in SAL (#wikimedia-operations) [2023-02-22T14:15:34Z] <ladsgroup@deploy1002> Synchronized wmf-config/core-Permissions.php: Move all of userrights config out of IS.php to a dedicated file, part I (T308932) (duration: 68m 38s)

Mentioned in SAL (#wikimedia-operations) [2023-02-22T14:23:08Z] <ladsgroup@deploy1002> Synchronized multiversion/MWConfigCacheGenerator.php: Move all of userrights config out of IS.php to a dedicated file, part II (T308932) (duration: 07m 01s)

Mentioned in SAL (#wikimedia-operations) [2023-02-22T14:30:03Z] <ladsgroup@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Move all of userrights config out of IS.php to a dedicated file, part III (T308932) (duration: 06m 16s)

Change 891565 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/mediawiki-config@master] Move more ns-related config out of InitialiseSettings

https://gerrit.wikimedia.org/r/891565

Change 891565 merged by jenkins-bot:

[operations/mediawiki-config@master] Move more ns-related config out of InitialiseSettings

https://gerrit.wikimedia.org/r/891565

Mentioned in SAL (#wikimedia-operations) [2023-02-23T15:15:00Z] <ladsgroup@deploy1002> Synchronized wmf-config/core-Namespaces.php: Move more ns-related config out of InitialiseSettings, part I (T308932) (duration: 07m 01s)

Mentioned in SAL (#wikimedia-operations) [2023-02-23T15:27:39Z] <ladsgroup@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Move more ns-related config out of InitialiseSettings, part II (T308932) (duration: 06m 35s)

The infrastructure for splitting IS.php is in place now and it has become around 40% of its original size. More cleanups should be done but it won't ever end. So I close this ticket and people can take over the rest.

The next candidates are:

  • Wikibase even with dedicated file is still one of the biggest extensions in IS.php
  • CX: around 500 lines
  • VisualEditor: Around 400
  • GrowthExperiments: around 600 but a lot of them can be moved to CS.php as they are just defaults. Maybe we need a dedicated file for configs common settings that is 1- Actually common settings 2- an array instead of programmatically written (unlike CS.php). Out of scope anyways.
  • MediaViewer: 400 lines
  • WikibaseQualityConstraints: 500 lines
  • MachineVision 300 lines.
  • MobileFrontend: 300 lines
  • Vector: 300 lines

..

Mentioned in SAL (#wikimedia-operations) [2023-02-27T12:55:35Z] <ladsgroup@deploy1002> ladsgroup: Completely get rid of responsiveimages removal, part I (T308932) synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet

Mentioned in SAL (#wikimedia-operations) [2023-02-27T13:30:07Z] <ladsgroup@deploy1002> sync-file aborted: Completely get rid of responsiveimages removal, part I (T308932) (duration: 44m 38s)

Change 895792 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/mediawiki-config@master] Move wgEventStreams settings into ext-EventStreamConfig.php

https://gerrit.wikimedia.org/r/895792

Change 895792 merged by jenkins-bot:

[operations/mediawiki-config@master] Move wgEventStreams settings into ext-EventStreamConfig.php

https://gerrit.wikimedia.org/r/895792

Change 895831 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/mediawiki-config@master] ext-EventStreamConfig.php - wgEventStreams lives here

https://gerrit.wikimedia.org/r/895831

Change 895832 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/mediawiki-config@master] wgEventStreams etc. - Remove duplicate configs after refactor

https://gerrit.wikimedia.org/r/895832

Change 895883 had a related patch set uploaded (by Zabe; author: Zabe):

[operations/mediawiki-config@master] noc: Publicly expose new setting files

https://gerrit.wikimedia.org/r/895883

Change 895883 merged by jenkins-bot:

[operations/mediawiki-config@master] noc: Publicly expose new setting files

https://gerrit.wikimedia.org/r/895883

Mentioned in SAL (#wikimedia-operations) [2023-03-08T23:49:58Z] <zabe@deploy2002> Finished scap: T308932 (duration: 07m 15s)

Change 895831 merged by jenkins-bot:

[operations/mediawiki-config@master] ext-EventStreamConfig.php - wgEventStreams lives here

https://gerrit.wikimedia.org/r/895831

Mentioned in SAL (#wikimedia-operations) [2023-03-09T10:29:36Z] <otto@deploy2002> Synchronized wmf-config/ext-EventStreamConfig.php: Step 1a: ext-EventStreamConfig.php - wgEventStreams lives here - T308932 (duration: 06m 43s)

Mentioned in SAL (#wikimedia-operations) [2023-03-09T10:39:36Z] <otto@deploy2002> Synchronized multiversion/MWConfigCacheGenerator.php: Step 1b: MWConfigCacheGenerator.php - load ext-EventStreamConfig.php - T308932 (duration: 06m 23s)

Change 895832 merged by jenkins-bot:

[operations/mediawiki-config@master] wgEventStreams etc. - Remove duplicate configs after refactor

https://gerrit.wikimedia.org/r/895832

Mentioned in SAL (#wikimedia-operations) [2023-03-09T10:50:55Z] <otto@deploy2002> Synchronized wmf-config/ext-EventLogging.php: Step 2a: ext-EventLogging.php - remove duplicate configs - T308932 (duration: 06m 32s)

Mentioned in SAL (#wikimedia-operations) [2023-03-09T11:00:25Z] <otto@deploy2002> Synchronized wmf-config/InitialiseSettings.php: Step 2b: InitialiseSettings.php - remove duplicate configs - T308932 (duration: 06m 37s)

Change 896121 had a related patch set uploaded (by Zabe; author: Zabe):

[operations/mediawiki-config@master] noc: Publicly expose EventStreamConfig settings

https://gerrit.wikimedia.org/r/896121

Change 896121 merged by jenkins-bot:

[operations/mediawiki-config@master] noc: Publicly expose EventStreamConfig settings

https://gerrit.wikimedia.org/r/896121

Mentioned in SAL (#wikimedia-operations) [2023-03-09T16:09:39Z] <zabe@deploy2002> Finished scap: T308932 (duration: 07m 19s)

Change 921599 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[operations/mediawiki-config@master] Migrate GrowthExperiments config to its own file

https://gerrit.wikimedia.org/r/921599

Change 921599 merged by jenkins-bot:

[operations/mediawiki-config@master] Migrate GrowthExperiments config to its own file

https://gerrit.wikimedia.org/r/921599

Mentioned in SAL (#wikimedia-operations) [2023-05-24T08:28:39Z] <urbanecm@deploy1002> Started scap: Backport for [[gerrit:921599|Migrate GrowthExperiments config to its own file (T308932)]]

Mentioned in SAL (#wikimedia-operations) [2023-05-24T08:35:59Z] <urbanecm@deploy1002> Finished scap: Backport for [[gerrit:921599|Migrate GrowthExperiments config to its own file (T308932)]] (duration: 07m 20s)

Change 1011189 had a related patch set uploaded (by Krinkle; author: Krinkle):

[operations/mediawiki-config@master] InitialiseSettings: Factor out ext-MobileFrontend.php to its own file

https://gerrit.wikimedia.org/r/1011189

Change #891733 had a related patch set uploaded (by Krinkle; author: Krinkle):

[operations/mediawiki-config@master] Move etcd.php from wmf-config/ to src/

https://gerrit.wikimedia.org/r/891733