Change Details

Arising from James's previous musing, and discussions at the 2019 Hackathon. # What * InitialiseSettings.php (and much of CommonSettings.php) is replaced with per-wiki inheritable YAML files (to allow comments). * Actually-variant config goes into a much slimmer CommonSettings.php (or re-worked to not vary). * On merge, the YAML files are converted into one JSON file per wiki, for the currently-deployed version(s) of MW, which are stored in git. * This replaces the opportunistic cache in /tmp that we current have. Inheritance tree: allwikis.yaml | Default values for all wikis (e.g. wgNamespacesWithSubpages which is over-ridden, or wgEnableCanonicalServerLink which isn't) | +- wikipedias.yaml | Standard values for Wikipedias, where they differ from defaults (e.g. wgSitename or the fallback logo) and special inheritances | +- dewiki.yaml   Bespoke values for the German Wikipedia (e.g. the logo, or FlaggedRevisions configuration) and other special inheritances # Comparison: | Task | Current situation | Future state | | --- | --- | --- | | Config authored in | InitialiseSettings.php | `wikipedias.yaml` //etc.// | | Config build step | Runtime cache, in `/tmp/` | Build time static file, in `/srv/mediawiki/` | | mw-config merge | Trivial rebase | Full production build of on JSON static file per wiki | | Config read step | From cache or computed live | Always read from built static file | # Pros * Variant configuration will be static, making it more plausible to inject into docker images. * It will be much clearer exactly which wikis' config is changing, so deployers have more confidence. * YAML configuration files explicitly set the inheritance pattern. * Easier to compare one wiki's config with another's (//e.g.// "how different is dewiki from frwiki?"). * Clear when the rump of CommonSettings refers to undefined variables; variant config forced to be merged first. # Cons * Merging is harder (and slower?). * Harder to audit all wikis' config for settings that "shouldn't" be over-ridden, or see how values vary. * Production branch pruning, currently just a disc operation and a sync, now needs a commit to mw-config as well as a deploy to delete. * ~~First time we're reading YAML files in PHP prod.~~ We're not reading them in prod, only in CI. # Former questions * Deterministic sort of output files to avoid noise. ** Assuming that alphasort of the array by keys (`ksort`) sufficient. * How do we do splicing in private settings at run time? ** Private settings are already spliced in in CommonSettings; no change. * Syntax for specifying config, and that a document inherits from another. ** Roughly worked out; to be documented. * Syntax for specifying that descendent config can't over-ride (//e.g.// `wgContentHandlerUseDB`) ** For now, this is just a simple `all.yaml` file that is re-applied at the end and so can't be over-ridden. * Do we need to vary on the PHP run-time still? (once HHVM is un-deployed can this go away, or are there reasons beyond PHP serialisation format that we think this might vary?) ** No. Nothing has been variant between HHVM and Zend for a while. No reason to continue. # Open questions * How does the CI work for this? * Do we need to check on build time that a vanilla MediaWiki install (//i.e.//, DefaultSettings) doesn't set any config that isn't represented in `all.yaml`? * What do we do about variant non-static config? # Planned steps [x] Write cached config to JSON as well as serialised PHP https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/533592/ [x] Read cached config from JSON not serialised PHP https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/533593/ [x] Stop writing cached config in serialised PHP https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/533594/ [x] Write YAML files reader and hook into CI [x] Migrate dblist definition into the YAML structure and make the dblist files auto-generated [ ] Write "InitialiseSettings.json" JSON file writer and hook into CI [ ] Migrate initial settings from InitialiseSettings.php to YAML [ ] Completely move all static variant config into YAML; scrap dblists and InitialiseSettings.php [ ] Pre-calculate config for each wiki and store it in config.git https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/507729/ [ ] Make scap steps to change target for each wiki update cached config in config.git, not just wikiversions.json [ ] Load config from config.git on disc, not from /tmp

Arising from James's previous musing, and discussions at the 2019 Hackathon. # What * InitialiseSettings.php (and much of CommonSettings.php) is replaced with per-wiki inheritable YAML files (to allow comments). * Actually-variant config goes into a much slimmer CommonSettings.php (or re-worked to not vary). * On merge, the YAML files are converted into one JSON file per wiki, for the currently-deployed version(s) of MW, which are stored in git. * This replaces the opportunistic cache in /tmp that we current have. Inheritance tree: allwikis.yaml | Default values for all wikis (e.g. wgNamespacesWithSubpages which is over-ridden, or wgEnableCanonicalServerLink which isn't) | +- wikipedias.yaml | Standard values for Wikipedias, where they differ from defaults (e.g. wgSitename or the fallback logo) and special inheritances | +- dewiki.yaml   Bespoke values for the German Wikipedia (e.g. the logo, or FlaggedRevisions configuration) and other special inheritances # Comparison: | Task | Current situation | Future state | | --- | --- | --- | | Config authored in | InitialiseSettings.php | `wikipedias.yaml` //etc.// | | Config build step | Runtime cache, in `/tmp/` | Build time static file, in `/srv/mediawiki/` | | mw-config merge | Trivial rebase | Full production build of on JSON static file per wiki | | Config read step | From cache or computed live | Always read from built static file | # Pros * Variant configuration will be static, making it more plausible to inject into docker images. * It will be much clearer exactly which wikis' config is changing, so deployers have more confidence. * YAML configuration files explicitly set the inheritance pattern. * Easier to compare one wiki's config with another's (//e.g.// "how different is dewiki from frwiki?"). * Clear when the rump of CommonSettings refers to undefined variables; variant config forced to be merged first. # Cons * Merging is harder (and slower?). * Harder to audit all wikis' config for settings that "shouldn't" be over-ridden, or see how values vary. * Production branch pruning, currently just a disc operation and a sync, now needs a commit to mw-config as well as a deploy to delete. * ~~First time we're reading YAML files in PHP prod.~~ We're not reading them in prod, only in CI. # Former questions * Deterministic sort of output files to avoid noise. ** Assuming that alphasort of the array by keys (`ksort`) sufficient. * How do we do splicing in private settings at run time? ** Private settings are already spliced in in CommonSettings; no change. * Syntax for specifying config, and that a document inherits from another. ** Roughly worked out; to be documented. * Syntax for specifying that descendent config can't over-ride (//e.g.// `wgMiserMode`) ** For now, this is just a simple `all.yaml` file that is re-applied at the end and so can't be over-ridden. * Do we need to vary on the PHP run-time still? (once HHVM is un-deployed can this go away, or are there reasons beyond PHP serialisation format that we think this might vary?) ** No. Nothing has been variant between HHVM and Zend for a while. No reason to continue. # Open questions * How does the CI work for this? * Do we need to check on build time that a vanilla MediaWiki install (//i.e.//, DefaultSettings) doesn't set any config that isn't represented in `all.yaml`? * What do we do about variant non-static config? # Planned steps [x] Write cached config to JSON as well as serialised PHP https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/533592/ [x] Read cached config from JSON not serialised PHP https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/533593/ [x] Stop writing cached config in serialised PHP https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/533594/ [x] Write YAML files reader and hook into CI [x] Migrate dblist definition into the YAML structure and make the dblist files auto-generated [ ] Write "InitialiseSettings.json" JSON file writer and hook into CI [ ] Migrate initial settings from InitialiseSettings.php to YAML [ ] Completely move all static variant config into YAML; scrap dblists and InitialiseSettings.php [ ] Pre-calculate config for each wiki and store it in config.git https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/507729/ [ ] Make scap steps to change target for each wiki update cached config in config.git, not just wikiversions.json [ ] Load config from config.git on disc, not from /tmp

Arising from James's previous musing, and discussions at the 2019 Hackathon. # What * InitialiseSettings.php (and much of CommonSettings.php) is replaced with per-wiki inheritable YAML files (to allow comments). * Actually-variant config goes into a much slimmer CommonSettings.php (or re-worked to not vary). * On merge, the YAML files are converted into one JSON file per wiki, for the currently-deployed version(s) of MW, which are stored in git. * This replaces the opportunistic cache in /tmp that we current have. Inheritance tree: allwikis.yaml | Default values for all wikis (e.g. wgNamespacesWithSubpages which is over-ridden, or wgEnableCanonicalServerLink which isn't) | +- wikipedias.yaml | Standard values for Wikipedias, where they differ from defaults (e.g. wgSitename or the fallback logo) and special inheritances | +- dewiki.yaml   Bespoke values for the German Wikipedia (e.g. the logo, or FlaggedRevisions configuration) and other special inheritances # Comparison: | Task | Current situation | Future state | | --- | --- | --- | | Config authored in | InitialiseSettings.php | `wikipedias.yaml` //etc.// | | Config build step | Runtime cache, in `/tmp/` | Build time static file, in `/srv/mediawiki/` | | mw-config merge | Trivial rebase | Full production build of on JSON static file per wiki | | Config read step | From cache or computed live | Always read from built static file | # Pros * Variant configuration will be static, making it more plausible to inject into docker images. * It will be much clearer exactly which wikis' config is changing, so deployers have more confidence. * YAML configuration files explicitly set the inheritance pattern. * Easier to compare one wiki's config with another's (//e.g.// "how different is dewiki from frwiki?"). * Clear when the rump of CommonSettings refers to undefined variables; variant config forced to be merged first. # Cons * Merging is harder (and slower?). * Harder to audit all wikis' config for settings that "shouldn't" be over-ridden, or see how values vary. * Production branch pruning, currently just a disc operation and a sync, now needs a commit to mw-config as well as a deploy to delete. * ~~First time we're reading YAML files in PHP prod.~~ We're not reading them in prod, only in CI. # Former questions * Deterministic sort of output files to avoid noise. ** Assuming that alphasort of the array by keys (`ksort`) sufficient. * How do we do splicing in private settings at run time? ** Private settings are already spliced in in CommonSettings; no change. * Syntax for specifying config, and that a document inherits from another. ** Roughly worked out; to be documented. * Syntax for specifying that descendent config can't over-ride (//e.g.// `wgContentHandlerUseDB`MiserMode`) ** For now, this is just a simple `all.yaml` file that is re-applied at the end and so can't be over-ridden. * Do we need to vary on the PHP run-time still? (once HHVM is un-deployed can this go away, or are there reasons beyond PHP serialisation format that we think this might vary?) ** No. Nothing has been variant between HHVM and Zend for a while. No reason to continue. # Open questions * How does the CI work for this? * Do we need to check on build time that a vanilla MediaWiki install (//i.e.//, DefaultSettings) doesn't set any config that isn't represented in `all.yaml`? * What do we do about variant non-static config? # Planned steps [x] Write cached config to JSON as well as serialised PHP https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/533592/ [x] Read cached config from JSON not serialised PHP https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/533593/ [x] Stop writing cached config in serialised PHP https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/533594/ [x] Write YAML files reader and hook into CI [x] Migrate dblist definition into the YAML structure and make the dblist files auto-generated [ ] Write "InitialiseSettings.json" JSON file writer and hook into CI [ ] Migrate initial settings from InitialiseSettings.php to YAML [ ] Completely move all static variant config into YAML; scrap dblists and InitialiseSettings.php [ ] Pre-calculate config for each wiki and store it in config.git https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/507729/ [ ] Make scap steps to change target for each wiki update cached config in config.git, not just wikiversions.json [ ] Load config from config.git on disc, not from /tmp