Page MenuHomePhabricator

Roll out loading of default settings via SettingsBuilder
Open, HighPublic

Description

One step away from doing everything in global variables is loading the config schema, and thus default values, via SettingsBuilder rather than including DefaultSettings.php in global scope.

This should be done carefully in WMF production, to avoid breakage and performance degradation. The following steps are needed:

  • Load the config schema via SettingsLoader if the MW_USE_CONFIG_SCHEMA environment variable is set. Otherwise, load DefaultSettings.php as before.
  • Collect timing information about loading deaults in Setup.php.
  • Make a dashboard for timing information about loading deaults in Setup.php. https://grafana.wikimedia.org/d/ge9ndts7z/config-loading-performance
  • Make MWMultiVersion set MW_USE_CONFIG_SCHEMA if the file /var/run/php/use-config-schema exists on the current host. https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/772937
  • create a use-config-schema file in the container that runs the beta sites and watch for errors.
  • create a use-config-schema file one a debug host and watch for errors.
  • create a use-config-schema file one a single host serving web requests, watch for errors, and observe timing metrics.
  • create a use-config-schema file hosts serving API requests and running jobs. Watch for errors, and observe timing metrics.
  • if all is good, set the variable unconditionally in MWMultiVersion. Watch for errors, and observe timing metrics.
  • if all is still good after a few days, remove the conditional from Setup.php https://gerrit.wikimedia.org/r/c/mediawiki/core/+/775896
  • Finally, after the above has been deployed everywhere for at least a week (!), remove the relevant code from MWMultiVersion.

Event Timeline

Change 772836 had a related patch set uploaded (by Daniel Kinzler; author: Daniel Kinzler):

[mediawiki/core@master] Emit timing stats about loading default config.

https://gerrit.wikimedia.org/r/772836

Change 772935 had a related patch set uploaded (by Daniel Kinzler; author: Daniel Kinzler):

[mediawiki/core@master] Support MW_USE_CONFIG_SCHEMA constant.

https://gerrit.wikimedia.org/r/772935

Change 772937 had a related patch set uploaded (by Daniel Kinzler; author: Daniel Kinzler):

[operations/mediawiki-config@master] Set MW_USE_CONFIG_SCHEMA constant of the file use-config-schema exists.

https://gerrit.wikimedia.org/r/772937

daniel triaged this task as High priority.Mar 23 2022, 10:23 AM
daniel moved this task from Ideas to Doing on the MediaWiki-SettingsBuilder board.

Change 772935 merged by jenkins-bot:

[mediawiki/core@master] Support MW_USE_CONFIG_SCHEMA constant.

https://gerrit.wikimedia.org/r/772935

I assume tag was unintentionally inherited from parent (ref T239378).

Reedy subscribed.
Reedy unsubscribed.

Change 772836 merged by jenkins-bot:

[mediawiki/core@master] Emit timing stats about loading default config.

https://gerrit.wikimedia.org/r/772836

daniel updated the task description. (Show Details)

Change 772937 merged by jenkins-bot:

[operations/mediawiki-config@master] Set MW_USE_CONFIG_SCHEMA constant if file exists.

https://gerrit.wikimedia.org/r/772937

Mentioned in SAL (#wikimedia-operations) [2022-03-31T07:30:29Z] <daniel@deploy1002> Synchronized multiversion/defines.php: Config: [[gerrit:772937|Set MW_USE_CONFIG_SCHEMA constant if file exists. (T304460)]] (duration: 00m 52s)

Mentioned in SAL (#wikimedia-operations) [2022-03-31T09:08:50Z] <duesen> created /var/run/php/use-config-schema on mwdebug1002 to enable config schema loading (T304460)

Mentioned in SAL (#wikimedia-operations) [2022-03-31T09:16:48Z] <duesen> created /var/run/php/use-config-schema on canary mw1415 to enable config schema loading (T304460)

Mentioned in SAL (#wikimedia-operations) [2022-03-31T09:25:01Z] <duesen> removed /var/run/php/use-config-schema from mwdebug1002 to disable config schema loading (T304460)

I enabled schema loading on mw1415 for ten minutes or so, to collect statistics. The box saw about 64 requests per second during this tim. The results are surprisingly good: loading the schema clocks in at about 0.05ms, while loading DefaultSettings.php takes nearly three times as long, about 0.140ms

Grafana-Config-Loading-Performance.png (1×1 px, 126 KB)

See https://grafana.wikimedia.org/d/ge9ndts7z/config-loading-performance?orgId=1

I will enabled this again for a longer period (and on more hosts) later to collect more stats.

Mentioned in SAL (#wikimedia-operations) [2022-03-31T14:22:13Z] <duesen> (late) about 5 hours ago, I removed /var/run/php/use-config-schema from mw1415 to disable config schema loading (T304460)

Mentioned in SAL (#wikimedia-operations) [2022-03-31T16:33:27Z] <duesen> creating /var/run/php/use-config-schema on canaries mw1415, mw1438, and mw1448 to enable config schema loading (T304460)

Mentioned in SAL (#wikimedia-operations) [2022-03-31T18:43:14Z] <duesen> removing /var/run/php/use-config-schema from canaries mw1415, mw1438, and mw1448 to disable config schema loading (T304460)

I tested this on mw1415, mw1438, and mw1448 for a couple of hourse. No errors showed up in the logs. The performance was as before: loading the defaults took < 0.05ms.

Grafana-Config-Loading-Performance.png (1×1 px, 147 KB)

Switched all servers to use the new code. Load time is about 0.05ms, down from about 0.2ms.

Config Loading Switch.png (1×1 px, 178 KB)