Page MenuHomePhabricator

Deployment to change wgFragmentMode not consistent on all app servers
Closed, ResolvedPublic

Description

2017-12-01

  • 00:13 <maxsem@tin> Synchronized wmf-config/InitialiseSettings.php: HTML5 sections be upon us! (duration: 00m 45s)

This deployed rOMWC28a6ee5fc0b0: Switch all wikis to HTML5 section IDs.

As expected, one minute later, ResourceLoader module mediawiki.util was invalidated as its version hash changed. (It exports the value of wgFragmentMode to JavaScript).

However, about three hours later, the module started to change again, and again, and again.

#wikimedia-perf-bots
# 00:14 UTC (below time is in SF)
16:14 <•perflogbot> mediawiki.util: 14kc4ko => 000pyx0 (after 10 days)
# 03:42 UTC (below time is in SF)
19:42 <•perflogbot> mediawiki.util: 000pyx0 => 14kc4ko (after 3 hours)
19:42 <•perflogbot> mediawiki.util: 14kc4ko => 000pyx0 (after a few seconds)
19:42 <•perflogbot> mediawiki.util: 000pyx0 => 14kc4ko (after a few seconds)
19:47 <•perflogbot> mediawiki.util: 14kc4ko => 000pyx0 (after 5 minutes)
20:28 <•perflogbot> mediawiki.util: 000pyx0 => 14kc4ko (after 42 minutes)
20:29 <•perflogbot> mediawiki.util: 14kc4ko => 000pyx0 (after a few seconds)
20:29 <•perflogbot> mediawiki.util: 000pyx0 => 14kc4ko (after a few seconds)
20:33 <•perflogbot> mediawiki.util: 14kc4ko => 000pyx0 (after 4 minutes)

Note that the "other" version is in fact the old one. This strongly suggests an app server is pooled that did not receive the configuration change.

Module in question can be accessed at https://en.wikipedia.org/w/load.php?debug=false&lang=en&modules=mediawiki.util, which shows:

mw.loader.implement( "mediawiki.util@000pyx0", function($,jQuery,require,module){
  mw.config.set({"wgFragmentMode": ["html5","legacy"] // html5 first (new)

Bypassing cache various times I do indeed actively get the old version still from certain backend servers.

mw.loader.implement( "mediawiki.util@14kc4ko", function($,jQuery,require,module){
  mw.config.set({"wgFragmentMode": ["legacy","html5"] });( // legacy first (old)

At least the following app servers are affected:

  • server: mw1220.eqiad.wmnet
  • server: mw1327.eqiad.wmnet

Checking the raw file shows the correct content, actually:

krinkle@mw1220:~$ cat /srv/mediawiki/wmf-config/InitialiseSettings.php | grep -A4 wgFragment
'wgFragmentMode' => [
	'default' => [ 'html5', 'legacy' ], // new
],

krinkle@mw1327:~$ cat /srv/mediawiki/wmf-config/InitialiseSettings.php | grep -A4 wgFragment
'wgFragmentMode' => [
	'default' => [ 'html5', 'legacy' ], // new
],

But, hacking through absence of mwscript via hhvm, shows that it does indeed have a stale impression:

krinkle@mw1327:~$ . /etc/profile.d/mediawiki.sh
krinkle@mw1327:~$ sudo -u "$MEDIAWIKI_WEB_USER" hhvm "$MEDIAWIKI_DEPLOYMENT_DIR/multiversion/MWScript.php" eval.php --wiki enwiki
> var_dump($wgFragmentMode);
array(2) {
  [0]=>
  string(6) "legacy" // old!
  [1]=>
  string(5) "html5"
}

Event Timeline

[21:14:15] <logmsgbot> !log legoktm@tin Synchronized wmf-config/InitialiseSettings.php: touch (duration: 00m 44s)

legoktm@mw1220:~$ sudo -u "$MEDIAWIKI_WEB_USER" hhvm "$MEDIAWIKI_DEPLOYMENT_DIR/multiversion/MWScript.php" eval.php --wiki enwiki
> var_dump($wgFragmentMode);
array(2) {
  [0]=>
  string(5) "html5"
  [1]=>
  string(6) "legacy"
}
legoktm@mw1327:~$ sudo -u "$MEDIAWIKI_WEB_USER" hhvm "$MEDIAWIKI_DEPLOYMENT_DIR/multiversion/MWScript.php" eval.php --wiki enwiki
> var_dump($wgFragmentMode);
array(2) {
  [0]=>
  string(5) "html5"
  [1]=>
  string(6) "legacy"
}
Legoktm claimed this task.

Marking as resolved because perflogbot in IRC hasn't complained about this module since I touched it.