Page MenuHomePhabricator

1.32.0-wmf.10 deployment blockers
Closed, ResolvedPublicRelease

Details

Release Version
1.32.0-wmf.10
Release Date
1529884800

2018 week 26 1.32-wmf.10 Changes wmf/1.32.0-wmf.10

This MediaWiki Train Deployment is scheduled for the week of Monday, June 25th:

Monday June 25thTuesday, June 26thWednesday, June 27thThursday, June 28thFriday
SWAT deployments only.Branch wmf.10 and deploy to Group 0 Wikis.Deploy wmf.10 to Group 1 Wikis.Deploy wmf.10 to all Wikis.No deployments on fridays

How this works

  • Any serious bugs affecting wmf.10 should be added as subtasks beneath this one.
  • Any open subtask(s) block the train from moving forward.This means no further deployments until the blockers are resolved.
  • If something is serious enough to warrant a rollback then you should bring it to the attention of deployers on the #wikimedia-operations IRC channel.
  • For more info about deployment blockers, see Holding the train.

Related Links

Other Deployments

Previous: 1.32.0-wmf.9
Next: 1.32.0-wmf.11

Event Timeline

greg created this task.Mar 29 2018, 6:08 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 29 2018, 6:08 PM
greg triaged this task as Normal priority.Mar 29 2018, 6:16 PM
greg assigned this task to dduvall.Jun 11 2018, 10:45 PM

Trace of l10update failure (see related task and SAL message):

17:59:05 Started scap: testwiki to php-1.32.0-wmf.10 and rebuild l10n cache
17:59:06 Copying to deploy1001.eqiad.wmnet from deploy1001.eqiad.wmnet
17:59:06 Started rsync common
cannot delete non-empty directory: php-1.31.0-wmf.28/cache/l10n
cannot delete non-empty directory: php-1.31.0-wmf.28/cache/l10n
cannot delete non-empty directory: php-1.31.0-wmf.28/cache
cannot delete non-empty directory: php-1.31.0-wmf.28/cache
cannot delete non-empty directory: php-1.31.0-wmf.28
17:59:28 Finished rsync common (duration: 00m 22s)
17:59:28 Started cache_git_info
18:00:03 Finished cache_git_info (duration: 00m 35s)
18:00:04 Started l10n-update
18:00:04 Updating ExtensionMessages-1.32.0-wmf.8.php
18:00:05 Updating LocalisationCache for 1.32.0-wmf.8 using 30 thread(s)
18:01:27 Generating JSON versions and md5 files
18:01:27 Creating empty /srv/mediawiki-staging/wmf-config/ExtensionMessages-1.32.0-wmf.10.php
18:01:27 Bootstrapping l10n cache for 1.32.0-wmf.10
18:01:28 Last output:
[Tue Jun 26 18:01:28 2018] [hphp] [433:7f2d762fa3c0:0:000001] []
Fatal error: Uncaught exception 'Exception' with message '/srv/mediawiki-staging/php-1.32.0-wmf.10/extensions/MwEmbedSupport/extension.json
 does not exist!' in /srv/mediawiki-staging/php-1.32.0-wmf.10/includes/registration/ExtensionRegistry.php:105
Stack trace:
#0 /srv/mediawiki-staging/php-1.32.0-wmf.10/includes/GlobalFunctions.php(50): ExtensionRegistry->queue()
#1 /srv/mediawiki/wmf-config/CommonSettings.php(904): wfLoadExtension()
#2 /srv/mediawiki-staging/php-1.32.0-wmf.10/LocalSettings.php(4): include()
#3 /srv/mediawiki-staging/php-1.32.0-wmf.10/includes/Setup.php(100): include()
#4 /srv/mediawiki-staging/php-1.32.0-wmf.10/maintenance/doMaintenance.php(81): include()
#5 /srv/mediawiki-staging/php-1.32.0-wmf.10/maintenance/rebuildLocalisationCache.php(181): include()
#6 /srv/mediawiki-staging/multiversion/MWScript.php(100): include()
#7 {main}
Stashbot added a subscriber: Stashbot.

Mentioned in SAL (#wikimedia-operations) [2018-06-26T18:39:41Z] <marxarelli> setting $wmgUseMwEmbedSupport = false in php-1.32.0-wmf.10/LocalSettings.php to extension registry exception (see T197918 and T191056)

Change 442163 had a related patch set uploaded (by Dduvall; owner: Jforrester):
[operations/mediawiki-config@master] Be able to over-ride wmgUseMwEmbedSupport from LocalSettings.php

https://gerrit.wikimedia.org/r/442163

Change 442163 merged by jenkins-bot:
[operations/mediawiki-config@master] Be able to over-ride wmgUseMwEmbedSupport from LocalSettings.php

https://gerrit.wikimedia.org/r/442163

Now cdb-rebuild failed on mwdebug2002 due to lack of disk space.

19:40:01 Failure processing (u'/srv/mediawiki/php-1.32.0-wmf.10/cache/l10n', u'l10n_cache-mt.cdb', True)
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/scap/tasks.py", line 501, in update_l10n_cdb_wrapper
    return update_l10n_cdb(*args)
  File "/usr/lib/python2.7/dist-packages/scap/utils.py", line 402, in context_wrapper
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/scap/tasks.py", line 477, in update_l10n_cdb
    writer.put(key.encode('utf-8'), value.encode('utf-8'))
  File "/usr/lib/python2.7/dist-packages/scap/cdblib.py", line 128, in put
    self.fp.write(value)
IOError: [Errno 28] No space left on device

Since the sync succeeded other than cdb update on mwdeploy2002, I'm going to move ahead with group0 today.

Manually cleaned up old deployments (wmf.3, wmf.4, wmf.5) to ensure enough disk space. I'm re-syncing now.

I think commons data loss occurring from Wednesday could be related to train? T198177

Current plan for today's train rollout is mentioned in {T198350#4322794}. It'll be a graduated rollout of (group1 - commonswiki), then commonswiki, then time for thorough verification that the issue is resolved, then group2 time permitting.