Page MenuHomePhabricator

1.36.0-wmf.9 deployment blockers
Closed, ResolvedPublicRelease


Release Version
Release Date
Mon, Sep 14, 12:00 AM
Backup Conductor

2020 week 38 1.36-wmf.9 Changes wmf/1.36.0-wmf.9

This MediaWiki Train Deployment is scheduled for the week of Monday, September 14th:

Monday September 14thTuesday, September 15thWednesday, September 16thThursday, September 17thFriday
Backports only.Branch wmf.9 and deploy to Group 0 Wikis.Deploy wmf.9 to Group 1 Wikis.Deploy wmf.9 to all Wikis.No deployments on fridays

How this works

  • Any serious bugs affecting wmf.9 should be added as subtasks beneath this one.
  • Any open subtask(s) block the train from moving forward.This means no further deployments until the blockers are resolved.
  • If something is serious enough to warrant a rollback then you should bring it to the attention of deployers on the #wikimedia-operations IRC channel.
  • For more info about deployment blockers, see Holding the train.

Related Links

Other Deployments

Previous: 1.36.0-wmf.8
Next: 1.36.0-wmf.10

Event Timeline

mmodell created this task.Jul 14 2020, 8:19 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 14 2020, 8:19 PM
thcipriani triaged this task as Medium priority.
thcipriani updated Backup Conductor, added: brennen.
brennen added a subscriber: brennen.
hashar added a subscriber: hashar.Tue, Sep 8, 7:13 PM
brennen moved this task from Backlog to Next on the User-brennen board.Wed, Sep 9, 7:16 PM

Wroth noting is that we now have git 2.2.20 on Stretch and git protocol version 2 enabled. That should not affect the train (beside slightly faster git fetches), but might have some unexpected one.

Change 627400 had a related patch set uploaded (by DannyS712; owner: trainbranchbot):
[mediawiki/core@wmf/1.36.0-wmf.9] Branch commit for wmf/1.36.0-wmf.9

Change 627400 merged by jenkins-bot:
[mediawiki/core@wmf/1.36.0-wmf.9] Branch commit for wmf/1.36.0-wmf.9

Branch cut happened automatically by cron job, I CR+2'd it at 9:34 my time (UTC+3).

I applied security patches and attempted to deploy to testwikis, around 20 minutes later. That failed (T262900: Rebuilding l10n cache fails for train), because scap failed to build l10n cache. Scap managed to sync to deploy1001, but nowhere else.

liw@deploy1001:/srv/mediawiki-staging$ scap sync "testwikis to 1.36.0-wmf.9"
           ___ ____
          \  //==--'
     _//|,.·//==--'    ____________________________
    _OO�=-  ︶ ᴹw �_§ ______  ___\ ___\ ,\__ \/ __ \
   (�)_, )  (     |  ______/__  \/ /__ / /_/ / /_/ /
     ¨--¨|| |- (  / ______\____/ \___/ \__^_/  .__/
         ««_/  «_/ jgs/bd808                /_/

07:54:38 Checking for new runtime errors locally
07:54:38 Started scap: testwikis to 1.36.0-wmf.9
07:54:38 Copying from deploy1001.eqiad.wmnet to deploy1001.eqiad.wmnet
07:54:39 Started rsync common
08:00:29 Finished rsync common (duration: 05m 50s)
08:00:29 Started cache_git_info
08:01:41 Finished cache_git_info (duration: 01m 12s)
08:01:41 Started l10n-update
08:01:41 Updating ExtensionMessages-1.36.0-wmf.8.php
08:01:42 Updating LocalisationCache for 1.36.0-wmf.8 using 30 thread(s)
08:05:08 Generating JSON versions and md5 files
08:05:48 Creating empty /srv/mediawiki-staging/wmf-config/ExtensionMessages-1.36.0-wmf.9.php
08:05:48 Bootstrapping l10n cache for 1.36.0-wmf.9
08:05:48 Last output:

followed by error messages.

Sent a "train status update" email to ops@ and wikitech-l@, but I don't get copies of emails I send to those so I don't know yet if it went through.

The wmf.9 being on deploy1001 only is causing alerts on #wikimedia-operations. Rolling back... fails.

l10n cache missing for 1.36.0-wmf.9: u'/srv/mediawiki-staging/php-1.36.0-wmf.9/cache/l10n/l10n_cache-en.cdb'

Same reason: no l10n cache.

LarsWirzenius added a comment.EditedTue, Sep 15, 9:31 AM

This too failed:

liw@deploy1001:/srv/mediawiki-staging$ ~/release/bin/deploy-promote testwikis 1.36.0-wmf.8
Git repo /home/liw/release is not clean!

On branch master
Your branch is up to date with 'origin/master'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

     modified:   bin/deploy-promote

no changes added to commit (use "git add" and/or "git commit -a")

Continue anyway? [y/N] y
Promote testwikis from 1.36.0-wmf.8 to 1.36.0-wmf.8 [y/N] y
09:29:40 Updated wikiversions.json: 3 inserted, 0 migrated.
09:29:40 Symlink updated
[master db94e0f37] testwikis wikis to 1.36.0-wmf.8
 1 file changed, 1 insertion(+), 1 deletion(-)
Enumerating objects: 9, done.
Counting objects: 100% (9/9), done.
Delta compression using up to 32 threads
Compressing objects: 100% (5/5), done. 
Writing objects: 100% (6/6), 685 bytes | 685.00 KiB/s, done.
Total 6 (delta 3), reused 0 (delta 0)  
remote: Resolving deltas: 100% (3/3)   
remote: Processing changes: refs: 1, done
remote: ERROR: commit 826cadf: missing Change-Id in message footer
remote: Hint: to automatically insert a Change-Id, install the hook:
remote:   gitdir=$(git rev-parse --git-dir); scp -p -P 29418 ${gitdir}/ho
remote: and then amend the commit:
remote:   git commit --amend --no-edit 
remote: Finally, push your changes again
To ssh://
 ! [remote rejected]     HEAD -> refs/for/master%topic=1.36.0-wmf.8,l=Code-Review+2 (commit 826cadf: missing Change
-Id in message footer)
error: failed to push some refs to 'ssh://'

(The uncommitted changes to deploy-promote is because I have a local change to use scap sync instead of scap sync-world.)

@LarsWirzenius one of the revert commit missed a Change-Id because git revert does not seem to trigger the git hook commit-msg which inserts the Change-Id header. I have edited it, you can push for review again ;)

@LarsWirzenius one of the revert commit missed a Change-Id because git revert does not seem to trigger the git hook commit-msg which inserts the Change-Id header. I have edited it, you can push for review again ;)


brennen moved this task from Next to In Progress on the User-brennen board.Tue, Sep 15, 3:12 PM
LarsWirzenius added a subscriber: LarsWirzenius.

Over to you, Brennen, with my apologies and thanks.

brennen reassigned this task from brennen to LarsWirzenius.Tue, Sep 15, 10:28 PM

End-of-local-day notes:

  • I fetched (but did not sync out) backports for T258001 to the wmf.9 branch on deploy1001.
  • Earlier I noticed an extraneous /srv/mediawiki-staging/php-1.36.0-wmf.9/php-1.36.0-wmf.8, a symlink to itself. I thought this might be a symptom of some other breakage, but there's nothing obvious. Deleted.

We're now past the deploy cutoff for today, handing back to Lars for the European morning window, assuming a resolution to T262900.

T262900: Rebuilding l10n cache fails for train is supposed to be fixed, though not closed yet. Backported by @hashar. Trying to deploy to testwikis.

LarsWirzenius added a comment.EditedWed, Sep 16, 10:07 AM

This week's train is powered by Assam, Lapsang souchong, and Creme caramel tea, and the official soundtrack is Scooter's "Mind the gap". Just for the record.

Deployment to testwikis worked. Going for group0 now.

We are at group0 now.

Train promoted to group1.

Over to you, with thanks.

brennen reassigned this task from brennen to LarsWirzenius.Wed, Sep 16, 11:27 PM

End-of-day here, passing back to Lars.

Promoted to group2 about 15 minutes ago. Filed T263128: ResponseException from line 56 of extensions/GeoData/includes/Searcher.php: but not yet as a blocker.

@brennen is taking over for the rest of Thursday. Thanks.

brennen closed this task as Resolved.Fri, Sep 18, 3:20 PM
brennen moved this task from In Progress to Done / Defunct on the User-brennen board.