1.36.0-wmf.9 deployment blockers
Closed, ResolvedPublicRelease
Actions

Assigned To

Authored By

	• mmodell
	Jul 14 2020, 8:19 PM

Details

Backup Train Conductor: brennen
Release Version: 1.36.0-wmf.9
Release Date: Sep 14 2020, 12:00 AM

2020 week 38 1.36-wmf.9 Changes wmf/1.36.0-wmf.9

This MediaWiki Train Deployment is scheduled for the week of Monday, September 14th:

Monday September 14th	Tuesday, September 15th	Wednesday, September 16th	Thursday, September 17th	Friday
Backports only.	Branch `wmf.9` and deploy to Group 0 Wikis.	Deploy `wmf.9` to Group 1 Wikis.	Deploy `wmf.9` to all Wikis.	No deployments on fridays

See https://wikitech.wikimedia.org/wiki/Deployments for full schedule.

How this works

Any serious bugs affecting wmf.9 should be added as subtasks beneath this one.
- Use this form to create one.
Any open subtask(s) block the train from moving forward. This means no further deployments until the blockers are resolved.
If something is serious enough to warrant a rollback then you should bring it to the attention of deployers on the #wikimedia-operations IRC channel.
If you have a risky change in this week's train add a comment to this task using the Risky patch template
For more info about deployment blockers, see Holding the train.

Other Deployments

Previous: 1.36.0-wmf.8

Next: 1.36.0-wmf.10

	Subject	Repo	Branch	Lines +/-
	Branch commit for wmf/1.36.0-wmf.9	mediawiki/core	wmf/1.36.0-wmf.9	+966 -3

Customize query in gerrit

Related Objects
Search...

Status	Subtype	Assigned	Task
Resolved	Release	brennen	T257977 1.36.0-wmf.9 deployment blockers
Resolved	PRODUCTION ERROR	tstarling	T262900 Rebuilding l10n cache fails for train
Resolved		Ammarpad	T258001 Deprecate Skin methods that are aliases for footerLink
Resolved		• Gilles	T263047 Uncaught TypeError: Cannot read property 'node' of undefined

Event Timeline

• mmodell created this task.Jul 14 2020, 8:19 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 14 2020, 8:19 PM

thcipriani assigned this task to • LarsWirzenius.Aug 11 2020, 10:14 PM

thcipriani triaged this task as Medium priority.

thcipriani updated Other Assignee, added: brennen.

thcipriani added a project: Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)).

thcipriani moved this task from INBOX to Deployment on the Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)) board.

brennen added a project: User-brennen.Sep 8 2020, 4:24 PM

brennen subscribed.

Reedy mentioned this in T262213: XSS on Pages viewed on Mobile (CVE-2020-26120).Sep 8 2020, 6:53 PM

hashar subscribed.Sep 8 2020, 7:13 PM

brennen moved this task from Backlog to Next on the User-brennen board.Sep 9 2020, 7:16 PM

Wroth noting is that we now have git 2.2.20 on Stretch and git protocol version 2 enabled. That should not affect the train (beside slightly faster git fetches), but might have some unexpected one.

Change 627400 had a related patch set uploaded (by DannyS712; owner: trainbranchbot):
[mediawiki/core@wmf/1.36.0-wmf.9] Branch commit for wmf/1.36.0-wmf.9

https://gerrit.wikimedia.org/r/627400

gerritbot added a project: Patch-For-Review.Sep 15 2020, 2:07 AM

Change 627400 merged by jenkins-bot:
[mediawiki/core@wmf/1.36.0-wmf.9] Branch commit for wmf/1.36.0-wmf.9

https://gerrit.wikimedia.org/r/627400

Mentioned in SAL (#wikimedia-operations) [2020-09-15T06:57:45Z] <liw> 1.36.0-wmf.9 was branched at rMW7269b6b57b6f for T257977

Branch cut happened automatically by cron job, I CR+2'd it at 9:34 my time (UTC+3). https://gerrit.wikimedia.org/r/c/mediawiki/core/+/627400

I applied security patches and attempted to deploy to testwikis, around 20 minutes later. That failed (T262900: Rebuilding l10n cache fails for train), because scap failed to build l10n cache. Scap managed to sync to deploy1001, but nowhere else.

liw@deploy1001:/srv/mediawiki-staging$ scap sync "testwikis to 1.36.0-wmf.9"
           ___ ____
         ��
          \  //==--'
     _//|,.·//==--'    ____________________________
    _OO�=-  ︶ ᴹw �_§ ______  ___\ ___\ ,\__ \/ __ \
   (�)_, )  (     |  ______/__  \/ /__ / /_/ / /_/ /
     ¨--¨|| |- (  / ______\____/ \___/ \__^_/  .__/
         ««_/  «_/ jgs/bd808                /_/

07:54:38 Checking for new runtime errors locally
07:54:38 Started scap: testwikis to 1.36.0-wmf.9
07:54:38 Copying from deploy1001.eqiad.wmnet to deploy1001.eqiad.wmnet
07:54:39 Started rsync common
08:00:29 Finished rsync common (duration: 05m 50s)
08:00:29 Started cache_git_info
08:01:41 Finished cache_git_info (duration: 01m 12s)
08:01:41 Started l10n-update
08:01:41 Updating ExtensionMessages-1.36.0-wmf.8.php
08:01:42 Updating LocalisationCache for 1.36.0-wmf.8 using 30 thread(s)
08:05:08 Generating JSON versions and md5 files
08:05:48 Creating empty /srv/mediawiki-staging/wmf-config/ExtensionMessages-1.36.0-wmf.9.php
08:05:48 Bootstrapping l10n cache for 1.36.0-wmf.9
08:05:48 Last output:

followed by error messages.

Sent a "train status update" email to ops@ and wikitech-l@, but I don't get copies of emails I send to those so I don't know yet if it went through.

The wmf.9 being on deploy1001 only is causing alerts on #wikimedia-operations. Rolling back... fails.

l10n cache missing for 1.36.0-wmf.9: u'/srv/mediawiki-staging/php-1.36.0-wmf.9/cache/l10n/l10n_cache-en.cdb'

Same reason: no l10n cache.

This too failed:

liw@deploy1001:/srv/mediawiki-staging$ ~/release/bin/deploy-promote testwikis 1.36.0-wmf.8
Git repo /home/liw/release is not clean!

On branch master
Your branch is up to date with 'origin/master'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

     modified:   bin/deploy-promote

no changes added to commit (use "git add" and/or "git commit -a")

Continue anyway? [y/N] y
continuing...
Promote testwikis from 1.36.0-wmf.8 to 1.36.0-wmf.8 [y/N] y
09:29:40 Updated wikiversions.json: 3 inserted, 0 migrated.
09:29:40 Symlink updated
[master db94e0f37] testwikis wikis to 1.36.0-wmf.8
 1 file changed, 1 insertion(+), 1 deletion(-)
Enumerating objects: 9, done.
Counting objects: 100% (9/9), done.
Delta compression using up to 32 threads
Compressing objects: 100% (5/5), done. 
Writing objects: 100% (6/6), 685 bytes | 685.00 KiB/s, done.
Total 6 (delta 3), reused 0 (delta 0)  
remote: Resolving deltas: 100% (3/3)   
remote: Processing changes: refs: 1, done
remote: ERROR: commit 826cadf: missing Change-Id in message footer
remote:
remote: Hint: to automatically insert a Change-Id, install the hook:
remote:   gitdir=$(git rev-parse --git-dir); scp -p -P 29418 liw@gerrit.wikimedia.org:hooks/commit-msg ${gitdir}/ho
oks/
remote: and then amend the commit:
remote:   git commit --amend --no-edit 
remote: Finally, push your changes again
remote:
To ssh://gerrit.wikimedia.org:29418/operations/mediawiki-config
 ! [remote rejected]     HEAD -> refs/for/master%topic=1.36.0-wmf.8,l=Code-Review+2 (commit 826cadf: missing Change
-Id in message footer)
error: failed to push some refs to 'ssh://liw@gerrit.wikimedia.org:29418/operations/mediawiki-config'

(The uncommitted changes to deploy-promote is because I have a local change to use scap sync instead of scap sync-world.)

@LarsWirzenius one of the revert commit missed a Change-Id because git revert does not seem to trigger the git hook commit-msg which inserts the Change-Id header. I have edited it, you can push for review again ;)

In T257977#6462063, @hashar wrote:

@LarsWirzenius one of the revert commit missed a Change-Id because git revert does not seem to trigger the git hook commit-msg which inserts the Change-Id header. I have edited it, you can push for review again ;)

Thanks.

brennen moved this task from Next to Doing on the User-brennen board.Sep 15 2020, 3:12 PM

Over to you, Brennen, with my apologies and thanks.

Krinkle added a subtask: T258001: Deprecate Skin methods that are aliases for footerLink.Sep 15 2020, 7:49 PM

Krinkle reopened subtask T258001: Deprecate Skin methods that are aliases for footerLink as Open.

Jdlrobson closed subtask T258001: Deprecate Skin methods that are aliases for footerLink as Resolved.Sep 15 2020, 8:39 PM

End-of-local-day notes:

I fetched (but did not sync out) backports for T258001 to the wmf.9 branch on deploy1001.
Earlier I noticed an extraneous /srv/mediawiki-staging/php-1.36.0-wmf.9/php-1.36.0-wmf.8, a symlink to itself. I thought this might be a symptom of some other breakage, but there's nothing obvious. Deleted.

We're now past the deploy cutoff for today, handing back to Lars for the European morning window, assuming a resolution to T262900.

T262900: Rebuilding l10n cache fails for train is supposed to be fixed, though not closed yet. Backported by @hashar. Trying to deploy to testwikis.

This week's train is powered by Assam, Lapsang souchong, and Creme caramel tea, and the official soundtrack is Scooter's "Mind the gap". Just for the record.