Page MenuHomePhabricator

1.36.0-wmf.4 deployment blockers
Closed, ResolvedPublicRelease

Details

Release Version
1.36.0-wmf.4
Release Date
Aug 10 2020, 12:00 AM
Backup Conductor
mmodell

2020 week 33 1.36-wmf.4 Changes wmf/1.36.0-wmf.4

This MediaWiki Train Deployment is scheduled for the week of Monday, August 10th:

Monday August 10thTuesday, August 11thWednesday, August 12thThursday, August 13thFriday
Backports only.Branch wmf.4 and deploy to Group 0 Wikis.Deploy wmf.4 to Group 1 Wikis.Deploy wmf.4 to all Wikis.No deployments on fridays

How this works

  • Any serious bugs affecting wmf.4 should be added as subtasks beneath this one.
  • Any open subtask(s) block the train from moving forward.This means no further deployments until the blockers are resolved.
  • If something is serious enough to warrant a rollback then you should bring it to the attention of deployers on the #wikimedia-operations IRC channel.
  • For more info about deployment blockers, see Holding the train.

Related Links

Other Deployments

Previous: 1.36.0-wmf.3
Next: 1.36.0-wmf.5

Event Timeline

mmodell created this task.Jul 14 2020, 8:19 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 14 2020, 8:19 PM
thcipriani assigned this task to hashar.Aug 7 2020, 6:44 PM
thcipriani triaged this task as Medium priority.
thcipriani updated Backup Conductor, added: mmodell.

Change 619397 had a related patch set uploaded (by DannyS712; owner: trainbranchbot):
[mediawiki/core@wmf/1.36.0-wmf.4] Branch commit for wmf/1.36.0-wmf.4

https://gerrit.wikimedia.org/r/619397

Change 619397 merged by jenkins-bot:
[mediawiki/core@wmf/1.36.0-wmf.4] Branch commit for wmf/1.36.0-wmf.4

https://gerrit.wikimedia.org/r/619397

Mentioned in SAL (#wikimedia-operations) [2020-08-11T13:12:36Z] <hashar> Applied 1.36.0-wmf.4 security patches # T257972

Had to abort the deployment to testwiki due to some disk space issue (T260145, T260146). I am not running scap sync to ensure everything is in sync.

status

  • We had to clear out very old mediawiki versions from the servers since at least mwdebug1001 went out of disk space (no user impact). Fixed now.
  • A lot of Icinga alarms when I aborted sync wikiversions. Thewikiversions.json had different versions for the testswikis, hardly a big issue afaik.
  • Some faulty patch in Minervaneue got promptly reverted

Logstash is all clear for now so that sounds promising.

Next steps:

  • Wed, 12 Aug, EU Train Window, 13:00 UTC: Group1
  • Wed 12 Aug, US Train Window, 19:00 UTC: All Wikis

Change 619613 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/mediawiki-config@master] Revert "group1 wikis to 1.36.0-wmf.4"

https://gerrit.wikimedia.org/r/619613

Change 619613 merged by jenkins-bot:
[operations/mediawiki-config@master] Revert "group1 wikis to 1.36.0-wmf.4"

https://gerrit.wikimedia.org/r/619613

Blocked on MediaWiki application servers going out of memory. Private report is https://docs.google.com/document/d/1x1sXbklz98fBTKMBsv4fnbeAedG0RMjGEFJ8mrvD454/edit#

Went with sysctl -w vm/compact_memory=1 which should workaround the issue. SRE stated we can resume.

[13:27:13] <logmsgbot> !log hashar@deploy1001 Synchronized php: group1 wikis to 1.36.0-wmf.4 (duration: 01m 16s)
hashar closed this task as Resolved.Aug 12 2020, 7:57 PM

Looks like it is a success ! :]