Page MenuHomePhabricator
Create Task
Maniphest T308068

1.39.0-wmf.15 deployment blockers
Closed, ResolvedPublicRelease
Actions

Details

Backup Train Conductor
jeena
Release Version
1.39.0-wmf.15
Release Date
Jun 6 2022, 12:00 AM

2022 week 23 1.39-wmf.15 Changes wmf/1.39.0-wmf.15

This MediaWiki Train Deployment is scheduled for the week of Monday, June 6th:

Monday June 6thTuesday, June 7thWednesday, June 8thThursday, June 9thFriday
Backports only.Branch wmf.15 and deploy to Group 0 Wikis.Deploy wmf.15 to Group 1 Wikis.Deploy wmf.15 to all Wikis.No deployments on fridays

How this works

  • Any serious bugs affecting wmf.15 should be added as subtasks beneath this one.
  • Any open subtask(s) block the train from moving forward. This means no further deployments until the blockers are resolved.
  • If something is serious enough to warrant a rollback then you should bring it to the attention of deployers on the #wikimedia-operations IRC channel.
  • If you have a risky change in this week's train add a comment to this task using the Risky patch template
  • For more info about deployment blockers, see Holding the train.

Related Links

Other Deployments

Previous: 1.39.0-wmf.14
Next: 1.39.0-wmf.16
SubjectRepoBranchLines +/-
all wikis to 1.39.0-wmf.15operations/mediawiki-configmaster+323 -323
group1 wikis to 1.39.0-wmf.15operations/mediawiki-configmaster+529 -529
group0 wikis to 1.39.0-wmf.15operations/mediawiki-configmaster+128 -128
group1 wikis to 1.39.0-wmf.15operations/mediawiki-configmaster+529 -529
group0 wikis to 1.39.0-wmf.15operations/mediawiki-configmaster+128 -128
logstash_checker.py: Change `--delay` argument type to floatoperations/puppetproduction+2 -2
testwikis wikis to 1.39.0-wmf.15operations/mediawiki-configmaster+4 -4
Customize query in gerrit

Related Objects
Search...

Event Timeline

thcipriani created this task.May 10 2022, 9:53 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 10 2022, 9:53 PM
thcipriani assigned this task to jeena.May 10 2022, 10:07 PM
thcipriani triaged this task as Medium priority.
thcipriani updated Other Assignee, added: dancy.
thcipriani added a project: Release-Engineering-Team (Priority Backlog 📥).
thcipriani updated Other Assignee, added: • dduvall; removed: dancy.May 24 2022, 3:33 PM
thcipriani reassigned this task from jeena to • dduvall.May 31 2022, 5:10 PM
thcipriani updated Other Assignee, added: jeena; removed: • dduvall.
thcipriani added a subscriber: jeena.
Jdforrester-WMF added subscribers: matthiasmullie, Jdforrester-WMF.Jun 3 2022, 5:41 PM
Risky Patch! 🚂🔥
gerritbot added a comment.Jun 7 2022, 4:04 PM

Change 803561 had a related patch set uploaded (by Dduvall; author: Dduvall):

[operations/mediawiki-config@master] testwikis wikis to 1.39.0-wmf.15

https://gerrit.wikimedia.org/r/803561

gerritbot added a project: Patch-For-Review.Jun 7 2022, 4:04 PM

Change 803561 merged by jenkins-bot:

[operations/mediawiki-config@master] testwikis wikis to 1.39.0-wmf.15

https://gerrit.wikimedia.org/r/803561

Stashbot added a comment.Jun 7 2022, 4:06 PM

Mentioned in SAL (#wikimedia-operations) [2022-06-07T16:06:55Z] <dduvall@deploy1002> Started scap: testwikis wikis to 1.39.0-wmf.15 refs T308068

Stashbot added a comment.Jun 7 2022, 4:32 PM

Mentioned in SAL (#wikimedia-operations) [2022-06-07T16:32:07Z] <dduvall> scap deploy-promote testwikis failed at invocation of logstash_checker.py ("logstash_checker.py: error: argument --delay: invalid int value: '40.406498670578'") T308068

• dduvall added a subtask: T310089: The `logstash_checker.py` script should accept a float for `--delay`.Jun 7 2022, 4:55 PM
Lucas_Werkmeister_WMDE subscribed.Jun 7 2022, 4:57 PM
gerritbot added a comment.Jun 7 2022, 5:00 PM

Change 803583 had a related patch set uploaded (by Dduvall; author: Dduvall):

[operations/puppet@production] logstash_checker.py: Change `--delay` argument type to float

https://gerrit.wikimedia.org/r/803583

gerritbot added a comment.Jun 7 2022, 5:07 PM

Change 803583 merged by Jbond:

[operations/puppet@production] logstash_checker.py: Change `--delay` argument type to float

https://gerrit.wikimedia.org/r/803583

Stashbot added a comment.Jun 7 2022, 5:13 PM

Mentioned in SAL (#wikimedia-operations) [2022-06-07T17:13:04Z] <dduvall@deploy1002> Started scap: testwikis wikis to 1.39.0-wmf.15 refs T308068

• dduvall closed subtask T310089: The `logstash_checker.py` script should accept a float for `--delay` as Resolved.Jun 7 2022, 5:31 PM
Stashbot added a comment.Jun 7 2022, 5:43 PM

Mentioned in SAL (#wikimedia-operations) [2022-06-07T17:43:26Z] <dduvall@deploy1002> Finished scap: testwikis wikis to 1.39.0-wmf.15 refs T308068 (duration: 30m 22s)

gerritbot added a comment.Jun 7 2022, 6:01 PM

Change 803591 had a related patch set uploaded (by Dduvall; author: Dduvall):

[operations/mediawiki-config@master] group0 wikis to 1.39.0-wmf.15

https://gerrit.wikimedia.org/r/803591

gerritbot added a comment.Jun 7 2022, 6:02 PM

Change 803591 merged by jenkins-bot:

[operations/mediawiki-config@master] group0 wikis to 1.39.0-wmf.15

https://gerrit.wikimedia.org/r/803591

Stashbot added a comment.Jun 7 2022, 6:06 PM

Mentioned in SAL (#wikimedia-operations) [2022-06-07T18:06:30Z] <dduvall@deploy1002> rebuilt and synchronized wikiversions files: group0 wikis to 1.39.0-wmf.15 refs T308068

• dduvall added a subtask: T310096: DomainException: HTMLForm::getField: no field named discussiontools-newtopictool.Jun 7 2022, 6:42 PM
Zabe added a subtask: T310053: DiscussionTools throws an exception in preferences..Jun 7 2022, 6:52 PM
matmarex closed subtask T310053: DiscussionTools throws an exception in preferences. as Resolved.Jun 7 2022, 9:10 PM
gerritbot added a comment.Jun 8 2022, 6:12 PM

Change 803967 had a related patch set uploaded (by Dduvall; author: Dduvall):

[operations/mediawiki-config@master] group1 wikis to 1.39.0-wmf.15

https://gerrit.wikimedia.org/r/803967

gerritbot added a comment.Jun 8 2022, 6:13 PM

Change 803967 merged by jenkins-bot:

[operations/mediawiki-config@master] group1 wikis to 1.39.0-wmf.15

https://gerrit.wikimedia.org/r/803967

Stashbot added a comment.Jun 8 2022, 6:17 PM

Mentioned in SAL (#wikimedia-operations) [2022-06-08T18:17:27Z] <dduvall@deploy1002> rebuilt and synchronized wikiversions files: group1 wikis to 1.39.0-wmf.15 refs T308068

Stashbot added a comment.Jun 8 2022, 6:20 PM

Mentioned in SAL (#wikimedia-operations) [2022-06-08T18:20:56Z] <dduvall@deploy1002> Synchronized php: group1 wikis to 1.39.0-wmf.15 refs T308068 (duration: 03m 25s)

• dduvall added a subtask: T310214: Wikimedia\Rdbms\DBQueryError: Error 1146: Table 'enwikinews.`categorylinks`' doesn't exist.Jun 8 2022, 6:30 PM
• dduvall added a subtask: T310216: Search backend error during sending 1 documents to the commonswiki_file_ [...] primary shard is not active.Jun 8 2022, 6:42 PM
gerritbot added a comment.Jun 9 2022, 5:48 PM

Change 804403 had a related patch set uploaded (by Dduvall; author: Dduvall):

[operations/mediawiki-config@master] group0 wikis to 1.39.0-wmf.15

https://gerrit.wikimedia.org/r/804403

gerritbot added a comment.Jun 9 2022, 5:49 PM

Change 804403 merged by jenkins-bot:

[operations/mediawiki-config@master] group0 wikis to 1.39.0-wmf.15

https://gerrit.wikimedia.org/r/804403

Stashbot added a comment.Jun 9 2022, 5:53 PM

Mentioned in SAL (#wikimedia-operations) [2022-06-09T17:53:34Z] <dduvall@deploy1002> rebuilt and synchronized wikiversions files: group0 wikis to 1.39.0-wmf.15 refs T308068

gerritbot added a comment.Jun 9 2022, 6:26 PM

Change 804420 had a related patch set uploaded (by Dduvall; author: Dduvall):

[operations/mediawiki-config@master] group1 wikis to 1.39.0-wmf.15

https://gerrit.wikimedia.org/r/804420

gerritbot added a comment.Jun 9 2022, 6:27 PM

Change 804420 merged by jenkins-bot:

[operations/mediawiki-config@master] group1 wikis to 1.39.0-wmf.15

https://gerrit.wikimedia.org/r/804420

Stashbot added a comment.Jun 9 2022, 6:31 PM

Mentioned in SAL (#wikimedia-operations) [2022-06-09T18:31:33Z] <dduvall@deploy1002> rebuilt and synchronized wikiversions files: group1 wikis to 1.39.0-wmf.15 refs T308068

Stashbot added a comment.Jun 9 2022, 6:35 PM

Mentioned in SAL (#wikimedia-operations) [2022-06-09T18:35:07Z] <dduvall@deploy1002> Synchronized php: group1 wikis to 1.39.0-wmf.15 refs T308068 (duration: 03m 34s)

Agusbou2015 subscribed.Jun 9 2022, 6:44 PM
gerritbot added a comment.Jun 9 2022, 6:50 PM

Change 804425 had a related patch set uploaded (by Dduvall; author: Dduvall):

[operations/mediawiki-config@master] all wikis to 1.39.0-wmf.15

https://gerrit.wikimedia.org/r/804425

gerritbot added a comment.Jun 9 2022, 6:51 PM

Change 804425 merged by jenkins-bot:

[operations/mediawiki-config@master] all wikis to 1.39.0-wmf.15

https://gerrit.wikimedia.org/r/804425

Stashbot added a comment.Jun 9 2022, 6:56 PM

Mentioned in SAL (#wikimedia-operations) [2022-06-09T18:56:00Z] <dduvall@deploy1002> rebuilt and synchronized wikiversions files: all wikis to 1.39.0-wmf.15 refs T308068

• dduvall closed this task as Resolved.Jun 9 2022, 8:01 PM
• dduvall closed subtask T310214: Wikimedia\Rdbms\DBQueryError: Error 1146: Table 'enwikinews.`categorylinks`' doesn't exist as Resolved.
JMeybohm reopened this task as Open.Jun 10 2022, 3:52 PM
JMeybohm added subscribers: Krinkle, JMeybohm.

I see mcrouter get+gets increased significantly since the rollout to group2: https://grafana-rw.wikimedia.org/d/RIA1lzDZk/application-servers-red-dashboard?orgId=1&from=1654530600000&to=1654876200000&viewPanel=26
Is that something that is expected? (cc @Krinkle )

thcipriani added a comment.Jun 13 2022, 5:37 PM
In T308068#7995056, @JMeybohm wrote:

I see mcrouter get+gets increased significantly since the rollout to group2: https://grafana-rw.wikimedia.org/d/RIA1lzDZk/application-servers-red-dashboard?orgId=1&from=1654530600000&to=1654876200000&viewPanel=26
Is that something that is expected? (cc @Krinkle )

Filed as T310532: Investigate McRouter GET request spike from wmf.15 and blocked this week's train until there's some investigation so we don't make something unintentional even worse.

Krinkle mentioned this in T310532: Investigate McRouter GET request spike from wmf.15.Jun 15 2022, 1:03 AM
• dduvall closed this task as Resolved.Jun 28 2022, 4:12 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL