Page MenuHomePhabricator
Create Task
Maniphest T305215

1.39.0-wmf.9 deployment blockers
Closed, ResolvedPublic5 Estimated Story PointsRelease
Actions

Details

Backup Train Conductor
jeena
Release Version
1.39.0-wmf.9
Release Date
Apr 25 2022, 12:00 AM

2022 week 17 1.39-wmf.9 Changes wmf/1.39.0-wmf.9

This MediaWiki Train Deployment is scheduled for the week of Monday, April 25th:

Monday April 25thTuesday, April 26thWednesday, April 27thThursday, April 28thFriday
Backports only.Branch wmf.9 and deploy to Group 0 Wikis.Deploy wmf.9 to Group 1 Wikis.Deploy wmf.9 to all Wikis.No deployments on fridays

How this works

  • Any serious bugs affecting wmf.9 should be added as subtasks beneath this one.
  • Any open subtask(s) block the train from moving forward. This means no further deployments until the blockers are resolved.
  • If something is serious enough to warrant a rollback then you should bring it to the attention of deployers on the #wikimedia-operations IRC channel.
  • If you have a risky change in this week's train add a comment to this task using the Risky patch template
  • For more info about deployment blockers, see Holding the train.

Related Links

Other Deployments

Previous: 1.39.0-wmf.8
Next: 1.39.0-wmf.10

Related Objects

Event Timeline

thcipriani created this task.Apr 1 2022, 12:51 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 1 2022, 12:51 AM
thcipriani assigned this task to brennen.Apr 24 2022, 11:20 PM
thcipriani triaged this task as Medium priority.
thcipriani updated Other Assignee, added: jeena.
thcipriani set the point value for this task to 5.
thcipriani added a project: Release-Engineering-Team (🌱 Spring Cleaning — April 2022).
Ladsgroup subscribed.Apr 25 2022, 6:49 AM
dcausse subscribed.EditedApr 25 2022, 7:45 AM

removed the risky patch section I added because testing on the beta cluster showed some problems so we reverted it.

AlexisJazz subscribed.EditedApr 26 2022, 11:37 AM

T306882 might be something. (if I'm at the wrong version again please copy this comment to somewhere else)

Urbanecm added a subtask: T306882: internal_api_error_TypeError (Argument 1 passed to MediaWiki\Auth\Throttler::__construct() must be of the type array or null).Apr 26 2022, 11:44 AM
Urbanecm added a subtask: T306878: TypeError: Argument 1 passed to MediaWiki\Auth\Throttler::__construct() must be of the type array or null, integer given .Apr 26 2022, 11:48 AM
Urbanecm removed a subtask: T306882: internal_api_error_TypeError (Argument 1 passed to MediaWiki\Auth\Throttler::__construct() must be of the type array or null).
Urbanecm removed a subtask: T306878: TypeError: Argument 1 passed to MediaWiki\Auth\Throttler::__construct() must be of the type array or null, integer given .Apr 26 2022, 12:07 PM
AlexisJazz unsubscribed.Apr 26 2022, 12:39 PM
TheDJ subscribed.Apr 26 2022, 1:57 PM
Stashbot added a comment.Apr 26 2022, 3:40 PM

Mentioned in SAL (#wikimedia-releng) [2022-04-26T15:40:00Z] <brennen> train 1.39.0-wmf.9 (T305215): no current blockers - expect to start train ops after the toolhub deployment window wraps, so some time after 17:00 UTC; taking a pre-train stroll-around-the-block break before that.

gerritbot added a comment.Apr 26 2022, 4:43 PM

Change 786349 had a related patch set uploaded (by Brennen Bearnes; author: Brennen Bearnes):

[operations/mediawiki-config@master] testwikis wikis to 1.39.0-wmf.9 refs T305215

https://gerrit.wikimedia.org/r/786349

gerritbot added a project: Patch-For-Review.Apr 26 2022, 4:43 PM

Change 786349 merged by jenkins-bot:

[operations/mediawiki-config@master] testwikis wikis to 1.39.0-wmf.9 refs T305215

https://gerrit.wikimedia.org/r/786349

Stashbot added a comment.Apr 26 2022, 4:44 PM

Mentioned in SAL (#wikimedia-operations) [2022-04-26T16:44:41Z] <brennen@deploy1002> Started scap: testwikis wikis to 1.39.0-wmf.9 refs T305215

Stashbot added a comment.Apr 26 2022, 4:48 PM

Mentioned in SAL (#wikimedia-operations) [2022-04-26T16:48:20Z] <brennen@deploy1002> Started scap: testwikis wikis to 1.39.0-wmf.9 refs T305215

brennen added a comment.Apr 26 2022, 4:59 PM

(Above double-sync due to having forgotten a SCAP=scap while using a local checkout of scap.)

brennen added a subscriber: Dzahn.Apr 26 2022, 5:18 PM

Had some hosts time out during deploy-promote --yes testwikis 1.39.0-wmf.9:

P26581 2022-04-26 train timeouts
116:53:39 brennen@deploy1002 ~ $ grep 'Timeout, server' ./1.39.0-wmf.9.log
2Timeout, server mw2263.codfw.wmnet not responding.
3Timeout, server mw1390.eqiad.wmnet not responding.
4Timeout, server snapshot1010.eqiad.wmnet not responding.
5Timeout, server mw1413.eqiad.wmnet not responding.
6Timeout, server mw1400.eqiad.wmnet not responding.
7Timeout, server mw1372.eqiad.wmnet not responding.
8Timeout, server mw1393.eqiad.wmnet not responding.
9Timeout, server wtp1032.eqiad.wmnet not responding.
10Timeout, server wtp1035.eqiad.wmnet not responding.
11Timeout, server mw1327.eqiad.wmnet not responding.
12Timeout, server mw1347.eqiad.wmnet not responding.
13Timeout, server wtp1046.eqiad.wmnet not responding.
14Timeout, server mw1348.eqiad.wmnet not responding.
15Timeout, server mw1322.eqiad.wmnet not responding.
16Timeout, server mw1352.eqiad.wmnet not responding.
17Timeout, server mw1362.eqiad.wmnet not responding.
18Timeout, server wtp1033.eqiad.wmnet not responding.
19Timeout, server mw1353.eqiad.wmnet not responding.
20Timeout, server mw1320.eqiad.wmnet not responding.
21Timeout, server mw1344.eqiad.wmnet not responding.
22Timeout, server mw1307.eqiad.wmnet not responding.
23Timeout, server mw2295.codfw.wmnet not responding.
24Timeout, server mw2284.codfw.wmnet not responding.
25Timeout, server mw2269.codfw.wmnet not responding.
26Timeout, server mw2333.codfw.wmnet not responding.
27Timeout, server parse2006.codfw.wmnet not responding.
28Timeout, server mw2322.codfw.wmnet not responding.
29Timeout, server parse2012.codfw.wmnet not responding.
30Timeout, server mw2309.codfw.wmnet not responding.
31Timeout, server mw2253.codfw.wmnet not responding.
32Timeout, server mw2366.codfw.wmnet not responding.
33Timeout, server mw2258.codfw.wmnet not responding.

Per @Dzahn, these seem to be pooled and can sync.

Planning to re-run sync-world and see if these fail again.

Stashbot added a comment.Apr 26 2022, 5:23 PM

Mentioned in SAL (#wikimedia-operations) [2022-04-26T17:22:57Z] <brennen@deploy1002> Finished scap: testwikis wikis to 1.39.0-wmf.9 refs T305215 (duration: 34m 37s)

Dzahn added a comment.Apr 26 2022, 5:23 PM

Yea, I picked mw1362 and wtp1046 an mw2309.. they are pooled, are in conftool-data, could "scap pull" directly from the host without issue.

And the codfw hosts have been rebooted yesterday but seem to be just fine as well.

Stashbot added a comment.Apr 26 2022, 5:26 PM

Mentioned in SAL (#wikimedia-operations) [2022-04-26T17:26:21Z] <brennen@deploy1002> Started scap: Re-running sync-world to see if timeouts recur for 32 hosts (T305215)

Stashbot added a comment.Apr 26 2022, 5:28 PM

Mentioned in SAL (#wikimedia-operations) [2022-04-26T17:28:04Z] <brennen@deploy1002> Finished scap: Re-running sync-world to see if timeouts recur for 32 hosts (T305215) (duration: 01m 43s)

gerritbot added a comment.Apr 26 2022, 6:03 PM

Change 786359 had a related patch set uploaded (by Brennen Bearnes; author: Brennen Bearnes):

[operations/mediawiki-config@master] group0 wikis to 1.39.0-wmf.9 refs T305215

https://gerrit.wikimedia.org/r/786359

gerritbot added a comment.Apr 26 2022, 6:04 PM

Change 786359 merged by jenkins-bot:

[operations/mediawiki-config@master] group0 wikis to 1.39.0-wmf.9 refs T305215

https://gerrit.wikimedia.org/r/786359

Stashbot added a comment.Apr 26 2022, 6:06 PM

Mentioned in SAL (#wikimedia-operations) [2022-04-26T18:06:02Z] <brennen@deploy1002> rebuilt and synchronized wikiversions files: group0 wikis to 1.39.0-wmf.9 refs T305215

brennen mentioned this in T306925: Deprecations: QuickTemplate::(get/html/text/haveData) with parameters bottomscripts and headelement.Apr 26 2022, 6:18 PM
brennen mentioned this in T306926: PHP Deprecated: Use of BaseTemplate::getTrail was deprecated in MediaWiki 1.39. [Called from MediaWiki\Skin\WikimediaApiPortal\ComponentFactory::createMainComponent].
Dzahn added a comment.Apr 26 2022, 7:49 PM

The timeout issues were gone when Brennen repeated the sync a little while later.

Stashbot added a comment.Apr 27 2022, 6:00 PM

Mentioned in SAL (#wikimedia-operations) [2022-04-27T18:00:11Z] <brennen> train 1.39.0-wmf.9 (T305215): no current blockers, proceeding to group1

gerritbot added a comment.Apr 27 2022, 6:13 PM

Change 787047 had a related patch set uploaded (by Brennen Bearnes; author: Brennen Bearnes):

[operations/mediawiki-config@master] group1 wikis to 1.39.0-wmf.9 refs T305215

https://gerrit.wikimedia.org/r/787047

gerritbot added a comment.Apr 27 2022, 6:14 PM

Change 787047 merged by jenkins-bot:

[operations/mediawiki-config@master] group1 wikis to 1.39.0-wmf.9 refs T305215

https://gerrit.wikimedia.org/r/787047

Stashbot added a comment.Apr 27 2022, 6:24 PM

Mentioned in SAL (#wikimedia-operations) [2022-04-27T18:24:35Z] <brennen@deploy1002> rebuilt and synchronized wikiversions files: group1 wikis to 1.39.0-wmf.9 refs T305215

Stashbot added a comment.Apr 27 2022, 6:25 PM

Mentioned in SAL (#wikimedia-operations) [2022-04-27T18:25:33Z] <brennen@deploy1002> Synchronized php: group1 wikis to 1.39.0-wmf.9 refs T305215 (duration: 00m 56s)

brennen mentioned this in T307037: DBQueryError: Error 1066: Not unique table/alias: 'page' ([db]) Function: SpecialExport::getLinksQuery.Apr 27 2022, 7:59 PM
ItamarWMDE subscribed.Apr 28 2022, 8:15 AM
Stashbot added a comment.Apr 28 2022, 6:01 PM

Mentioned in SAL (#wikimedia-operations) [2022-04-28T18:01:01Z] <brennen> train 1.39.0-wmf.9 (T305215): no current blockers, logs fairly clear, proceeding to all wikis as soon as i finish this burrito

gerritbot added a comment.Apr 28 2022, 6:04 PM

Change 787540 had a related patch set uploaded (by Brennen Bearnes; author: Brennen Bearnes):

[operations/mediawiki-config@master] all wikis to 1.39.0-wmf.9 refs T305215

https://gerrit.wikimedia.org/r/787540

gerritbot added a comment.Apr 28 2022, 6:05 PM

Change 787540 merged by jenkins-bot:

[operations/mediawiki-config@master] all wikis to 1.39.0-wmf.9 refs T305215

https://gerrit.wikimedia.org/r/787540

Stashbot added a comment.Apr 28 2022, 6:07 PM

Mentioned in SAL (#wikimedia-operations) [2022-04-28T18:07:43Z] <brennen@deploy1002> rebuilt and synchronized wikiversions files: all wikis to 1.39.0-wmf.9 refs T305215

brennen mentioned this in T307131: InvalidArgumentException: [page] is not a TranslatableBundle.Apr 28 2022, 6:15 PM
brennen mentioned this in T307132: PHP Deprecated: Caller from Wikimedia\Rdbms\Database::ping ignored an error originally raised from WikiPage::updateRevisionOn: [1205] Lock wait timeout exceeded; try restarting transaction ([db]).Apr 28 2022, 6:31 PM
brennen closed this task as Resolved.Apr 28 2022, 9:26 PM
brennen added a project: User-brennen.

Things generally quiet on all wikis since deploy.

Optimistically resolving this.

brennen moved this task from Backlog to Done or Declined on the User-brennen board.Apr 28 2022, 9:26 PM
taavi added a subtask: T307302: TitleBlacklist class renames breaking other extensions.May 1 2022, 2:58 PM
taavi removed a subtask: T307302: TitleBlacklist class renames breaking other extensions.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL