Page MenuHomePhabricator
Create Task
Maniphest T330209

1.41.0-wmf.3 deployment blockers
Closed, ResolvedPublic5 Estimated Story PointsRelease
Actions

Details

Backup Train Conductor
dduvall
Release Version
1.41.0-wmf.3
Release Date
Apr 3 2023, 12:00 AM

2023 week 14 1.41-wmf.3 Changes wmf/1.41.0-wmf.3

This MediaWiki Train Deployment is scheduled for the week of Monday, April 3rd:

Monday April 3rdTuesday, April 4thWednesday, April 5thThursday, April 6thFriday
Backports only.Branch wmf.3 and deploy to Group 0 Wikis.Deploy wmf.3 to Group 1 Wikis.Deploy wmf.3 to all Wikis.No deployments on fridays

How this works

  • Any serious bugs affecting wmf.3 should be added as subtasks beneath this one.
  • Any open subtask(s) block the train from moving forward. This means no further deployments until the blockers are resolved.
  • If something is serious enough to warrant a rollback then you should bring it to the attention of deployers on the #wikimedia-operations IRC channel.
  • If you have a risky change in this week's train add a comment to this task using the Risky patch template
  • For more info about deployment blockers, see Holding the train.

Related Links

Other Deployments

Previous: 1.41.0-wmf.2
Next: 1.41.0-wmf.4
SubjectRepoBranchLines +/-
all wikis to 1.41.0-wmf.3operations/mediawiki-configmaster+328 -328
group1 wikis to 1.41.0-wmf.3operations/mediawiki-configmaster+543 -543
group0 wikis to 1.41.0-wmf.3operations/mediawiki-configmaster+128 -128
Revert "group0 wikis to 1.41.0-wmf.3"operations/mediawiki-configmaster+128 -128
group0 wikis to 1.41.0-wmf.3operations/mediawiki-configmaster+128 -128
testwikis wikis to 1.41.0-wmf.3operations/mediawiki-configmaster+4 -4
Branch commit for wmf/1.41.0-wmf.3mediawiki/corewmf/1.41.0-wmf.3+991 -3
Customize query in gerrit

Related Objects
Search...

Event Timeline

thcipriani created this task.Feb 21 2023, 5:57 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 21 2023, 5:57 PM
thcipriani assigned this task to • demon.Mar 24 2023, 9:45 PM
thcipriani reassigned this task from • demon to hashar.
thcipriani triaged this task as Medium priority.
thcipriani updated Other Assignee, added: dduvall.
thcipriani set the point value for this task to 5.
thcipriani added a project: Release-Engineering-Team (Priority Backlog 📥).
thcipriani added a subscriber: • demon.
hashar awarded a token.Mar 24 2023, 9:54 PM
Jdlrobson added a parent task: T333200: TypeError: blockTargetWidget.getValue(...).trim is not a function.Mar 27 2023, 4:56 PM
gerritbot added a comment.Apr 4 2023, 2:07 AM

Change 904919 had a related patch set uploaded (by TrainBranchBot; author: trainbranchbot):

[mediawiki/core@wmf/1.41.0-wmf.3] Branch commit for wmf/1.41.0-wmf.3

https://gerrit.wikimedia.org/r/904919

gerritbot added a project: Patch-For-Review.Apr 4 2023, 2:07 AM
gerritbot added a comment.Apr 4 2023, 2:21 AM

Change 904919 merged by jenkins-bot:

[mediawiki/core@wmf/1.41.0-wmf.3] Branch commit for wmf/1.41.0-wmf.3

https://gerrit.wikimedia.org/r/904919

hashar added a comment.Apr 4 2023, 8:19 AM
> Systemd timer ran the following command:
> 
>     /usr/bin/scap stage-train -Dfull_image_build:True --yes auto
> 
> Its return value was 1 and emitted the following output:
> 
> 03:00:00 Initializing stage-train auto mode
> 03:00:00 Retrieving train information...
> Aborting: Phabricator task T330209 says the train version is '1.41.0-wmf.3', but 'branch_cut_pretest' is the latest available in Gerrit.
hashar added a subscriber: dancy.Apr 4 2023, 8:22 AM

And indeed there is a new branch:

* [new branch]              wmf/branch_cut_pretest -> origin/wmf/branch_cut_pretest

I guess something in the logic is borked and does not properly filter the branches reported by Gerrit. @dancy warned about the script being rewritten to Python.

hashar added a comment.Apr 4 2023, 8:26 AM

https://gerrit.wikimedia.org/r/admin/repos/mediawiki/core,branches shows the latest branches as:

wmf/1.41.0-wmf.3rMW16e55ac99a08
wmf/branch_cut_pretestrMW430d25d1a185

The commits are:

commit 16e55ac99a08f8460615610404a685bf0c76a8cb (origin/wmf/1.41.0-wmf.3)
Author:     trainbranchbot <trainbranchbot@releases-jenkins.wikimedia.org>
AuthorDate: Tue Apr 4 02:07:52 2023 +0000
Commit:     trainbranchbot <trainbranchbot@releases-jenkins.wikimedia.org>
CommitDate: Tue Apr 4 02:07:52 2023 +0000

    Branch commit for wmf/1.41.0-wmf.3
    
    Bug: T330209
    Change-Id: I3eb055c9f3416960892edcc580e008bcc14ea4cb

And

commit 430d25d1a1858edfa4a6199dfe1f0eb3743a219a (origin/wmf/branch_cut_pretest)
Author:     trainbranchbot <trainbranchbot@releases-jenkins.wikimedia.org>
AuthorDate: Tue Apr 4 00:39:27 2023 +0000
Commit:     trainbranchbot <trainbranchbot@releases-jenkins.wikimedia.org>
CommitDate: Tue Apr 4 00:39:27 2023 +0000

    Branch commit for wmf/branch_cut_pretest
    
    Change-Id: I412ee39543ece3db9c1dddbf5818c9e2cd3176e9

I am going to delete that wmf/branch_cut_pretest and manually rerun the script then recreate the branch.

Stashbot added a comment.Apr 4 2023, 8:28 AM

Mentioned in SAL (#wikimedia-operations) [2023-04-04T08:28:35Z] <hashar> Deleting mediawiki/core branch wmf/branch_cut_pretest pointing at 430d25d1a1858edfa4a6199dfe1f0eb3743a219a # T330209

gerritbot added a comment.Apr 4 2023, 8:30 AM

Change 905568 had a related patch set uploaded (by TrainBranchBot; author: Hashar):

[operations/mediawiki-config@master] testwikis wikis to 1.41.0-wmf.3

https://gerrit.wikimedia.org/r/905568

gerritbot added a comment.Apr 4 2023, 8:31 AM

Change 905568 merged by jenkins-bot:

[operations/mediawiki-config@master] testwikis wikis to 1.41.0-wmf.3

https://gerrit.wikimedia.org/r/905568

Stashbot added a comment.Apr 4 2023, 8:32 AM

Mentioned in SAL (#wikimedia-operations) [2023-04-04T08:32:12Z] <hashar@deploy2002> Started scap: testwikis wikis to 1.41.0-wmf.3 refs T330209

hashar added a comment.Apr 4 2023, 8:40 AM

The train script breaking on the branch is from https://gitlab.wikimedia.org/repos/releng/scap/-/blob/master/scap/utils.py:

def get_current_train_version_from_gerrit(gerrit_url) -> str:
    """Returns a string like '1.39.0-wmf.19'"""

    url = os.path.join(gerrit_url, "mediawiki/core")

    # output will be something like '3137081c2ab92df3bc9c97956b00fb3017d7b511\trefs/heads/wmf/1.39.0-wmf.19'
    output = subprocess.check_output(["git", "ls-remote", "--sort=version:refname", url, "refs/heads/wmf/*"],
                                     text=True)
    res = re.sub(r"^.*wmf/(.*)$", "\\1", output.splitlines()[-1])

    return res

But the branch_cut_pretest version comes after 1.41.*. Deleting the branch worked around it. Jaime will propose a patch and I guess Ahmon will review/approve it later today.

hashar added a subtask: T333922: [k8s,infra] k8s control plane freezing and other stability issues.Apr 4 2023, 9:23 AM
hashar mentioned this in T333922: [k8s,infra] k8s control plane freezing and other stability issues.
gerritbot added a comment.Apr 4 2023, 9:34 AM

Change 905586 had a related patch set uploaded (by TrainBranchBot; author: Hashar):

[operations/mediawiki-config@master] group0 wikis to 1.41.0-wmf.3

https://gerrit.wikimedia.org/r/905586

gerritbot added a comment.Apr 4 2023, 9:35 AM

Change 905586 merged by jenkins-bot:

[operations/mediawiki-config@master] group0 wikis to 1.41.0-wmf.3

https://gerrit.wikimedia.org/r/905586

hashar removed a subtask: T333922: [k8s,infra] k8s control plane freezing and other stability issues.Apr 4 2023, 9:40 AM
Stashbot added a comment.Apr 4 2023, 9:42 AM

Mentioned in SAL (#wikimedia-operations) [2023-04-04T09:42:05Z] <hashar@deploy2002> rebuilt and synchronized wikiversions files: group0 wikis to 1.41.0-wmf.3 refs T330209

gerritbot added a comment.Apr 4 2023, 9:49 AM

Change 905590 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/mediawiki-config@master] Revert "group0 wikis to 1.41.0-wmf.3"

https://gerrit.wikimedia.org/r/905590

gerritbot added a comment.Apr 4 2023, 9:50 AM

Change 905590 merged by jenkins-bot:

[operations/mediawiki-config@master] Revert "group0 wikis to 1.41.0-wmf.3"

https://gerrit.wikimedia.org/r/905590

Stashbot added a comment.Apr 4 2023, 9:51 AM

Mentioned in SAL (#wikimedia-operations) [2023-04-04T09:51:21Z] <hashar@deploy2002> rebuilt and synchronized wikiversions files: Revert "group0 wikis to 1.41.0-wmf.3" | T330209

hashar mentioned this in T333926: PHP Deprecated: Accessing $wgHooks directly is deprecated, use HookContainer::getHandlers() or HookContainer::register() instead. [Called from {closure}].Apr 4 2023, 9:53 AM
hashar added a subtask: T333926: PHP Deprecated: Accessing $wgHooks directly is deprecated, use HookContainer::getHandlers() or HookContainer::register() instead. [Called from {closure}].
hashar mentioned this in T331602: Make $wgHooks a regular configuration variable.Apr 4 2023, 9:56 AM
Lydia_Pintscher subscribed.Apr 4 2023, 2:35 PM

T333966 seems pretty bad. Marking as blocker for now.

Lucas_Werkmeister_WMDE added a subtask: T333966: message keys shown on beta and group0 wikis (test Wikidata, etc.).Apr 4 2023, 2:35 PM
hashar added a comment.EditedApr 4 2023, 3:12 PM

The train is on hold due to T333966, the new version is only deployed on testwikis for now.

I have send the blocking announcement to the mailling lists (wikitech-l) and on the internal WMF Slack channel.

hashar added a subscriber: dduvall.Apr 4 2023, 3:35 PM

I am hanging off for kid & dinner etc. I have let others from releng knows about it, I guess @dduvall can promote group 0 wikis once the fix for T333966 has landed.

The other blocker (T333926) was a deprecation notice about a hook using $wgHooks but that got fixed earlier today, the task is still open pending the deployment to group 0 to confirm it has indeed been fixed.

Ladsgroup removed a subtask: T333966: message keys shown on beta and group0 wikis (test Wikidata, etc.).Apr 4 2023, 5:25 PM
gerritbot added a comment.Apr 4 2023, 7:02 PM

Change 905704 had a related patch set uploaded (by TrainBranchBot; author: Hashar):

[operations/mediawiki-config@master] group0 wikis to 1.41.0-wmf.3

https://gerrit.wikimedia.org/r/905704

gerritbot added a comment.Apr 4 2023, 7:03 PM

Change 905704 merged by jenkins-bot:

[operations/mediawiki-config@master] group0 wikis to 1.41.0-wmf.3

https://gerrit.wikimedia.org/r/905704

Stashbot added a comment.Apr 4 2023, 7:10 PM

Mentioned in SAL (#wikimedia-operations) [2023-04-04T19:10:09Z] <hashar@deploy2002> rebuilt and synchronized wikiversions files: group0 wikis to 1.41.0-wmf.3 refs T330209

gerritbot added a comment.Apr 5 2023, 8:45 AM

Change 905938 had a related patch set uploaded (by TrainBranchBot; author: Hashar):

[operations/mediawiki-config@master] group1 wikis to 1.41.0-wmf.3

https://gerrit.wikimedia.org/r/905938

gerritbot added a comment.Apr 5 2023, 8:45 AM

Change 905938 merged by jenkins-bot:

[operations/mediawiki-config@master] group1 wikis to 1.41.0-wmf.3

https://gerrit.wikimedia.org/r/905938

Stashbot added a comment.Apr 5 2023, 8:52 AM

Mentioned in SAL (#wikimedia-operations) [2023-04-05T08:52:17Z] <hashar@deploy2002> rebuilt and synchronized wikiversions files: group1 wikis to 1.41.0-wmf.3 refs T330209

Stashbot added a comment.Apr 5 2023, 8:58 AM

Mentioned in SAL (#wikimedia-operations) [2023-04-05T08:58:04Z] <hashar@deploy2002> Synchronized php: group1 wikis to 1.41.0-wmf.3 refs T330209 (duration: 05m 46s)

hashar closed subtask T333926: PHP Deprecated: Accessing $wgHooks directly is deprecated, use HookContainer::getHandlers() or HookContainer::register() instead. [Called from {closure}] as Resolved.Apr 5 2023, 9:05 AM
gerritbot added a comment.Apr 6 2023, 8:01 AM

Change 906530 had a related patch set uploaded (by TrainBranchBot; author: Hashar):

[operations/mediawiki-config@master] all wikis to 1.41.0-wmf.3

https://gerrit.wikimedia.org/r/906530

gerritbot added a comment.Apr 6 2023, 8:02 AM

Change 906530 merged by jenkins-bot:

[operations/mediawiki-config@master] all wikis to 1.41.0-wmf.3

https://gerrit.wikimedia.org/r/906530

Stashbot added a comment.Apr 6 2023, 8:09 AM

Mentioned in SAL (#wikimedia-operations) [2023-04-06T08:09:01Z] <hashar@deploy2002> rebuilt and synchronized wikiversions files: all wikis to 1.41.0-wmf.3 refs T330209

hashar added a comment.Apr 6 2023, 9:51 AM

Not much happened after promoting to all wikis. We investigated some elevated traffic rates and latency increases that happened over night but dismissed all of them. I will do some further error log triages after lunch, but so far it looks like it is stable.

TheDJ subscribed.Apr 6 2023, 12:18 PM

Note that the release notes are MIA https://www.mediawiki.org/wiki/MediaWiki_1.41/wmf.3

hashar mentioned this in T334211: train-deploy-notes failed with "1.41.0-wmf.3 does not match the latest wmf branch".Apr 6 2023, 1:58 PM
In T330209#8753449, @hashar wrote:

The train script breaking on the branch is from https://gitlab.wikimedia.org/repos/releng/scap/-/blob/master/scap/utils.py:

def get_current_train_version_from_gerrit(gerrit_url) -> str:
    """Returns a string like '1.39.0-wmf.19'"""

    url = os.path.join(gerrit_url, "mediawiki/core")

    # output will be something like '3137081c2ab92df3bc9c97956b00fb3017d7b511\trefs/heads/wmf/1.39.0-wmf.19'
    output = subprocess.check_output(["git", "ls-remote", "--sort=version:refname", url, "refs/heads/wmf/*"],
                                     text=True)
    res = re.sub(r"^.*wmf/(.*)$", "\\1", output.splitlines()[-1])

    return res

But the branch_cut_pretest version comes after 1.41.*. Deleting the branch worked around it. Jaime will propose a patch and I guess Ahmon will review/approve it later today.

This has been solved by https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/115

hashar closed this task as Resolved.Apr 6 2023, 1:59 PM

I am claiming 1.41.0-wmf.3 to be a success.

hashar mentioned this in T334322: wmf/branch_cut_pretest branch on mediawiki/core causes multiple problems.Apr 13 2023, 1:40 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL