Page MenuHomePhabricator
Create Task
Maniphest T398873

Move nightly image build from releases-jenkins to deployment.eqiad.wmnet
Closed, ResolvedPublicFeature
Actions

Description

Adopt the behavior described in the design agreements that came out of T379683: [FY24-25 WE6.2.6] Create design document for Pretrain (née Group -1) deployment.

A new wmf/next image will be created once per day starting at 01:00 UTC. This image will be deployed via an automated process triggered by a scheduled task running from the active deployment server Monday through Thursday at 02:00 UTC.

We will be leaving the nightly branch cut process on deployment-jenkins at this time. The job that is moving is the image build and publish for that branch.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
pretrain: Run one hour later, at 02:00UTCoperations/puppetproduction+1 -1
pretrain: Use && instead of ';' to separate commandsoperations/puppetproduction+1 -1
pretrain: Use bash to execute multiple commandsoperations/puppetproduction+1 -1
data.yaml: Allow release-engineering to administer pretrain timeroperations/puppetproduction+4 -1
deployment_server: Add pretrain systemd timeroperations/puppetproduction+14 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

bd808 created this task.Jul 7 2025, 8:29 PM
bd808 triaged this task as High priority.Jul 7 2025, 8:32 PM
bd808 changed the subtype of this task from "Task" to "Feature Request".
bd808 added a comment.Jul 8 2025, 9:21 PM

https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/blob/master/conf/releasing/casc/jobs/publishMWSingleVersion.groovy is the current job.

dancy claimed this task.Jul 8 2025, 9:43 PM
bd808 updated the task description. (Show Details)Jul 8 2025, 9:51 PM
dancy added a subscriber: dduvall.Jul 8 2025, 10:12 PM

@dduvall There's a hazard in the way that the MediaWiki branch and publish WMF single-version image job handles the wmf/next branches. It deletes the branches first, then creates fresh branches in the relevant repos. Then it creates a commit to set up .gitmodules, etc and pushes it to Gerrit and it waits for CI to finish. If CI fails (which happens about 10% of the time), we're left with an unusable wmf/next branch until the next run. If scap prep next runs before MediaWiki branch and publish WMF single-version image runs again, it will fail like so:

dancy@deploy1003:/srv/mediawiki-staging$ scap prep next
21:56:24 Started scap prep next
21:56:24 Copying patches from /srv/patches/1.45.0-wmf.9 to /srv/patches/next
21:56:24 Clone https://gerrit.wikimedia.org/r/mediawiki/core (wmf/next branch) in /srv/mediawiki-staging/php-next
21:56:41 https://gerrit.wikimedia.org/r/mediawiki/core checked out at commit a9e4ca532000bcf0f190da549db728e71b1c943c
21:56:41 Finished scap prep next (duration: 00m 16s)
/srv/mediawiki-staging/php-next/.gitmodules does not exist. Did the train branch commit get merged?
CodeReviewBot added a project: Patch-For-Review.Jul 11 2025, 7:04 PM

dancy opened https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/888

utils.py: Make select_latest_patches treat /srv/patches/next as latest

CodeReviewBot added a comment.Jul 14 2025, 3:29 PM

dancy merged https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/888

utils.py: Make select_latest_patches treat /srv/patches/next as latest

Maintenance_bot removed a project: Patch-For-Review.Jul 14 2025, 3:31 PM
Stashbot added a comment.Jul 16 2025, 7:14 PM

Mentioned in SAL (#wikimedia-operations) [2025-07-16T19:14:51Z] <dancy@deploy1003> Started scap build-images: Testing T398873

Stashbot added a comment.Jul 16 2025, 7:19 PM

Mentioned in SAL (#wikimedia-operations) [2025-07-16T19:19:25Z] <dancy@deploy1003> Finished scap build-images: Testing T398873 (duration: 04m 34s)

bd808 moved this task from Priority Backlog 📥 to Doing 😎 on the Release-Engineering-Team board.Jul 18 2025, 4:53 PM
bd808 edited projects, added Release-Engineering-Team (Doing 😎); removed Release-Engineering-Team (Priority Backlog 📥).
bd808 mentioned this in T399970: Audit alerting for `wmf/next` branch creation/update failures to ensure that failures are visible to RelEng and other interested parties.Jul 18 2025, 5:01 PM
dduvall added a comment.Jul 22 2025, 9:44 PM
In T398873#10986500, @dancy wrote:

If CI fails (which happens about 10% of the time), we're left with an unusable wmf/next branch until the next run

The several most recent failures look to be related to flaky selenium tests.

Perhaps we could configure Zuul to skip these tests for the initial branch commit? (They should not be skipped for backports to the branch.)

dancy added a comment.Jul 22 2025, 9:47 PM

@dduvall I'd like to see MediaWiki branch and publish WMF single-version image changed so that instead of destroying and recreating the wmf/next branch each time it runs, it updates wmf/next if it already exists. This means being able to handle added/dropped extensions.

dduvall added a comment.Jul 22 2025, 9:49 PM
In T398873#11026078, @dancy wrote:

@dduvall I'd like to see MediaWiki branch and publish WMF single-version image changed so that instead of destroying and recreating the wmf/next branch each time it runs, it updates wmf/next if it already exists. This means being able to handle added/dropped extensions.

Wouldn't that still require an update to the .gitmodules in mediawiki/core? Meaning a new change submitted to Gerrit and a CI run (and potential failure).

dancy added a comment.EditedJul 22 2025, 9:52 PM
In T398873#11026080, @dduvall wrote:
In T398873#11026078, @dancy wrote:

@dduvall I'd like to see MediaWiki branch and publish WMF single-version image changed so that instead of destroying and recreating the wmf/next branch each time it runs, it updates wmf/next if it already exists. This means being able to handle added/dropped extensions.

Wouldn't that still require an update to the .gitmodules in mediawiki/core? Meaning a new change submitted to Gerrit and a CI run (and potential failure).

Yes, but in the meantime scap prep next would continue to work correctly (assuming at least one prior successful branch cut was merged). As it stands, if MediaWiki branch and publish WMF single-version image fails, a subsequent scap prep next will fail.

dduvall added a comment.Jul 22 2025, 9:57 PM
In T398873#11026091, @dancy wrote:

Yes, but in the meantime scap prep next would continue to work correctly (assuming at least one prior successful branch cut was merged). As it stands, if`MediaWiki branch and publish WMF single-version image` fails, a subsequent scap prep next will fail.

That makes sense. I was thinking more about the root cause, but making scap prep next more tolerant of failure seems just as important.

gerritbot added a comment.Jul 25 2025, 5:12 PM

Change #1172678 had a related patch set uploaded (by Ahmon Dancy; author: Ahmon Dancy):

[operations/puppet@production] deployment_server: Add pretrain systemd timer

https://gerrit.wikimedia.org/r/1172678

gerritbot added a project: Patch-For-Review.Jul 25 2025, 5:12 PM
gerritbot added a comment.Jul 28 2025, 5:07 PM

Change #1172678 merged by Clément Goubert:

[operations/puppet@production] deployment_server: Add pretrain systemd timer

https://gerrit.wikimedia.org/r/1172678

Maintenance_bot removed a project: Patch-For-Review.Jul 28 2025, 5:31 PM
gerritbot added a comment.Jul 28 2025, 6:53 PM

Change #1173446 had a related patch set uploaded (by Ahmon Dancy; author: Ahmon Dancy):

[operations/puppet@production] data.yaml: Allow release-engineering to administer pretrain timer

https://gerrit.wikimedia.org/r/1173446

gerritbot added a project: Patch-For-Review.Jul 28 2025, 6:53 PM
CodeReviewBot added a comment.Jul 28 2025, 7:28 PM

dancy opened https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/merge_requests/101

Disable the 'MediaWiki publish WMF single-version image' job

Jdforrester-WMF awarded a token.Jul 28 2025, 7:33 PM
CodeReviewBot added a comment.Jul 28 2025, 7:44 PM

dancy merged https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/merge_requests/101

Disable the 'MediaWiki publish WMF single-version image' job

Stashbot added a comment.Jul 28 2025, 7:46 PM

Mentioned in SAL (#wikimedia-operations) [2025-07-28T19:46:42Z] <dancy@deploy1003> Started deploy [releng/jenkins-deploy@b89eed0] (releasing): Disabling the MediaWiki publish WMF single-version image job (T398873)

Stashbot added a comment.Jul 28 2025, 7:47 PM

Mentioned in SAL (#wikimedia-operations) [2025-07-28T19:47:30Z] <dancy@deploy1003> Finished deploy [releng/jenkins-deploy@b89eed0] (releasing): Disabling the MediaWiki publish WMF single-version image job (T398873) (duration: 01m 11s)

gerritbot added a comment.Jul 29 2025, 9:45 AM

Change #1173446 merged by Clément Goubert:

[operations/puppet@production] data.yaml: Allow release-engineering to administer pretrain timer

https://gerrit.wikimedia.org/r/1173446

Maintenance_bot removed a project: Patch-For-Review.Jul 29 2025, 10:31 AM
gerritbot added a comment.Jul 29 2025, 5:48 PM

Change #1173991 had a related patch set uploaded (by Ahmon Dancy; author: Ahmon Dancy):

[operations/puppet@production] pretrain: Use bash to execute multiple commands

https://gerrit.wikimedia.org/r/1173991

gerritbot added a project: Patch-For-Review.Jul 29 2025, 5:48 PM
gerritbot added a comment.Jul 29 2025, 7:01 PM

Change #1173991 merged by Dzahn:

[operations/puppet@production] pretrain: Use bash to execute multiple commands

https://gerrit.wikimedia.org/r/1173991

Maintenance_bot removed a project: Patch-For-Review.Jul 29 2025, 7:31 PM
gerritbot added a comment.Jul 29 2025, 9:36 PM

Change #1174051 had a related patch set uploaded (by Ahmon Dancy; author: Ahmon Dancy):

[operations/puppet@production] pretrain: Use && instead of ';' to separate commands

https://gerrit.wikimedia.org/r/1174051

gerritbot added a project: Patch-For-Review.Jul 29 2025, 9:36 PM
gerritbot added a comment.Jul 29 2025, 9:50 PM

Change #1174051 merged by Dzahn:

[operations/puppet@production] pretrain: Use && instead of ';' to separate commands

https://gerrit.wikimedia.org/r/1174051

Maintenance_bot removed a project: Patch-For-Review.Jul 29 2025, 10:30 PM
dancy closed this task as Resolved.Jul 30 2025, 2:56 PM

This is live.

bd808 mentioned this in T401749: Modify wmf/next branching process to avoid leaving an incomplete branch on failure.Aug 12 2025, 8:33 PM
gerritbot added a comment.Fri, Jan 23, 4:16 PM

Change #1230952 had a related patch set uploaded (by Ahmon Dancy; author: Ahmon Dancy):

[operations/puppet@production] pretrain: Run one hour later, at 02:00UTC

https://gerrit.wikimedia.org/r/1230952

gerritbot added a project: Patch-For-Review.Fri, Jan 23, 4:16 PM
gerritbot added a comment.Mon, Jan 26, 1:04 PM

Change #1230952 merged by Dzahn:

[operations/puppet@production] pretrain: Run one hour later, at 02:00UTC

https://gerrit.wikimedia.org/r/1230952

Stashbot added a comment.Mon, Jan 26, 1:07 PM

Mentioned in SAL (#wikimedia-operations) [2026-01-26T13:07:53Z] <mutante> pre-train sync shifted to one hour later T398873

Maintenance_bot removed a project: Patch-For-Review.Mon, Jan 26, 1:31 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits