Move beta cluster automatic deployment to a dedicated infrastructure
Closed, ResolvedPublicFeature
Actions

Description

CI takes care of updating the Beta cluster infrastructure. To limit the scope of CI Jenkins and decouple it from Beta cluster updating, we could move that out of the CI Jenkins to a dedicated infrastructure.

The job lists can be seen at: https://integration.wikimedia.org/ci/view/Beta/

beta-code-update-eqiad	Triggers every 10 minutes, update mediawiki code from Gerrit master branches
beta-scap-eqiad	Scap job triggered by beta-code-update-eqiad
beta-update-databases-eqiad	Triggers every hour, runs MediaWiki update.php against each wikis

The jobs are defined using the Jenkins job builder DSL. Our tutorial to set up jenkins job builder and some overview is at https://www.mediawiki.org/wiki/CI/JJB

Upstream doc https://jenkins-job-builder.readthedocs.io/en/latest/

Our DSL files are in integration/config.git under ./jjb/

@hashar guesses those jobs can be moved to a dedicated Jenkins, possibly part of Beta-Cluster-Infrastructure .

Details

Related Changes in Gerrit:

Subject	Author	Repo	Branch	Lines +/-
beta::autoupdater: Add tail-beta-update-logs utility	Ahmon Dancy	operations/puppet	production	+45 -0
beta: Add a wmf-beta-update-all timer and script	BryanDavis	operations/puppet	production	+220 -209
jjb: remove Beta cluster jobs	Hashar	integration/config	master	+5 -169
jjb: rename beta-irc macro to notify-releng-irc	Hashar	integration/config	master	+23 -23
jjb: beta.yml: Note that jobs are disabled	Reedy	integration/config	master	+1 -1
jjb: disable beta updaters	Hashar	integration/config	master	+2 -0

Customize query in gerrit

Related Objects
Search...

Status	Subtype	Assigned	Task
			Restricted Task
Declined		None	T254919 Run two Jenkins master
Resolved		None	T72597 Jenkins Gearman plugin has deadlock on executor threads (was: Beta Cluster stopped receiving code updates (beta-update-databases-eqiad hung)
Resolved		hashar	T418109 Update Jenkins hosts from Java 17 to Java 21
Resolved		Dzahn	T418521 setup 2 contint machines for jenkins
Resolved	Feature	bd808	T256168 Move beta cluster automatic deployment to a dedicated infrastructure

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

@hashar I'm interesting on working on this but I think this would require access to the beta cluster openstack project (which makes me remove good first task).

Restricted Application added a project: User-Majavah. · View Herald TranscriptJun 26 2020, 7:00 AM

good first task is there for the Release-Engineering-Team WMF team. We have a new person coming in and we created a few basic tasks for the onboarding :-]

@hashar: good first task are for any contributors and requires that tasks do not require any special permissions in order to test the contribution or to fix the task. Anyone should be able to reproduce.. If this is for onboarding in some team then please either use a dedicated workboard column on your team's workboard for onboarding task, or set the task assignee to the new person coming in, but (in my understanding) please don't set good first task - thanks!

taavi removed taavi as the assignee of this task.Jun 27 2020, 4:45 AM

taavi removed a project: User-Majavah.

• thcipriani edited projects, added: Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)); removed: Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)).Oct 21 2020, 1:07 AM

• thcipriani removed a project: Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)).Apr 20 2021, 12:21 AM

• thcipriani added a project: Release-Engineering-Team-TODO (2021-04-01 to 2021-06-30 (Q4)).Apr 20 2021, 12:34 AM

• thcipriani removed a project: Release-Engineering-Team (CI & Testing services).Apr 20 2021, 1:09 AM

• thcipriani edited projects, added: Release-Engineering-Team (Doing); removed: Release-Engineering-Team-TODO (2021-04-01 to 2021-06-30 (Q4)).Apr 20 2021, 4:25 AM

hashar mentioned this in T188367: Use cron instead of Jenkins for beta deployments.Jun 4 2021, 12:41 PM

hashar added a parent task: T72597: Jenkins Gearman plugin has deadlock on executor threads (was: Beta Cluster stopped receiving code updates (beta-update-databases-eqiad hung).

Adding #quality-and-test-engineering-team_qte to determine as product owners if this is still the right path

• Jrbranaa moved this task from Inbox to Test Infrastructure on the Quality-and-Test-Engineering-Team board.Sep 7 2021, 4:27 PM

The releases Jenkins is now deployed with scap using YAML to define the state of the instance (see T323909). From there we should be able to create the config needed to generate and deploy a standalone Jenkins on Beta-Cluster-Infrastructure

Restricted Application added a project: Continuous-Integration-Infrastructure. · View Herald TranscriptFeb 8 2023, 1:17 PM

hashar mentioned this in T183164: Move the beta cluster jobs to a dedicated/standalone Jenkins instance.Feb 8 2023, 1:19 PM

Jdforrester-WMF subscribed.Feb 8 2023, 1:59 PM

• TAdeleye_WMF edited projects, added: Quality-and-Test-Engineering-Team (Test Infrastructure); removed: Quality-and-Test-Engineering-Team.Mar 1 2023, 2:56 PM

• thcipriani triaged this task as Low priority.May 1 2023, 10:41 PM

Was this task created in anticipation of the CI Jenkins instance being retired? The description here is very much about the "what" of the work, but seems to assume that everyone would already know the "why".

In T256168#10082504, @bd808 wrote:

Was this task created in anticipation of the CI Jenkins instance being retired? The description here is very much about the "what" of the work, but seems to assume that everyone would already know the "why".

Took a crack at updating the task description with the rationale as I remember it. @hashar can correct me if I'm wrong :)

hashar edited projects, added: Continuous-Integration-Infrastructure (Zuul upgrade); removed: Continuous-Integration-Infrastructure.Jan 7 2025, 9:03 PM

Restricted Application edited projects, added: Continuous-Integration-Infrastructure; removed: Continuous-Integration-Infrastructure (Zuul upgrade). · View Herald TranscriptJan 7 2025, 9:03 PM

hashar mentioned this in T383067: EPIC: Migrate jobs from Zuulv2 to Zuulv3+.Jan 7 2025, 9:52 PM

hashar updated the task description. (Show Details)Aug 20 2025, 7:51 AM

In T256168#8597665, @hashar wrote:

The releases Jenkins is now deployed with scap using YAML to define the state of the instance (see T323909). From there we should be able to create the config needed to generate and deploy a standalone Jenkins on Beta-Cluster-Infrastructure

This idea was from quite a while ago now. I am wondering if we likely to keep Jenkins deployment tooling around after Continuous-Integration-Infrastructure (Zuul upgrade) happens?

I'm trying to think about how this might interact with other Beta Cluster future ideas like T276650: Re-consider setting up a Kubernetes cluster on the Beta cluster too. If we move MediaWiki to Kubernetes in Beta Cluster some of the work done by the Jenkins jobs could move to systemd timers to match Pretrain things in WMF production. We would still need some extra bits to prep the staging directory and update databases once the new container is available. Having something like Jenkins to make it easier to see job status and trends would be nice, but we could probably do everything on the deployment server itself too.

bd808 moved this task from To Triage to Future on the Beta-Cluster-Infrastructure board.Sep 17 2025, 10:56 PM

bd808 raised the priority of this task from Low to Medium.Mar 25 2026, 3:40 PM

bd808 changed the subtype of this task from "Task" to "Feature Request".

In T256168#11103426, @bd808 wrote:

Having something like Jenkins to make it easier to see job status and trends would be nice, but we could probably do everything on the deployment server itself too.

The more I think about this, a Jenkins in Beta, the more I dislike it. The visibility of logging and ease of disabling the jobs are good, but authentication for Jenkins and patching for CVEs are big downsides.

bd808 mentioned this in T421244: Replace deployment-deploy04 with a Bookworm instance with Java 21.Mar 25 2026, 4:00 PM

bd808 changed the task status from Open to In Progress.Apr 23 2026, 3:05 PM

bd808 claimed this task.

bd808 added a parent task: T418109: Update Jenkins hosts from Java 17 to Java 21.

bd808 added a project: Release-Engineering-Team (Doing 😎).

Restricted Application added a project: User-bd808. · View Herald TranscriptApr 23 2026, 3:07 PM

Dzahn added a parent task: T418521: setup 2 contint machines for jenkins.Apr 23 2026, 3:23 PM

Having discovered that the Debian Bookworm OS needed for T421244: Replace deployment-deploy04 with a Bookworm instance with Java 21 does not actually support the PHP 8.3 packages needed for operating a deployment server matching the MediaWiki servers in the environment makes this the better option to move forward with quickly. Getting full PHP 8.3 support for Bookworm would be nice, but involves more people.

Things needed:

beta-code-update-eqiad equivalent task to run /usr/bin/scap prep auto and then trigger a beta-scap-sync-world equivalent.
beta-scap-sync-world equivalent task to run /usr/bin/scap sync-world
beta-update-databases-eqiad equivalent task to run /usr/local/bin/wmf-beta-update-databases.py

Today the beta-code-update-eqiad is triggered by a timer every 10 minutes. beta-scap-sync-world is triggered by the success of beta-code-update-eqiad. beta-update-databases-eqiad is triggered by a timer every hour. Things really are as "simple" at the driver script level as those three commands. Tying all three together into a single script task that runs them in series and raises a failure warning if any of them break is probably enough to get us started here. That script can be a systemd unit that is triggered by a systemd timer every 10 minutes.

In T256168#11852212, @bd808 wrote:

Getting full PHP 8.3 support for Bookworm would be nice, but involves more people.

Note that there's T397075: Package Wikimedia's PHP 8.3 component for bookworm which should be done Soon™ if you can wait a bit.

Change #1276813 had a related patch set uploaded (by BryanDavis; author: Bryan Davis):

[operations/puppet@production] beta: Add a wmf-beta-update-all timer and script

https://gerrit.wikimedia.org/r/1276813

gerritbot added a project: Patch-For-Review.Apr 23 2026, 9:48 PM

Mentioned in SAL (#wikimedia-releng) [2026-04-23T22:01:58Z] <bd808> Cherry-picked {{gerrit|1276813}} to deployment-puppetserver-1 (T256168)

Mentioned in SAL (#wikimedia-releng) [2026-04-23T22:12:44Z] <bd808> Disabled https://integration.wikimedia.org/ci/job/beta-code-update-eqiad so that replacement script can be tested (T256168)

Mentioned in SAL (#wikimedia-releng) [2026-04-23T22:16:15Z] <bd808> Disabled https://integration.wikimedia.org/ci/view/Beta/job/beta-update-databases-eqiad so that replacement script can be tested (T256168)

bd808 updated the task description. (Show Details)Apr 23 2026, 10:17 PM

The manual run of sudo -u jenkins-deploy /usr/local/bin/wmf-beta-update-all seemed to work as hoped:

P91379 deployment-deploy04:/srv/beta-update-logs/202604232258.log

1	[22:58:47] + /usr/bin/scap prep auto
2	[22:58:48] 22:58:48 Started scap prep auto
3	[22:58:48] 22:58:48 Update https://gerrit.wikimedia.org/r/operations/mediawiki-config (master branch) in /srv/mediawiki-staging
4	[22:58:49] 22:58:49 https://gerrit.wikimedia.org/r/operations/mediawiki-config checked out at commit ca2f57fe1efb4abeaa4f5934a3ebe27b47c009fe
5	[22:58:50] 22:58:50 Update https://gerrit.wikimedia.org/r/mediawiki/core (master branch) in /srv/mediawiki-staging/php-master
6	[22:58:52] 22:58:52 https://gerrit.wikimedia.org/r/mediawiki/core checked out at commit f35b00970c8ea3a88d24156ff57c3c71cb6955ca
7	[22:58:52] 22:58:52 Update https://gerrit.wikimedia.org/r/mediawiki/extensions (master branch) in /srv/mediawiki-staging/php-master/extensions
8	[22:58:57] 22:58:57 https://gerrit.wikimedia.org/r/mediawiki/extensions checked out at commit ec14ec2122ec9bcadf28fd59640b72ad1fe13c16
9	[23:03:14] 23:03:14 Update https://gerrit.wikimedia.org/r/mediawiki/vendor (master branch) in /srv/mediawiki-staging/php-master/vendor
10	[23:03:15] 23:03:15 https://gerrit.wikimedia.org/r/mediawiki/vendor checked out at commit 7044ca843116097c8a00d5e2e5a432e5d643919f
11	[23:03:16] 23:03:16 Update https://gerrit.wikimedia.org/r/mediawiki/skins (master branch) in /srv/mediawiki-staging/php-master/skins
12	[23:03:17] 23:03:17 https://gerrit.wikimedia.org/r/mediawiki/skins checked out at commit 9ca1e0d1c87ebace4fd47b0554666ba509cc052a
13	[23:03:31] 23:03:31 MediaWiki master successfully checked out.
14	[23:03:31] 23:03:31 Finished scap prep auto (duration: 04m 42s)
15	[23:03:31] + LOCKFILE=/var/lock/scap.operations_mediawiki-config.lock
16	[23:03:31] + '[' -O /var/lock/scap.operations_mediawiki-config.lock ']'
17	[23:03:31] + ulimit -n 512
18	[23:03:31] + /usr/bin/scap version
19	[23:03:32] 4.228.0
20	[23:03:32] + /usr/bin/scap sync-world 'wmf-beta-update-all run'
21	[23:03:33] ___ ____
22	[23:03:33] ⎛ ⎛ ,----
23	[23:03:33] \ //==--'
24	[23:03:33] _//\|,.·//==--' ____________________________
25	[23:03:33] _OO≣=- ︶ ᴹw ⎞_§ ______ ___\ ___\ ,\__ \/ __ \
26	[23:03:33] (∞)_, ) ( \| ______/__ \/ /__ / /_/ / /_/ /
27	[23:03:33] ¨--¨\|\| \|- ( / ______\____/ \___/ \__^_/ .__/
28	[23:03:33] ««_/ «_/ jgs/bd808 /_/
29	[23:03:33]
30	[23:03:33] 23:03:33 Started scap sync-world: wmf-beta-update-all run
31	[23:03:41] 23:03:41 Started cache_git_info
32	[23:04:25] 23:04:25 Finished cache_git_info (duration: 00m 43s)
33	[23:04:25] 23:04:25 Started l10n-update
34	[23:04:30] 23:04:30 Updating ExtensionMessages-master.php
35	[23:04:44] 23:04:44 Updating LocalisationCache for master using 2 thread(s)
36	[23:04:44] 23:04:44 Running rebuildLocalisationCache.php
37	[23:04:58] 23:04:58 0 languages rebuilt out of 548
38	[23:04:58] 23:04:58 Use --force to rebuild the caches which are still fresh.
39	[23:05:03] 23:05:03 Running rebuildLocalisationCache.php
40	[23:05:10] 23:05:10 0 languages rebuilt out of 548
41	[23:05:10] 23:05:10 Use --force to rebuild the caches which are still fresh.
42	[23:05:14] 23:05:14 Generating JSON versions and md5 files (as www-data)
43	[23:05:30] 23:05:30 Finished l10n-update (duration: 01m 05s)
44	[23:05:30] 23:05:30 Checking for new runtime errors locally
45	[23:05:37] 23:05:37 Started sync-canaries
46	[23:05:37] 23:05:37 sync-canaries: 0% (ok: 0; fail: 0; left: 1)
47	[23:05:50] 23:05:50 sync-canaries: 100% (in-flight: 0; ok: 1; fail: 0; left: 0)
48	[23:05:50] 23:05:50 Per-host sync duration: average 12.8s, median 12.8s
49	[23:05:50] 23:05:50 rsync transfer: average 417199 bytes/host, total 417199 bytes
50	[23:05:50] 23:05:50 Started scap-cdb-rebuild-canaries
51	[23:05:50] 23:05:50 scap-cdb-rebuild: 0% (ok: 0; fail: 0; left: 1)
52	[23:05:52] 23:05:52 scap-cdb-rebuild: 100% (in-flight: 0; ok: 1; fail: 0; left: 0)
53	[23:05:52] 23:05:52 Finished scap-cdb-rebuild-canaries (duration: 00m 01s)
54	[23:05:52] 23:05:52 Started sync-wikiversions-canaries
55	[23:05:52] 23:05:52 sync-wikiversions: 0% (ok: 0; fail: 0; left: 1)
56	[23:05:53] 23:05:53 sync-wikiversions: 100% (in-flight: 0; ok: 1; fail: 0; left: 0)
57	[23:05:53] 23:05:53 Finished sync-wikiversions-canaries (duration: 00m 00s)
58	[23:05:53] 23:05:53 Started php-fpm-restarts-canaries
59	[23:05:53] 23:05:53 Running '/usr/local/sbin/restart-php-fpm-all' on 1 host(s)
60	[23:05:53] 23:05:53 php-fpm-restart: 0% (ok: 0; fail: 0; left: 0)
61	[23:05:54] 23:05:54 php-fpm-restart: 100% (in-flight: 0; ok: 1; fail: 0; left: 0)
62	[23:05:54] 23:05:54 Finished php-fpm-restarts-canaries (duration: 00m 00s)
63	[23:05:54] 23:05:54 Finished sync-canaries (duration: 00m 16s)
64	[23:05:54] 23:05:54 Waiting 20 seconds for canary traffic...
65	[23:06:14] 23:06:14 Logstash checker Counted 0 error(s) in the last 20 seconds. OK.
66	[23:06:14] 23:06:14 Started sync-apaches
67	[23:06:14] 23:06:14 sync-apaches: 0% (ok: 0; fail: 0; left: 3)
68	[23:06:45] 23:06:45 sync-apaches: 100% (in-flight: 0; ok: 3; fail: 0; left: 0)
69	[23:06:45] 23:06:45 sync-apaches: 100% (in-flight: 0; ok: 3; fail: 0; left: 0)
70	[23:06:45] 23:06:45 Per-host sync duration: average 16.2s, median 10.8s
71	[23:06:45] 23:06:45 rsync transfer: average 417199 bytes/host, total 1251597 bytes
72	[23:06:45] 23:06:45 Finished sync-apaches (duration: 00m 31s)
73	[23:06:45] 23:06:45 Started scap-cdb-rebuild-prod
74	[23:06:45] 23:06:45 scap-cdb-rebuild: 0% (ok: 0; fail: 0; left: 4)
75	[23:06:47] 23:06:47 scap-cdb-rebuild: 100% (in-flight: 0; ok: 4; fail: 0; left: 0)
76	[23:06:47] 23:06:47 Finished scap-cdb-rebuild-prod (duration: 00m 02s)
77	[23:06:47] 23:06:47 Started sync-wikiversions-prod
78	[23:06:47] 23:06:47 sync-wikiversions: 0% (ok: 0; fail: 0; left: 4)
79	[23:06:48] 23:06:48 sync-wikiversions: 100% (in-flight: 0; ok: 4; fail: 0; left: 0)
80	[23:06:48] 23:06:48 Finished sync-wikiversions-prod (duration: 00m 00s)
81	[23:06:48] 23:06:48 Started php-fpm-restarts-prod
82	[23:06:48] 23:06:48 Running '/usr/local/sbin/restart-php-fpm-all' on 2 host(s)
83	[23:06:48] 23:06:48 php-fpm-restart: 0% (ok: 0; fail: 0; left: 0)
84	[23:06:49] 23:06:49 php-fpm-restart: 100% (in-flight: 0; ok: 2; fail: 0; left: 0)
85	[23:06:49] 23:06:49 Finished php-fpm-restarts-prod (duration: 00m 01s)
86	[23:06:49] 23:06:49 Running purgeMessageBlobStore.php
87	[23:06:53] 23:06:53 Finished scap sync-world: wmf-beta-update-all run (duration: 03m 19s)
88	[23:06:53] + /usr/local/bin/wmf-beta-update-databases.py
89	[23:08:07] aawiki
90	[23:08:07] MediaWiki 1.46.0-alpha Updater
91	[23:08:07]
92	[23:08:07] Your composer.lock file is up to date with current dependencies!
93	[23:08:07] Going to run database updates for aawiki
94	[23:08:07] Depending on the size of your database this may take a while!
95	[23:08:07] Modifying cxl_end_time field of table cx_lists...done.
96	[23:08:07] Modifying cxc_timestamp field of table cx_corpora...done.
97	[23:08:07] Modifying translation_last_updated_timestamp field of table cx_translations...done.
98	[23:08:07] Skipped 172 updates that were already applied.
99	[23:08:07] Purging caches...
100	[23:08:07] Done in 2.7 s.
101	[23:08:07] apiportalwiki
102	[23:08:07] MediaWiki 1.46.0-alpha Updater
103	[23:08:07]
104	[23:08:07] Your composer.lock file is up to date with current dependencies!
105	[23:08:07] Going to run database updates for apiportalwiki
106	[23:08:07] Depending on the size of your database this may take a while!
107	[23:08:07] Modifying gb_id field of table globalblocks...done.
108	[23:08:07] Modifying gb_address field of table globalblocks...done.
109	[23:08:07] Skipped 189 updates that were already applied.
110	[23:08:07] Purging caches...
111	[23:08:07] Done in 3.6 s.
112	[23:08:07] arwiki
113	[23:08:07] MediaWiki 1.46.0-alpha Updater
114	[23:08:07]
115	[23:08:07] Your composer.lock file is up to date with current dependencies!
116	[23:08:07] Going to run database updates for arwiki
117	[23:08:07] Depending on the size of your database this may take a while!
118	[23:08:07] Modifying gb_id field of table globalblocks...done.
119	[23:08:07] Modifying gb_address field of table globalblocks...done.
120	[23:08:07] Modifying cxl_end_time field of table cx_lists...done.
121	[23:08:07] Modifying cxc_timestamp field of table cx_corpora...done.
122	[23:08:07] Modifying translation_last_updated_timestamp field of table cx_translations...done.
123	[23:08:07] Skipped 248 updates that were already applied.
124	[23:08:07] Purging caches...
125	[23:08:07] Done in 3.8 s.
126	[23:08:07] bnwiki
127	[23:08:07] MediaWiki 1.46.0-alpha Updater
128	[23:08:07]
129	[23:08:07] Your composer.lock file is up to date with current dependencies!
130	[23:08:07] Going to run database updates for bnwiki
131	[23:08:07] Depending on the size of your database this may take a while!
132	[23:08:07] Modifying gb_id field of table globalblocks...done.
133	[23:08:07] Modifying gb_address field of table globalblocks...done.
134	[23:08:07] Modifying cxl_end_time field of table cx_lists...done.
135	[23:08:07] Modifying cxc_timestamp field of table cx_corpora...done.
136	[23:08:07] Modifying translation_last_updated_timestamp field of table cx_translations...done.
137	[23:08:07] Skipped 235 updates that were already applied.
138	[23:08:07] Purging caches...
139	[23:08:07] Done in 3.8 s.
140	[23:08:07] cawiki
141	[23:08:07] MediaWiki 1.46.0-alpha Updater
142	[23:08:07]
143	[23:08:07] Your composer.lock file is up to date with current dependencies!
144	[23:08:07] Going to run database updates for cawiki
145	[23:08:07] Depending on the size of your database this may take a while!
146	[23:08:07] Modifying gb_id field of table globalblocks...done.
147	[23:08:07] Modifying gb_address field of table globalblocks...done.
148	[23:08:07] Modifying cxl_end_time field of table cx_lists...done.
149	[23:08:07] Modifying cxc_timestamp field of table cx_corpora...done.
150	[23:08:07] Modifying translation_last_updated_timestamp field of table cx_translations...done.
151	[23:08:07] Skipped 235 updates that were already applied.
152	[23:08:07] Purging caches...
153	[23:08:07] Done in 4.7 s.
154	[23:08:07] commonswiki
155	[23:08:07] MediaWiki 1.46.0-alpha Updater
156	[23:08:07]
157	[23:08:07] Your composer.lock file is up to date with current dependencies!
158	[23:08:07] Going to run database updates for commonswiki
159	[23:08:07] Depending on the size of your database this may take a while!
160	[23:08:07] Modifying gb_id field of table globalblocks...done.
161	[23:08:07] Modifying gb_address field of table globalblocks...done.
162	[23:08:07] Skipped 249 updates that were already applied.
163	[23:08:07] Set the local repo temp zone container to be private.
164	[23:08:07] Purging caches...
165	[23:08:07] Done in 18 s.
166	[23:08:07] crhwiki
167	[23:08:07] MediaWiki 1.46.0-alpha Updater
168	[23:08:07]
169	[23:08:07] Your composer.lock file is up to date with current dependencies!
170	[23:08:07] Going to run database updates for crhwiki
171	[23:08:07] Depending on the size of your database this may take a while!
172	[23:08:07] Modifying gb_id field of table globalblocks...done.
173	[23:08:07] Modifying gb_address field of table globalblocks...done.
174	[23:08:07] Modifying cxl_end_time field of table cx_lists...done.
175	[23:08:07] Modifying cxc_timestamp field of table cx_corpora...done.
176	[23:08:07] Modifying translation_last_updated_timestamp field of table cx_translations...done.
177	[23:08:07] Skipped 217 updates that were already applied.
178	[23:08:07] Purging caches...
179	[23:08:07] Done in 12 s.
180	[23:08:07] cswiki
181	[23:08:07] MediaWiki 1.46.0-alpha Updater
182	[23:08:07]
183	[23:08:07] Your composer.lock file is up to date with current dependencies!
184	[23:08:07] Going to run database updates for cswiki
185	[23:08:07] Depending on the size of your database this may take a while!
186	[23:08:07] Modifying gb_id field of table globalblocks...done.
187	[23:08:07] Modifying gb_address field of table globalblocks...done.
188	[23:08:07] Modifying cxl_end_time field of table cx_lists...done.
189	[23:08:07] Modifying cxc_timestamp field of table cx_corpora...done.
190	[23:08:07] Modifying translation_last_updated_timestamp field of table cx_translations...done.
191	[23:08:07] Skipped 220 updates that were already applied.
192	[23:08:07] Purging caches...
193	[23:08:07] Done in 12 s.
194	[23:08:07] dewiki
195	[23:08:07] MediaWiki 1.46.0-alpha Updater
196	[23:08:07]
197	[23:08:07] Your composer.lock file is up to date with current dependencies!
198	[23:08:07] Going to run database updates for dewiki
199	[23:08:07] Depending on the size of your database this may take a while!
200	[23:08:07] Modifying gb_id field of table globalblocks...done.
201	[23:08:07] Modifying gb_address field of table globalblocks...done.
202	[23:08:07] Modifying cxl_end_time field of table cx_lists...done.
203	[23:08:07] Modifying cxc_timestamp field of table cx_corpora...done.
204	[23:08:07] Modifying translation_last_updated_timestamp field of table cx_translations...done.
205	[23:08:07] Skipped 243 updates that were already applied.
206	[23:08:07] Purging caches...
207	[23:08:07] Done in 12 s.
208	[23:08:07] dewiktionary
209	[23:08:07] MediaWiki 1.46.0-alpha Updater
210	[23:08:07]
211	[23:08:07] Your composer.lock file is up to date with current dependencies!
212	[23:08:07] Going to run database updates for dewiktionary
213	[23:08:07] Depending on the size of your database this may take a while!
214	[23:08:07] Modifying gb_id field of table globalblocks...done.
215	[23:08:07] Modifying gb_address field of table globalblocks...done.
216	[23:08:07] Skipped 203 updates that were already applied.
217	[23:08:07] Purging caches...
218	[23:08:07] Done in 3.2 s.
219	[23:08:07] enwiki
220	[23:08:07] MediaWiki 1.46.0-alpha Updater
221	[23:08:07]
222	[23:08:07] Your composer.lock file is up to date with current dependencies!
223	[23:08:07] Going to run database updates for enwiki
224	[23:08:07] Depending on the size of your database this may take a while!
225	[23:08:07] Modifying gb_id field of table globalblocks...done.
226	[23:08:07] Modifying gb_address field of table globalblocks...done.
227	[23:08:07] Modifying cxl_end_time field of table cx_lists...done.
228	[23:08:07] Modifying cxc_timestamp field of table cx_corpora...done.
229	[23:08:07] Modifying translation_last_updated_timestamp field of table cx_translations...done.
230	[23:08:07] Skipped 254 updates that were already applied.
231	[23:08:07] Purging caches...
232	[23:08:07] Done in 12 s.
233	[23:08:07] enwikibooks
234	[23:08:07] MediaWiki 1.46.0-alpha Updater
235	[23:08:07]
236	[23:08:07] Your composer.lock file is up to date with current dependencies!
237	[23:08:07] Going to run database updates for enwikibooks
238	[23:08:07] Depending on the size of your database this may take a while!
239	[23:08:07] Modifying gb_id field of table globalblocks...done.
240	[23:08:07] Modifying gb_address field of table globalblocks...done.
241	[23:08:07] Skipped 202 updates that were already applied.
242	[23:08:07] Purging caches...
243	[23:08:07] Done in 2.6 s.
244	[23:08:07] enwikinews
245	[23:08:07] MediaWiki 1.46.0-alpha Updater
246	[23:08:07]
247	[23:08:07] Your composer.lock file is up to date with current dependencies!
248	[23:08:07] Going to run database updates for enwikinews
249	[23:08:07] Depending on the size of your database this may take a while!
250	[23:08:07] Modifying gb_id field of table globalblocks...done.
251	[23:08:07] Modifying gb_address field of table globalblocks...done.
252	[23:08:07] Skipped 209 updates that were already applied.
253	[23:08:07] Purging caches...
254	[23:08:07] Done in 2.7 s.
255	[23:08:07] enwikiquote
256	[23:08:07] MediaWiki 1.46.0-alpha Updater
257	[23:08:07]
258	[23:08:07] Your composer.lock file is up to date with current dependencies!
259	[23:08:07] Going to run database updates for enwikiquote
260	[23:08:07] Depending on the size of your database this may take a while!
261	[23:08:07] Modifying gb_id field of table globalblocks...done.
262	[23:08:07] Modifying gb_address field of table globalblocks...done.
263	[23:08:07] Skipped 191 updates that were already applied.
264	[23:08:07] Purging caches...
265	[23:08:07] Done in 2.5 s.
266	[23:08:07] enwikisource
267	[23:08:07] MediaWiki 1.46.0-alpha Updater
268	[23:08:07]
269	[23:08:07] Your composer.lock file is up to date with current dependencies!
270	[23:08:07] Going to run database updates for enwikisource
271	[23:08:07] Depending on the size of your database this may take a while!
272	[23:08:07] Modifying gb_id field of table globalblocks...done.
273	[23:08:07] Modifying gb_address field of table globalblocks...done.
274	[23:08:07] Skipped 194 updates that were already applied.
275	[23:08:07] Purging caches...
276	[23:08:07] Done in 2.5 s.
277	[23:08:07] enwikiversity
278	[23:08:07] MediaWiki 1.46.0-alpha Updater
279	[23:08:07]
280	[23:08:07] Your composer.lock file is up to date with current dependencies!
281	[23:08:07] Going to run database updates for enwikiversity
282	[23:08:07] Depending on the size of your database this may take a while!
283	[23:08:07] Modifying gb_id field of table globalblocks...done.
284	[23:08:07] Modifying gb_address field of table globalblocks...done.
285	[23:08:07] Skipped 191 updates that were already applied.
286	[23:08:07] Purging caches...
287	[23:08:07] Done in 2.4 s.
288	[23:08:07] enwikivoyage
289	[23:08:07] MediaWiki 1.46.0-alpha Updater
290	[23:08:07]
291	[23:08:07] Your composer.lock file is up to date with current dependencies!
292	[23:08:07] Going to run database updates for enwikivoyage
293	[23:08:07] Depending on the size of your database this may take a while!
294	[23:08:07] Modifying gb_id field of table globalblocks...done.
295	[23:08:07] Modifying gb_address field of table globalblocks...done.
296	[23:08:07] Modifying cxl_end_time field of table cx_lists...done.
297	[23:08:07] Modifying cxc_timestamp field of table cx_corpora...done.
298	[23:08:07] Modifying translation_last_updated_timestamp field of table cx_translations...done.
299	[23:08:07] Skipped 215 updates that were already applied.
300	[23:08:07] Purging caches...
301	[23:09:07] Done in 2.8 s.
302	[23:09:07] enwiktionary
303	[23:09:07] MediaWiki 1.46.0-alpha Updater
304	[23:09:07]
305	[23:09:07] Your composer.lock file is up to date with current dependencies!
306	[23:09:07] Going to run database updates for enwiktionary
307	[23:09:07] Depending on the size of your database this may take a while!
308	[23:09:07] Modifying gb_id field of table globalblocks...done.
309	[23:09:07] Modifying gb_address field of table globalblocks...done.
310	[23:09:07] Skipped 199 updates that were already applied.
311	[23:09:07] Purging caches...
312	[23:09:07] Done in 2.5 s.
313	[23:09:07] eowiki
314	[23:09:07] MediaWiki 1.46.0-alpha Updater
315	[23:09:07]
316	[23:09:07] Your composer.lock file is up to date with current dependencies!
317	[23:09:07] Going to run database updates for eowiki
318	[23:09:07] Depending on the size of your database this may take a while!
319	[23:09:07] Modifying gb_id field of table globalblocks...done.
320	[23:09:07] Modifying gb_address field of table globalblocks...done.
321	[23:09:07] Modifying cxl_end_time field of table cx_lists...done.
322	[23:09:07] Modifying cxc_timestamp field of table cx_corpora...done.
323	[23:09:07] Modifying translation_last_updated_timestamp field of table cx_translations...done.
324	[23:09:07] Skipped 243 updates that were already applied.
325	[23:09:07] Purging caches...
326	[23:09:07] Done in 3.7 s.
327	[23:09:07] eswiki
328	[23:09:07] MediaWiki 1.46.0-alpha Updater
329	[23:09:07]
330	[23:09:07] Your composer.lock file is up to date with current dependencies!
331	[23:09:07] Going to run database updates for eswiki
332	[23:09:07] Depending on the size of your database this may take a while!
333	[23:09:07] Modifying gb_id field of table globalblocks...done.
334	[23:09:07] Modifying gb_address field of table globalblocks...done.
335	[23:09:07] Modifying cxl_end_time field of table cx_lists...done.
336	[23:09:07] Modifying cxc_timestamp field of table cx_corpora...done.
337	[23:09:07] Modifying translation_last_updated_timestamp field of table cx_translations...done.
338	[23:09:07] Skipped 222 updates that were already applied.
339	[23:09:07] Purging caches...
340	[23:09:07] Done in 3.8 s.
341	[23:09:07] eswikibooks
342	[23:09:07] MediaWiki 1.46.0-alpha Updater
343	[23:09:07]
344	[23:09:07] Your composer.lock file is up to date with current dependencies!
345	[23:09:07] Going to run database updates for eswikibooks
346	[23:09:07] Depending on the size of your database this may take a while!
347	[23:09:07] Modifying gb_id field of table globalblocks...done.
348	[23:09:07] Modifying gb_address field of table globalblocks...done.
349	[23:09:07] Skipped 195 updates that were already applied.
350	[23:09:07] Purging caches...
351	[23:09:07] Done in 2.6 s.
352	[23:09:07] fawiki
353	[23:09:07] MediaWiki 1.46.0-alpha Updater
354	[23:09:07]
355	[23:09:07] Your composer.lock file is up to date with current dependencies!
356	[23:09:07] Going to run database updates for fawiki
357	[23:09:07] Depending on the size of your database this may take a while!
358	[23:09:07] Modifying gb_id field of table globalblocks...done.
359	[23:09:07] Modifying gb_address field of table globalblocks...done.
360	[23:09:07] Modifying cxl_end_time field of table cx_lists...done.
361	[23:09:07] Modifying cxc_timestamp field of table cx_corpora...done.
362	[23:09:07] Modifying translation_last_updated_timestamp field of table cx_translations...done.
363	[23:09:07] Skipped 248 updates that were already applied.
364	[23:09:07] Purging caches...
365	[23:09:07] Done in 3.9 s.
366	[23:09:07] foundationwiki
367	[23:09:07] MediaWiki 1.46.0-alpha Updater
368	[23:09:07]
369	[23:09:07] Your composer.lock file is up to date with current dependencies!
370	[23:09:07] Going to run database updates for foundationwiki
371	[23:09:07] Depending on the size of your database this may take a while!
372	[23:09:07] Modifying gb_id field of table globalblocks...done.
373	[23:09:07] Modifying gb_address field of table globalblocks...done.
374	[23:09:07] Skipped 217 updates that were already applied.
375	[23:09:07] Purging caches...
376	[23:09:07] Done in 2.8 s.
377	[23:09:07] frwiki
378	[23:09:07] MediaWiki 1.46.0-alpha Updater
379	[23:09:07]
380	[23:09:07] Your composer.lock file is up to date with current dependencies!
381	[23:09:07] Going to run database updates for frwiki
382	[23:09:07] Depending on the size of your database this may take a while!
383	[23:09:07] Modifying gb_id field of table globalblocks...done.
384	[23:09:07] Modifying gb_address field of table globalblocks...done.
385	[23:09:07] Modifying cxl_end_time field of table cx_lists...done.
386	[23:09:07] Modifying cxc_timestamp field of table cx_corpora...done.
387	[23:09:07] Modifying translation_last_updated_timestamp field of table cx_translations...done.
388	[23:09:07] Skipped 220 updates that were already applied.
389	[23:09:07] Purging caches...
390	[23:09:07] Done in 12 s.
391	[23:09:07] frwiktionary
392	[23:09:07] MediaWiki 1.46.0-alpha Updater
393	[23:09:07]
394	[23:09:07] Your composer.lock file is up to date with current dependencies!
395	[23:09:07] Going to run database updates for frwiktionary
396	[23:09:07] Depending on the size of your database this may take a while!
397	[23:09:07] Modifying gb_id field of table globalblocks...done.
398	[23:09:07] Modifying gb_address field of table globalblocks...done.
399	[23:09:07] Skipped 224 updates that were already applied.
400	[23:09:07] Purging caches...
401	[23:09:07] Done in 2.8 s.
402	[23:09:07] hewiki
403	[23:09:07] MediaWiki 1.46.0-alpha Updater
404	[23:09:07]
405	[23:09:07] Your composer.lock file is up to date with current dependencies!
406	[23:09:07] Going to run database updates for hewiki
407	[23:09:07] Depending on the size of your database this may take a while!
408	[23:09:07] Modifying gb_id field of table globalblocks...done.
409	[23:09:07] Modifying gb_address field of table globalblocks...done.
410	[23:09:07] Modifying cxl_end_time field of table cx_lists...done.
411	[23:09:07] Modifying cxc_timestamp field of table cx_corpora...done.
412	[23:09:07] Modifying translation_last_updated_timestamp field of table cx_translations...done.
413	[23:09:07] Skipped 237 updates that were already applied.
414	[23:09:07] Purging caches...
415	[23:09:07] Done in 12 s.
416	[23:09:07] hewiktionary
417	[23:09:07] MediaWiki 1.46.0-alpha Updater
418	[23:09:07]
419	[23:09:07] Your composer.lock file is up to date with current dependencies!
420	[23:09:07] Going to run database updates for hewiktionary
421	[23:09:07] Depending on the size of your database this may take a while!
422	[23:09:07] Modifying gb_id field of table globalblocks...done.
423	[23:09:07] Modifying gb_address field of table globalblocks...done.
424	[23:09:07] Skipped 195 updates that were already applied.
425	[23:09:07] Purging caches...
426	[23:09:07] Done in 2.6 s.
427	[23:09:07] hiwiki
428	[23:09:07] MediaWiki 1.46.0-alpha Updater
429	[23:09:07]
430	[23:09:07] Your composer.lock file is up to date with current dependencies!
431	[23:09:07] Going to run database updates for hiwiki
432	[23:09:07] Depending on the size of your database this may take a while!
433	[23:09:07] Modifying gb_id field of table globalblocks...done.
434	[23:09:07] Modifying gb_address field of table globalblocks...done.
435	[23:09:07] Modifying cxl_end_time field of table cx_lists...done.
436	[23:09:07] Modifying cxc_timestamp field of table cx_corpora...done.
437	[23:09:07] Modifying translation_last_updated_timestamp field of table cx_translations...done.
438	[23:09:07] Skipped 231 updates that were already applied.
439	[23:09:07] Purging caches...
440	[23:09:07] Done in 5.1 s.
441	[23:09:07] incubatorwiki
442	[23:09:07] MediaWiki 1.46.0-alpha Updater
443	[23:09:07]
444	[23:09:07] Your composer.lock file is up to date with current dependencies!
445	[23:09:07] Going to run database updates for incubatorwiki
446	[23:09:07] Depending on the size of your database this may take a while!
447	[23:09:07] Modifying gb_id field of table globalblocks...done.
448	[23:09:07] Modifying gb_address field of table globalblocks...done.
449	[23:09:07] Skipped 218 updates that were already applied.
450	[23:09:07] Purging caches...
451	[23:09:07] Done in 2.8 s.
452	[23:09:07] jawiki
453	[23:09:07] MediaWiki 1.46.0-alpha Updater
454	[23:09:07]
455	[23:09:07] Your composer.lock file is up to date with current dependencies!
456	[23:09:07] Going to run database updates for jawiki
457	[23:09:07] Depending on the size of your database this may take a while!
458	[23:09:07] Modifying gb_id field of table globalblocks...done.
459	[23:09:07] Modifying gb_address field of table globalblocks...done.
460	[23:09:07] Modifying cxl_end_time field of table cx_lists...done.
461	[23:09:07] Modifying cxc_timestamp field of table cx_corpora...done.
462	[23:09:07] Modifying translation_last_updated_timestamp field of table cx_translations...done.
463	[23:09:07] Skipped 219 updates that were already applied.
464	[23:09:07] Purging caches...
465	[23:09:07] Done in 12 s.
466	[23:09:07] kowiki
467	[23:09:07] MediaWiki 1.46.0-alpha Updater
468	[23:09:07]
469	[23:09:07] Your composer.lock file is up to date with current dependencies!
470	[23:09:07] Going to run database updates for kowiki
471	[23:09:07] Depending on the size of your database this may take a while!
472	[23:09:07] Modifying gb_id field of table globalblocks...done.
473	[23:09:07] Modifying gb_address field of table globalblocks...done.
474	[23:09:07] Modifying cxl_end_time field of table cx_lists...done.
475	[23:09:07] Modifying cxc_timestamp field of table cx_corpora...done.
476	[23:09:07] Modifying translation_last_updated_timestamp field of table cx_translations...done.
477	[23:09:07] Skipped 222 updates that were already applied.
478	[23:09:07] Purging caches...
479	[23:09:07] Done in 3.7 s.
480	[23:09:07] loginwiki
481	[23:09:07] MediaWiki 1.46.0-alpha Updater
482	[23:09:07]
483	[23:09:07] Your composer.lock file is up to date with current dependencies!
484	[23:09:07] Going to run database updates for loginwiki
485	[23:09:07] Depending on the size of your database this may take a while!
486	[23:09:07] Modifying gb_id field of table globalblocks...done.
487	[23:09:07] Modifying gb_address field of table globalblocks...done.
488	[23:09:07] Skipped 134 updates that were already applied.
489	[23:09:07] Purging caches...
490	[23:09:07] Done in 1.8 s.
491	[23:09:07] metawiki
492	[23:09:07] MediaWiki 1.46.0-alpha Updater
493	[23:09:07]
494	[23:09:07] Your composer.lock file is up to date with current dependencies!
495	[23:09:07] Going to run database updates for metawiki
496	[23:09:07] Depending on the size of your database this may take a while!
497	[23:09:07] Modifying gb_id field of table globalblocks...done.
498	[23:09:07] Modifying gb_address field of table globalblocks...done.
499	[23:09:07] Skipped 221 updates that were already applied.
500	[23:09:07] Purging caches...
501	[23:09:07] Done in 2.8 s.
502	[23:09:07] nlwiki
503	[23:09:07] MediaWiki 1.46.0-alpha Updater
504	[23:09:07]
505	[23:09:07] Your composer.lock file is up to date with current dependencies!
506	[23:09:07] Going to run database updates for nlwiki
507	[23:09:07] Depending on the size of your database this may take a while!
508	[23:09:07] Modifying gb_id field of table globalblocks...done.
509	[23:09:07] Modifying gb_address field of table globalblocks...done.
510	[23:09:07] Modifying cxl_end_time field of table cx_lists...done.
511	[23:09:07] Modifying cxc_timestamp field of table cx_corpora...done.
512	[23:09:07] Modifying translation_last_updated_timestamp field of table cx_translations...done.
513	[23:09:07] Skipped 222 updates that were already applied.
514	[23:09:07] Purging caches...
515	[23:10:05] Done in 4.0 s.
516	[23:10:05] plwikivoyage
517	[23:10:05] MediaWiki 1.46.0-alpha Updater
518	[23:10:05]
519	[23:10:05] Your composer.lock file is up to date with current dependencies!
520	[23:10:05] Going to run database updates for plwikivoyage
521	[23:10:05] Depending on the size of your database this may take a while!
522	[23:10:05] Modifying gb_id field of table globalblocks...done.
523	[23:10:05] Modifying gb_address field of table globalblocks...done.
524	[23:10:05] Modifying cxl_end_time field of table cx_lists...done.
525	[23:10:05] Modifying cxc_timestamp field of table cx_corpora...done.
526	[23:10:05] Modifying translation_last_updated_timestamp field of table cx_translations...done.
527	[23:10:05] Skipped 213 updates that were already applied.
528	[23:10:05] Purging caches...
529	[23:10:05] Done in 3.6 s.
530	[23:10:05] ruwiki
531	[23:10:05] MediaWiki 1.46.0-alpha Updater
532	[23:10:05]
533	[23:10:05] Your composer.lock file is up to date with current dependencies!
534	[23:10:05] Going to run database updates for ruwiki
535	[23:10:05] Depending on the size of your database this may take a while!
536	[23:10:05] Modifying gb_id field of table globalblocks...done.
537	[23:10:05] Modifying gb_address field of table globalblocks...done.
538	[23:10:05] Modifying cxl_end_time field of table cx_lists...done.
539	[23:10:05] Modifying cxc_timestamp field of table cx_corpora...done.
540	[23:10:05] Modifying translation_last_updated_timestamp field of table cx_translations...done.
541	[23:10:05] Skipped 248 updates that were already applied.
542	[23:10:05] Purging caches...
543	[23:10:05] Done in 12 s.
544	[23:10:05] simplewiki
545	[23:10:05] MediaWiki 1.46.0-alpha Updater
546	[23:10:05]
547	[23:10:05] Your composer.lock file is up to date with current dependencies!
548	[23:10:05] Going to run database updates for simplewiki
549	[23:10:05] Depending on the size of your database this may take a while!
550	[23:10:05] Modifying gb_id field of table globalblocks...done.
551	[23:10:05] Modifying gb_address field of table globalblocks...done.
552	[23:10:05] Modifying cxl_end_time field of table cx_lists...done.
553	[23:10:05] Modifying cxc_timestamp field of table cx_corpora...done.
554	[23:10:05] Modifying translation_last_updated_timestamp field of table cx_translations...done.
555	[23:10:05] Skipped 220 updates that were already applied.
556	[23:10:05] Purging caches...
557	[23:10:05] Done in 3.4 s.
558	[23:10:05] sqwiki
559	[23:10:05] MediaWiki 1.46.0-alpha Updater
560	[23:10:05]
561	[23:10:05] Your composer.lock file is up to date with current dependencies!
562	[23:10:05] Going to run database updates for sqwiki
563	[23:10:05] Depending on the size of your database this may take a while!
564	[23:10:05] Modifying gb_id field of table globalblocks...done.
565	[23:10:05] Modifying gb_address field of table globalblocks...done.
566	[23:10:05] Modifying cxl_end_time field of table cx_lists...done.
567	[23:10:05] Modifying cxc_timestamp field of table cx_corpora...done.
568	[23:10:05] Modifying translation_last_updated_timestamp field of table cx_translations...done.
569	[23:10:05] Skipped 233 updates that were already applied.
570	[23:10:05] Purging caches...
571	[23:10:05] Done in 3.7 s.
572	[23:10:05] srwiki
573	[23:10:05] MediaWiki 1.46.0-alpha Updater
574	[23:10:05]
575	[23:10:05] Your composer.lock file is up to date with current dependencies!
576	[23:10:05] Going to run database updates for srwiki
577	[23:10:05] Depending on the size of your database this may take a while!
578	[23:10:05] Modifying gb_id field of table globalblocks...done.
579	[23:10:05] Modifying gb_address field of table globalblocks...done.
580	[23:10:05] Modifying cxl_end_time field of table cx_lists...done.
581	[23:10:05] Modifying cxc_timestamp field of table cx_corpora...done.
582	[23:10:05] Modifying translation_last_updated_timestamp field of table cx_translations...done.
583	[23:10:05] Skipped 222 updates that were already applied.
584	[23:10:05] Purging caches...
585	[23:10:05] Done in 4.5 s.
586	[23:10:05] svwiki
587	[23:10:05] MediaWiki 1.46.0-alpha Updater
588	[23:10:05]
589	[23:10:05] Your composer.lock file is up to date with current dependencies!
590	[23:10:05] Going to run database updates for svwiki
591	[23:10:05] Depending on the size of your database this may take a while!
592	[23:10:05] Modifying gb_id field of table globalblocks...done.
593	[23:10:05] Modifying gb_address field of table globalblocks...done.
594	[23:10:05] Modifying cxl_end_time field of table cx_lists...done.
595	[23:10:05] Modifying cxc_timestamp field of table cx_corpora...done.
596	[23:10:05] Modifying translation_last_updated_timestamp field of table cx_translations...done.
597	[23:10:05] Skipped 237 updates that were already applied.
598	[23:10:05] Purging caches...
599	[23:10:05] Done in 3.8 s.
600	[23:10:05] test2wiki
601	[23:10:05] MediaWiki 1.46.0-alpha Updater
602	[23:10:05]
603	[23:10:05] Your composer.lock file is up to date with current dependencies!
604	[23:10:05] Going to run database updates for test2wiki
605	[23:10:05] Depending on the size of your database this may take a while!
606	[23:10:05] Modifying gb_id field of table globalblocks...done.
607	[23:10:05] Modifying gb_address field of table globalblocks...done.
608	[23:10:05] Skipped 233 updates that were already applied.
609	[23:10:05] Purging caches...
610	[23:10:05] Done in 2.9 s.
611	[23:10:05] testwiki
612	[23:10:05] MediaWiki 1.46.0-alpha Updater
613	[23:10:05]
614	[23:10:05] Your composer.lock file is up to date with current dependencies!
615	[23:10:05] Going to run database updates for testwiki
616	[23:10:05] Depending on the size of your database this may take a while!
617	[23:10:05] Modifying gb_id field of table globalblocks...done.
618	[23:10:05] Modifying gb_address field of table globalblocks...done.
619	[23:10:05] Modifying cxl_end_time field of table cx_lists...done.
620	[23:10:05] Modifying cxc_timestamp field of table cx_corpora...done.
621	[23:10:05] Modifying translation_last_updated_timestamp field of table cx_translations...done.
622	[23:10:05] Skipped 275 updates that were already applied.
623	[23:10:05] Purging caches...
624	[23:10:05] Done in 5.6 s.
625	[23:10:05] ukwiki
626	[23:10:05] MediaWiki 1.46.0-alpha Updater
627	[23:10:05]
628	[23:10:05] Your composer.lock file is up to date with current dependencies!
629	[23:10:05] Going to run database updates for ukwiki
630	[23:10:05] Depending on the size of your database this may take a while!
631	[23:10:05] Modifying gb_id field of table globalblocks...done.
632	[23:10:05] Modifying gb_address field of table globalblocks...done.
633	[23:10:05] Modifying cxl_end_time field of table cx_lists...done.
634	[23:10:05] Modifying cxc_timestamp field of table cx_corpora...done.
635	[23:10:05] Modifying translation_last_updated_timestamp field of table cx_translations...done.
636	[23:10:05] Skipped 233 updates that were already applied.
637	[23:10:05] Purging caches...
638	[23:10:05] Done in 12 s.
639	[23:10:05] viwiki
640	[23:10:05] MediaWiki 1.46.0-alpha Updater
641	[23:10:05]
642	[23:10:05] Your composer.lock file is up to date with current dependencies!
643	[23:10:05] Going to run database updates for viwiki
644	[23:10:05] Depending on the size of your database this may take a while!
645	[23:10:05] Modifying gb_id field of table globalblocks...done.
646	[23:10:05] Modifying gb_address field of table globalblocks...done.
647	[23:10:05] Modifying cxl_end_time field of table cx_lists...done.
648	[23:10:05] Modifying cxc_timestamp field of table cx_corpora...done.
649	[23:10:05] Modifying translation_last_updated_timestamp field of table cx_translations...done.
650	[23:10:05] Skipped 219 updates that were already applied.
651	[23:10:05] Purging caches...
652	[23:10:05] Done in 4.0 s.
653	[23:10:05] votewiki
654	[23:10:05] MediaWiki 1.46.0-alpha Updater
655	[23:10:05]
656	[23:10:05] Your composer.lock file is up to date with current dependencies!
657	[23:10:05] Going to run database updates for votewiki
658	[23:10:05] Depending on the size of your database this may take a while!
659	[23:10:05] Skipped 113 updates that were already applied.
660	[23:10:05] Purging caches...
661	[23:10:05] Done in 1.4 s.
662	[23:10:05] wikidatawiki
663	[23:10:05]
664	[23:10:05] Your composer.lock file is up to date with current dependencies!
665	[23:10:05] Going to run database updates for votewiki
666	[23:10:05] Depending on the size of your database this may take a while!
667	[23:10:05] Skipped 113 updates that were already applied.
668	[23:10:05] Purging caches...
669	[23:10:05] Done in 1.4 s.
670	[23:10:05] wikidatawiki
671	[23:10:05] MediaWiki 1.46.0-alpha Updater
672	[23:10:05]
673	[23:10:05] Your composer.lock file is up to date with current dependencies!
674	[23:10:05] Going to run database updates for wikidatawiki
675	[23:10:05] Depending on the size of your database this may take a while!
676	[23:10:05] Modifying gb_id field of table globalblocks...done.
677	[23:10:05] Modifying gb_address field of table globalblocks...done.
678	[23:10:05] Skipped 259 updates that were already applied.
679	[23:10:05] Purging caches...
680	[23:10:05] Done in 3.2 s.
681	[23:10:05] zhwiki
682	[23:10:05] MediaWiki 1.46.0-alpha Updater
683	[23:10:05]
684	[23:10:05] Your composer.lock file is up to date with current dependencies!
685	[23:10:05] Going to run database updates for zhwiki
686	[23:10:05] Depending on the size of your database this may take a while!
687	[23:10:05] Modifying gb_id field of table globalblocks...done.
688	[23:10:05] Modifying gb_address field of table globalblocks...done.
689	[23:10:05] Modifying cxl_end_time field of table cx_lists...done.
690	[23:10:05] Modifying cxc_timestamp field of table cx_corpora...done.
691	[23:10:05] Modifying translation_last_updated_timestamp field of table cx_translations...done.
692	[23:10:05] Skipped 243 updates that were already applied.
693	[23:10:05] Purging caches...
694	[23:10:05] Done in 14 s.
695	[23:10:05] zhwikivoyage
696	[23:10:05] MediaWiki 1.46.0-alpha Updater
697	[23:10:05]
698	[23:10:05] Your composer.lock file is up to date with current dependencies!
699	[23:10:05] Going to run database updates for zhwikivoyage
700	[23:10:05] Depending on the size of your database this may take a while!
701	[23:10:05] Modifying gb_id field of table globalblocks...done.
702	[23:10:05] Modifying gb_address field of table globalblocks...done.
703	[23:10:05] Modifying cxl_end_time field of table cx_lists...done.
704	[23:10:05] Modifying cxc_timestamp field of table cx_corpora...done.
705	[23:10:05] Modifying translation_last_updated_timestamp field of table cx_translations...done.
706	[23:10:05] Skipped 215 updates that were already applied.
707	[23:10:05] Purging caches...
708	[23:10:05] Done in 5.1 s.

Mentioned in SAL (#wikimedia-releng) [2026-04-23T23:57:58Z] <bd808> Set profile::beta::autoupdater::run_updater: true for deployment-deploy04 via Horizon (T256168)

https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/12f66926a45f0277503cc7d3e903974ca46681aa%5E%21/#F0

diff --git a/deployment-prep/deployment-deploy04.deployment-prep.eqiad1.wikimedia.cloud.yaml b/deployment-prep/deployment-deploy04.deployment-prep.eqiad1.wikimedia.cloud.yaml
index 8c8d19a..1dfd2d2 100644
--- a/deployment-prep/deployment-deploy04.deployment-prep.eqiad1.wikimedia.cloud.yaml
+++ b/deployment-prep/deployment-deploy04.deployment-prep.eqiad1.wikimedia.cloud.yaml

@@ -1,4 +1,5 @@
 profile::base::labs::cleanup_puppet_client_bucket: true
+profile::beta::autoupdater::run_updater: true
 profile::ci::slave::labs::common::manage_srv: false
 profile::kubernetes::client::version: '1.23'
 profile::mediawiki::scap_client::is_master: true

Next I forced a puppet run on deployment-deploy04 to make sure the timer was enabled:

bd808@deployment-deploy04:~$ sudo -i run-puppet-agent
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for deployment-deploy04.deployment-prep.eqiad1.wikimedia.cloud
Info: Applying configuration version '(1dd499e77f) gitpuppet - beta: Add a wmf-beta-update-all timer and script'
Notice: /Stage[main]/Ssh::Client/File[/etc/ssh/ssh_known_hosts]/content:
--- /etc/ssh/ssh_known_hosts    2026-04-23 23:34:33.389165381 +0000
+++ /tmp/puppet-file20260423-3075096-3g6dvp     2026-04-23 23:57:53.607436243 +0000
@@ -1,6 +1,3 @@
-# HEADER: This file was autogenerated at 2026-04-23 23:34:33 +0000
-# HEADER: by puppet.  While it can still be managed manually, it
-# HEADER: is definitely not recommended.
 # This file is managed by puppet
 deployment-acme-chief05.deployment-prep.eqiad1.wikimedia.cloud,deployment-acme-chief05,172.16.4.3,fe80::f816:3eff:fe8e:e4dc ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNKxFW8xm8BypAq0/myF9ofyBgQ27G0WpM5Kc2bTm/nsiDgyJKc+QeN39JCMUCpiw6ggvV6EedLNLi6SjLxSCdE=
 deployment-acme-chief06.deployment-prep.eqiad1.wikimedia.cloud,deployment-acme-chief06,172.16.2.219,fe80::f816:3eff:fef4:9016 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLXu9iyX0uKDeAgBHri6gI+sn4UZMmQItQ4dPCLRNqtuNJSLPbT1urmCMfCLY15mDlxyoet3H+FBKKj4uW3BfHY=
@@ -62,4 +59,3 @@
 deployment-webperf22.deployment-prep.eqiad1.wikimedia.cloud,deployment-webperf22,172.16.5.13,fe80::f816:3eff:fece:d2f6 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNOz0NEzFJSCeQDayftQjduYJW0fxRCyMtVas4zr5fQDVtdlidlWhqIqUZZeZrb2dcK8A1W5MKYlvmGTDSDZpxo=
 deployment-wikifeeds02.deployment-prep.eqiad1.wikimedia.cloud,deployment-wikifeeds02,172.16.4.35,fe80::f816:3eff:fe6a:3c52 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJRov6bw0g/1fTFuFysHF1jEshVWHHbAU6ZJAPKRxE+sKuIBQn3UqGzUgay1DYed+TPErViVwuDasJwv5/S2OJE=
 deployment-zookeeper-3.deployment-prep.eqiad1.wikimedia.cloud,deployment-zookeeper-3,172.16.6.11,fe80::f816:3eff:feed:a89d ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAn5JXibRrHVMLH63TyxRHP9ZuJ8cRoTRoCoOfdUZ/zwDMirB1JZBNA5b04FpCdbs6y5X/QDTMS3kOEaBSsAYVo=
-gerrit.wikimedia.org,gerrit.discovery.wmnet,208.80.153.116,2620:0:860:4:208:80:153:116,208.80.154.225,2620:0:861:ed1a::2,208.80.153.225,2620:0:860:ed1a::2,185.15.59.225,2a02:ec80:300:ed1a::2,198.35.26.225,2620:0:863:ed1a::2,103.102.166.225,2001:df2:e500:ed1a::2,185.15.58.225,2a02:ec80:600:ed1a::2,195.200.68.225,2a02:ec80:700:ed1a::2 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQCF8pwFLehzCXhbF1jfHWtd9d1LFq2NirplEBQYs7AOrGwQ/6ZZI0gvZFYiEiaw1o+F1CMfoHdny1VfWOJF3mJ1y9QMKAacc8/Z3tG39jBKRQCuxmYLO1SWymv7/Uvx9WQlkNRoTdTTa9OJFy6UqvLQEXKYaokfMIUHZ+oVFf1CgQ==

Notice: /Stage[main]/Ssh::Client/File[/etc/ssh/ssh_known_hosts]/content:

Notice: /Stage[main]/Ssh::Client/File[/etc/ssh/ssh_known_hosts]/content: content changed '{sha256}4bb76f8b7f3391e274cf567f31e88d572c2943b0ec3118af9126b24309214f77' to '{sha256}e63cf8c3b094438409be5f595d6be261b99d11faf1241494d121ee5d2d3f0d0b' (corrective)
Notice: /Stage[main]/Profile::Ci::Slave::Labs::Common/Profile::Gerrit::Sshkey[gerrit]/Sshkey[gerrit.wikimedia.org@ssh-rsa]/ensure: created (corrective)
Info: Computing checksum on file /etc/ssh/ssh_known_hosts
Notice: /Stage[main]/Beta::Autoupdater/Systemd::Timer::Job[wmf-beta-update-all]/Systemd::Unit[wmf-beta-update-all.service]/File[/lib/systemd/system/wmf-beta-update-all.service]/ensure: defined content as '{sha256}f83a1992a5786c473005ba53fbaa196b705e34530e377cfaee78a4390b46693c'
Info: /Stage[main]/Beta::Autoupdater/Systemd::Timer::Job[wmf-beta-update-all]/Systemd::Unit[wmf-beta-update-all.service]/File[/lib/systemd/system/wmf-beta-update-all.service]: Scheduling refresh of Exec[systemd daemon-reload for wmf-beta-update-all.service (wmf-beta-update-all.service)]
Notice: /Stage[main]/Beta::Autoupdater/Systemd::Timer::Job[wmf-beta-update-all]/Systemd::Unit[wmf-beta-update-all.service]/Exec[systemd daemon-reload for wmf-beta-update-all.service (wmf-beta-update-all.service)]: Triggered 'refresh' from 1 event
Notice: /Stage[main]/Beta::Autoupdater/Systemd::Timer::Job[wmf-beta-update-all]/Systemd::Syslog[wmf-beta-update-all]/File[/var/log/wmf-beta-update-all]/ensure: created
Notice: /Stage[main]/Beta::Autoupdater/Systemd::Timer::Job[wmf-beta-update-all]/Systemd::Syslog[wmf-beta-update-all]/Rsyslog::Conf[wmf-beta-update-all]/File[/etc/rsyslog.d/40-wmf-beta-update-all.conf]/ensure: defined content as '{sha256}35a2d3c71369c0701b06fc7aeaa4ad32715ebaf0e933ed989efdabd48be8ae87'
Info: /Stage[main]/Beta::Autoupdater/Systemd::Timer::Job[wmf-beta-update-all]/Systemd::Syslog[wmf-beta-update-all]/Rsyslog::Conf[wmf-beta-update-all]/File[/etc/rsyslog.d/40-wmf-beta-update-all.conf]: Scheduling refresh of Service[rsyslog]
Notice: /Stage[main]/Beta::Autoupdater/Systemd::Timer::Job[wmf-beta-update-all]/Systemd::Syslog[wmf-beta-update-all]/Logrotate::Conf[wmf-beta-update-all]/File[/etc/logrotate.d/wmf-beta-update-all]/ensure: defined content as '{sha256}69b3e16b51198f165244a7565b60a8e509369e52beff2976bc3ca920cfab61cb'
Notice: /Stage[main]/Beta::Autoupdater/Systemd::Timer::Job[wmf-beta-update-all]/Systemd::Timer[wmf-beta-update-all]/Systemd::Service[wmf-beta-update-all]/Systemd::Unit[wmf-beta-update-all.timer]/File[/lib/systemd/system/wmf-beta-update-all.timer]/ensure: defined content as '{sha256}a0d6584b2c042ef00a07f469305e970a2c2d0ffbbceb5f3817e41fa7f56e5b42'
Info: /Stage[main]/Beta::Autoupdater/Systemd::Timer::Job[wmf-beta-update-all]/Systemd::Timer[wmf-beta-update-all]/Systemd::Service[wmf-beta-update-all]/Systemd::Unit[wmf-beta-update-all.timer]/File[/lib/systemd/system/wmf-beta-update-all.timer]: Scheduling refresh of Exec[systemd daemon-reload for wmf-beta-update-all.timer (wmf-beta-update-all.timer)]
Notice: /Stage[main]/Beta::Autoupdater/Systemd::Timer::Job[wmf-beta-update-all]/Systemd::Timer[wmf-beta-update-all]/Systemd::Service[wmf-beta-update-all]/Systemd::Unit[wmf-beta-update-all.timer]/Exec[systemd daemon-reload for wmf-beta-update-all.timer (wmf-beta-update-all.timer)]: Triggered 'refresh' from 1 event
Notice: /Stage[main]/Beta::Autoupdater/Systemd::Timer::Job[wmf-beta-update-all]/Systemd::Timer[wmf-beta-update-all]/Systemd::Service[wmf-beta-update-all]/Service[wmf-beta-update-all.timer]/ensure: ensure changed 'stopped' to 'running'
Info: /Stage[main]/Beta::Autoupdater/Systemd::Timer::Job[wmf-beta-update-all]/Systemd::Timer[wmf-beta-update-all]/Systemd::Service[wmf-beta-update-all]/Service[wmf-beta-update-all.timer]: Unscheduling refresh on Service[wmf-beta-update-all.timer]
Notice: /Stage[main]/Rsyslog/Service[rsyslog]: Triggered 'refresh' from 1 event
Notice: Applied catalog in 21.34 seconds

bd808@deployment-deploy04:~$ systemctl status wmf-beta-update-all.timer wmf-beta-update-all.service --no-pager
● wmf-beta-update-all.timer - Periodic execution of wmf-beta-update-all.service
     Loaded: loaded (/lib/systemd/system/wmf-beta-update-all.timer; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2026-04-23 23:58:09 UTC; 8min ago
    Trigger: n/a
   Triggers: ● wmf-beta-update-all.service

● wmf-beta-update-all.service - Update MediaWiki code, config, and databases
     Loaded: loaded (/lib/systemd/system/wmf-beta-update-all.service; static)
     Active: activating (start) since Fri 2026-04-24 00:00:03 UTC; 6min ago
TriggeredBy: ● wmf-beta-update-all.timer
       Docs: https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
   Main PID: 3075821 (bash)
      Tasks: 13 (limit: 9523)
     Memory: 337.8M
     CGroup: /system.slice/wmf-beta-update-all.service
             ├─3075821 bash /usr/local/bin/wmf-beta-update-all
             ├─3075838 bash /usr/local/bin/wmf-beta-update-all
             ├─3075842 /usr/bin/perl /usr/bin/ts [%H:%M:%S]
             ├─3075843 tee -a /srv/beta-update-logs/202604240000.log
             ├─3102421 python3 /usr/local/bin/wmf-beta-update-databases.py
             ├─3103667 /bin/sh -c echo 'incubatorwiki'
            /usr/local/b…
             ├─3103668 /bin/bash /usr/local/bin/mwscript update.php --wiki=incu…
             ├─3103669 /bin/sh -c echo 'jawiki'
            /usr/local/bin/mwsc…
             ├─3103670 /bin/bash /usr/local/bin/mwscript update.php --wiki=jawi…
             ├─3103677 sudo -u www-data php /srv/mediawiki-staging/multiversion…
             ├─3103678 sudo -u www-data php /srv/mediawiki-staging/multiversion…
             ├─3103679 php /srv/mediawiki-staging/multiversion/MWScript.php upd…
             └─3103680 php /srv/mediawiki-staging/multiversion/MWScript.php upd…

Restricted Application added a subscriber: Dragoniez. · View Herald TranscriptApr 24 2026, 12:06 AM

The new /usr/local/bin/wmf-beta-update-all script is currently being run by a systemd timer every 10 minutes on deployment-deploy04. There is currently no alerting for this scheduled job. My plan is to wire up Prometheus Alertmanager alerting based on notes in T315695: Add basic MediaWiki/web site up alerting to the Beta Cluster.

Until we have alerting:

systemctl status wmf-beta-update-all.timer wmf-beta-update-all.service --no-pager
sudo journalctl -u wmf-beta-update-all.service
per-run logs named by run time (YYYYMMDDHHmm.log) in /srv/beta-update-logs/

The https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/ and https://integration.wikimedia.org/ci/view/Beta/job/beta-update-databases-eqiad/ jobs are disabled so that two systems aren't fighting over things.

The new timer can be disabled by changing the profile::beta::autoupdater::run_updater hiera setting at https://horizon.wikimedia.org/project/instances/60461772-4536-4055-ab8e-9180c719cf25/ (deployment-deploy04's direct puppet configuration).

Change #1277589 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] jjb: disable beta updaters

https://gerrit.wikimedia.org/r/1277589

Change #1277589 merged by jenkins-bot:

[integration/config@master] jjb: disable beta updaters

https://gerrit.wikimedia.org/r/1277589

Mentioned in SAL (#wikimedia-releng) [2026-04-27T21:57:16Z] <bd808> Add web security group to deployment-deploy04 (T256168)

Mentioned in SAL (#wikimedia-releng) [2026-04-27T22:11:40Z] <bd808> Beta Cluster MediaWiki update logs now available via https://beta-update.wmcloud.org/ (T256168)

A_smart_kitten subscribed.Apr 27 2026, 10:18 PM

Mentioned in SAL (#wikimedia-releng) [2026-04-28T00:03:48Z] <bd808> Increase parallelism for wmf-beta-update-databases.py (T256168)

The run logs are now exposed at https://beta-update.wmcloud.org/. There is also now a timer to prune logs after find -mtime 3.

Change #1281777 had a related patch set uploaded (by Reedy; author: Reedy):

[integration/config@master] jjb: beta.yml: Note that jobs are disabled

https://gerrit.wikimedia.org/r/1281777

Change #1281777 merged by jenkins-bot:

[integration/config@master] jjb: beta.yml: Note that jobs are disabled

https://gerrit.wikimedia.org/r/1281777

@bd808 the Jenkins jobs are disabled, the systemd timer run and we have logs exposed :party: thank you!

Can we claim victory on this one did you have following steps in mind? The ones I think of are removing the agent in Jenkins and deleting the jobs (I can take care of that).

In T256168#11959969, @hashar wrote:

Can we claim victory on this one did you have following steps in mind? The ones I think of are removing the agent in Jenkins and deleting the jobs (I can take care of that).

On my side we are still missing any notification if the sync jobs break. I found a good blog post on using OnFailure stanzas with systemd units over the weekend that actually seems like a promising direction. I would like email and irc yelling so we don't miss things getting messed up.

In T256168#11960221, @bd808 wrote:

On my side we are still missing any notification if the sync jobs break. I found a good blog post on using OnFailure stanzas with systemd units over the weekend that actually seems like a promising direction. I would like email and irc yelling so we don't miss things getting messed up.

You don't need anything on the systemd unit side for that, as the node-exporter provided node_systemd_unit_state metric can be used already.

hashar mentioned this in T421114: Rebuild all Jenkins agents VM to Bookworm to support Java 21.May 28 2026, 6:33 AM

Mentioned in SAL (#wikimedia-releng) [2026-05-28T07:05:30Z] <hashar> integration: delete deployment-deploy04 agent from Jenkins controller. Batch job has been migrated to a systemd driven script # T256168

Change #1294819 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] jjb: rename beta-irc macro to notify-releng-irc

https://gerrit.wikimedia.org/r/1294819

Change #1294820 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] jjb: remove Beta cluster jobs

https://gerrit.wikimedia.org/r/1294820

In T256168#11960221, @bd808 wrote:

In T256168#11959969, @hashar wrote:

Can we claim victory on this one did you have following steps in mind? The ones I think of are removing the agent in Jenkins and deleting the jobs (I can take care of that).

On my side we are still missing any notification if the sync jobs break. I found a good blog post on using OnFailure stanzas with systemd units over the weekend that actually seems like a promising direction. I would like email and irc yelling so we don't miss things getting messed up.

In production, the train-presync.service uses /usr/local/bin/systemd-timer-mail-wrapper for this purpose. I wonder if it works in beta.

/usr/local/bin/systemd-timer-mail-wrapper --subject train-presync --mail-to releng@lists.wikimedia.org /usr/bin/scap stage-train --yes auto

In T256168#11985947, @dancy wrote:

In production, the train-presync.service uses /usr/local/bin/systemd-timer-mail-wrapper for this purpose. I wonder if it works in beta.

The newly updated unit in Beta now has

ExecStart=/usr/local/bin/systemd-timer-mail-wrapper --subject wmf-beta-update-all --mail-to releng@lists.wikimedia.org --mail-from noreply@deployment-deploy04.deployment-prep.eqiad1.wikimedia.cloud --only-on-error /usr/local/bin/wmf-beta-update-all

I tested email sending separately, and it seemed to work if the sender address was a valid FQDN.

Added the sender to the accepted senders for the releng mailing list just now, initially getting caught up in filters. Confirms email is working.

Change #1276813 merged by RLazarus:

[operations/puppet@production] beta: Add a wmf-beta-update-all timer and script

https://gerrit.wikimedia.org/r/1276813

Change #1305770 had a related patch set uploaded (by Ahmon Dancy; author: Ahmon Dancy):

[operations/puppet@production] beta::autoupdater: Add tail-beta-update-logs utility

https://gerrit.wikimedia.org/r/1305770

Updated:

@bd808 Seems like this ticket may be ready to close?

In T256168#12120028, @dancy wrote:

@bd808 Seems like this ticket may be ready to close?

Agreed. I was going to tweak the Apache config for the log viewing vhost before chasing down a root to merge things, but I apparently got distracted and wandered away. Thanks for all the help folks.

Change #1305770 merged by RLazarus: