Rough outline:

• Turn up deployment-mwlog03 with Trixie (T417002) and a 20 GiB Cinder volume for /srv
• Adjust udp2log_aggregator to send to deployment-mwlog03 [0].
• Adjust profile::rsyslog::udp_tee::destinations to forward all logs to udp2log instances on deployment-mwlog0{2,3}
• Change betacluster_udplog_host (Puppet: scap) and udp2log in wmf-config/LabsServices.php.
• Manual logrotate isn't needed, because daily rotation has probably already run.
• Sync /srv/mw-log/archive between hosts
• Decommission deployment-mwlog02

[0] Root partition is already at >= 90%. Interestingly, there's already a 15 GiB Cinder disk attached to this instance - however, it is not mounted, and neither is it in /etc/fstab.
[1] We have deployment servers *without* Kubernetes support, so this variable is only used in mediawiki::syslog.

southparkfan opened https://gitlab.wikimedia.org/cloudvps-repos/deployment-prep/tofu-provisioning/-/merge_requests/34

mwlog: add deployment-mwlog03

southparkfan merged https://gitlab.wikimedia.org/cloudvps-repos/deployment-prep/tofu-provisioning/-/merge_requests/34

mwlog: add deployment-mwlog03

I couldn't finish the Tofu run, because we have too many server groups: │ Error: Error creating openstack_compute_servergroup_v2 deployment-mwlog: Expected HTTP response code [200] when accessing [POST https://openstack.eqiad1.wikimediacloud.org:28774/v2.1/os-server-groups], but got 403 instead: {"forbidden": {"code": 403, "message": "Quota exceeded, too many server groups."}}. Opened T437117 for a quota increase.

southparkfan opened https://gitlab.wikimedia.org/cloudvps-repos/deployment-prep/tofu-provisioning/-/merge_requests/35

mwlog: add Cinder volume, add Puppet role

southparkfan merged https://gitlab.wikimedia.org/cloudvps-repos/deployment-prep/tofu-provisioning/-/merge_requests/35

mwlog: add Cinder volume, add Puppet role

Mentioned in SAL (#wikimedia-cloud) [2026-09-09T22:32:22Z] <Southparkfan> set role::logging::mediawiki::udp2log::monitor: false to unbreak T436469

Mentioned in SAL (#wikimedia-cloud) [2026-09-09T22:36:17Z] <Southparkfan> point profile::rsyslog::udp_tee::destinations to both deployment-mwlog02 and deployment-mwlog03 8421/udp - T436469

deployment-mwlog03 has been set up with a Cinder volume.

Due to a lack of NRPE on deployment-mwlog03, I had a hard time running Puppet without errors:

Error: Could not set 'file' on ensure: No such file or directory - A directory component in /usr/lib/nagios/plugins/check_udp2log_log_age20260909-20728-n42z8n.lock does not exist or is a dangling symbolic link (file: /srv/puppet_code/environments/production/modules/udp2log/manifests/monitoring.pp, line: 7)

Fixed by setting role::logging::mediawiki::udp2log::monitor: false in the project hiera. Afterwards, I have added profile::rsyslog::udp_tee::destinations to mirror the messages to the udp2log instances on both mwlog hosts. This ought to be moved to deployment-mwlog Prefix Puppet.

Change #1338305 had a related patch set uploaded (by Southparkfan; author: Southparkfan):

[operations/puppet@production] profile::mediawiki::scap_client: allow configuring betacluster_udp2log_host

https://gerrit.wikimedia.org/r/1338305

Change #1338306 had a related patch set uploaded (by Southparkfan; author: Southparkfan):

[operations/mediawiki-config@master] LabsServices: adjust udp2log to use service RR

https://gerrit.wikimedia.org/r/1338306

Change #1338306 merged by jenkins-bot:

[operations/mediawiki-config@master] LabsServices: adjust udp2log to use service RR

https://gerrit.wikimedia.org/r/1338306

Mentioned in SAL (#wikimedia-cloud) [2026-09-10T21:08:15Z] <Southparkfan> masked and stopped stray udp2log.service on deployment-mwlog03, claimed 8420/udp which should have been assigned to udp_tee - T436469

Mentioned in SAL (#wikimedia-cloud) [2026-09-10T22:01:49Z] <Southparkfan> decommission deployment-mwlog02 - T436469