Replace deployment-deploy04 with a Bookworm instance with Java 21
Closed, DeclinedPublicFeature
Actions

Assigned To

Authored By

	hashar
	Mar 25 2026, 3:17 PM

Project Tags

Referenced Files

None

Subscribers

Description

As part of upgrading the Jenkins controller from Java 17 to Java 21 (T418109) we need all agents to have Java 21 available. Instances based on Debian Bullseye lack Java 21 and we thus need to rebuild them to Debian Bookworm (T421114).

The deployment-deploy04.deployment-prep.eqiad1.wikimedia.cloud is based on Bullseye and attached to Jenkins (deployment-deploy04). The instance needs to be rebuild to Debian Bookworm.

Why not Trixie?

Bookworm has both Java 17 and Java 21. This let us rebuild the instance and attach it to the current Jenkins running Java 17. When we switch over to a new Jenkins that runs Java 21, we will be able to change the Java version on the agent from 17 to 21 and have them connected to the new Jenkins.

Trixie only has Java 21, which would not let us attach an instance based on it to the current Jenkins (which lacks Java 21).

Jenkins

Once the instance is build, it will be added to Jenkins with the label BetaClusterBastion. Maybe the existing one can be renamed ( https://integration.wikimedia.org/ci/computer/deployment%2Ddeploy04/ ), disconnected and reconnected to the new host.

The Jenkins jobs are tied to the label rather than an agent named and would thus start running on the new agent.

Configuration

deployment-depoy04 in Horizon

Per instance Puppet config:

Puppet Classes

role::beta::deploymentserver
role::deployment_server
profile::ci::slave::labs::common

Hiera Config

profile::base::labs::cleanup_puppet_client_bucket: true
profile::ci::slave::labs::common::manage_srv: false
profile::kubernetes::client::version: '1.23'
profile::mediawiki::scap_client::is_master: true

deployment-deploy Puppet prefix config at https://horizon.wikimedia.org/project/prefixpuppet/

Puppet Classes

profile::rsyslog::kafka_shipper

Hiera Config

aptly::manage_nginx: false
profile::base::labs::cleanup_puppet_client_bucket: true
profile::ci::slave::labs::common::manage_srv: false
profile::java::java_packages:
- variant: jre-headless
  version: '17'
profile::mediawiki::php::absented_php_versions:
- '8.1'
profile::mediawiki::php::php_versions:
- '8.3'
profile::mediawiki::scap_client::is_master: true
profile::mediawiki::webserver::default_php_version: '8.3'

Details

Related Changes in Gerrit:

	Subject	Author	Repo	Branch	Lines +/-
	deployment-prep: Remove hiera for deployment-deploy04	BryanDavis	operations/puppet	production	+0 -1

Customize query in gerrit

Related Changes in GitLab:

	Title	Reference	Author	Source Branch	Dest Branch
	deployment-deploy05: decomm host	cloudvps-repos/deployment-prep/tofu-provisioning!9	bd808	work/bd808/deployment-deploy05	main
	Add tofu config to provision deployment-deploy05	cloudvps-repos/deployment-prep/tofu-provisioning!8	bd808	work/bd808/service	main

Customize query in GitLab

Related Objects
Search...

Status	Subtype	Assigned	Task
Resolved		hashar	T418109 Update Jenkins hosts from Java 17 to Java 21
Resolved		hashar	T421114 Rebuild all Jenkins agents VM to Bookworm to support Java 21
			Restricted Task
Resolved		Dzahn	T418521 setup 2 contint machines for jenkins
Open		None	T401839 Migrate deployment-prep away from Debian Bullseye to Bookworm/Trixie
Declined	Feature	• bd808	T421244 Replace deployment-deploy04 with a Bookworm instance with Java 21

Event Timeline

hashar created this task.Mar 25 2026, 3:17 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 25 2026, 3:17 PM

hashar added a parent task: T421114: Rebuild all Jenkins agents VM to Bookworm to support Java 21.Mar 25 2026, 3:18 PM

hashar mentioned this in T421114: Rebuild all Jenkins agents VM to Bookworm to support Java 21.Mar 25 2026, 3:20 PM

• bd808 added a parent task: T401839: Migrate deployment-prep away from Debian Bullseye to Bookworm/Trixie.Mar 25 2026, 3:26 PM

From a discussion I had with bd808 and others, we might instead stop using to update the Beta-Cluster-Infrastructure in favor of a simpler system (sytemd timer?) in which case rebuilding the instance would no more be a blocker to the Jenkins Java 21 upgrade (T421114).

We need to build a new instance for T401839: Migrate deployment-prep away from Debian Bullseye to Bookworm/Trixie too, so the quicker thing is probably to do that now and also unblock the Jenkins stuff. We can revisit T256168: Move beta cluster automatic deployment to a dedicated infrastructure at another time. I do think this is a good opportunity to start on T394316: Use infrastructure as code techniques to rebuild the Beta Cluster.

• bd808 claimed this task.Mar 25 2026, 4:01 PM

• bd808 triaged this task as High priority.

• bd808 moved this task from To Triage to Future on the Beta-Cluster-Infrastructure board.

Restricted Application added a project: User-bd808. · View Herald TranscriptMar 25 2026, 4:01 PM

• bd808 renamed this task from Rebuild deployment-deploy04 with Bookworm to get Java 21 to Replace deployment-deploy04 with a Bookworm instance with Java 21.Mar 25 2026, 4:02 PM

• bd808 changed the subtype of this task from "Task" to "Feature Request".

In a quick chat with @hashar I asked about the cut-over from the Java 17 compatible Jenkins to Java 21 compatible Jenkins. On a Bookworm host we have the ability to choose between Java 17 and Java 21 via hiera settings. So we can build a Bookworm + Java 17 instance, connect it to the Java 17 Jenkins, and then when the Java 21 Jenkins is ready we can change the hiera to get Java 21 and connect with the Java 21 Jenkins.

bd808 opened https://gitlab.wikimedia.org/cloudvps-repos/deployment-prep/tofu-provisioning/-/merge_requests/8

Draft: Add tofu config to provision deployment-deploy05

bd808 merged https://gitlab.wikimedia.org/cloudvps-repos/deployment-prep/tofu-provisioning/-/merge_requests/8

Add tofu config to provision deployment-deploy05

Mentioned in SAL (#wikimedia-releng) [2026-04-21T20:17:41Z] <bd808> Running tofu for T421244

Apply complete! Resources: 6 added, 0 changed, 0 destroyed. https://gitlab.wikimedia.org/cloudvps-repos/deployment-prep/tofu-provisioning/-/jobs/804773

Maintenance_bot removed a project: Patch-For-Review.Apr 21 2026, 8:30 PM

wait for first puppet run so that ssh works as a project member
ssh deployment-deploy05.deployment-prep.eqiad1.wikimedia.cloud

bd808@deployment-deploy05:~$ sudo -i puppet agent -tv
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Function Call, Class[Profile::Base::Labs]: parameter 'cleanup_puppet_client_bucket' expects a Boolean value, got String (file: /srv/puppet_code/environments/production/modules/role/manifests/wmcs/instance.pp, line: 3, column: 5) on node deployment-deploy05.deployment-prep.eqiad1.wikimedia.cloud
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

Booleans were coerced to strings somewhere???

yamlencode({"a": true, "c": false}) works as expected in tofu console. Did I do something weird in the provider when I worked on clean round tripping?

HACK: manually corrected via horizon, but needs to get figured out in tofu so it's not a wheel war.
https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/5f0b112740454fc34da8649e8767ea6dae0a5902%5E%21/#F0

diff --git a/deployment-prep/deployment-deploy05.deployment-prep.eqiad1.wikimedia.cloud.yaml b/deployment-prep/deployment-deploy05.deployment-prep.eqiad1.wikimedia.cloud.yaml
index 33246ff..5c005f2 100644
--- a/deployment-prep/deployment-deploy05.deployment-prep.eqiad1.wikimedia.cloud.yaml
+++ b/deployment-prep/deployment-deploy05.deployment-prep.eqiad1.wikimedia.cloud.yaml

@@ -1,4 +1,4 @@
-profile::base::labs::cleanup_puppet_client_bucket: 'true'
-profile::ci::slave::labs::common::manage_srv: 'false'
+profile::base::labs::cleanup_puppet_client_bucket: true
+profile::ci::slave::labs::common::manage_srv: false
 profile::kubernetes::client::version: '1.23'
-profile::mediawiki::scap_client::is_master: 'false'
+profile::mediawiki::scap_client::is_master: false

Tofu really wants them to be string coerced, so either my inputs are wrong or the provider is doing something unexpected:

OpenTofu will perform the following actions:

  # module.deploy.cloudvps_puppet_prefix.vm["05"] will be updated in-place
  ~ resource "cloudvps_puppet_prefix" "vm" {
      ~ hiera = <<-EOT
          - "profile::base::labs::cleanup_puppet_client_bucket": true
          - "profile::ci::slave::labs::common::manage_srv": false
          + "profile::base::labs::cleanup_puppet_client_bucket": "true"
          + "profile::ci::slave::labs::common::manage_srv": "false"
            "profile::kubernetes::client::version": "1.23"
          - "profile::mediawiki::scap_client::is_master": false
          + "profile::mediawiki::scap_client::is_master": "false"
        EOT
        id    = 22945
        name  = "deployment-deploy05.deployment-prep.eqiad1.wikimedia.cloud"
        # (1 unchanged attribute hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

bd808@deployment-deploy05:~$ sudo -i puppet agent -tv
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Resource Statement, Duplicate declaration: File[/usr/bin/scap] is already declared at (file: /srv/puppet_code/environments/production/modules/scap/manifests/init.pp, line: 43); cannot redeclare (file: /srv/puppet_code/environments/production/modules/scap/manifests/master.pp, line: 94) (file: /srv/puppet_code/environments/production/modules/scap/manifests/master.pp, line: 94, column: 5) on node deployment-deploy05.deployment-prep.eqiad1.wikimedia.cloud
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

This is my mistake in the hiera settings apparently. profile::mediawiki::scap_client::is_master: true is expected by ::scap as a guard condition for initialization. $is_master and $::fqdn != $deployment_server is used to check if this deployment server is the active or not.

Because of the boolean problem in T421244#11847209 I will change this via Horizon for now.
https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/4ac90d7349bf16f7abfb7a00579941f01c703168%5E%21/#F0

diff --git a/deployment-prep/deployment-deploy05.deployment-prep.eqiad1.wikimedia.cloud.yaml b/deployment-prep/deployment-deploy05.deployment-prep.eqiad1.wikimedia.cloud.yaml
index 5c005f2..8c8d19a 100644
--- a/deployment-prep/deployment-deploy05.deployment-prep.eqiad1.wikimedia.cloud.yaml
+++ b/deployment-prep/deployment-deploy05.deployment-prep.eqiad1.wikimedia.cloud.yaml

@@ -1,4 +1,4 @@
 profile::base::labs::cleanup_puppet_client_bucket: true
 profile::ci::slave::labs::common::manage_srv: false
 profile::kubernetes::client::version: '1.23'
-profile::mediawiki::scap_client::is_master: false
+profile::mediawiki::scap_client::is_master: true

bd808@deployment-deploy05:~$ sudo -i puppet agent -tv
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Function Call, If you want non-sni TLS to be supported, you need to define profile::tlsproxy::envoy::global_cert_name (file: /srv/puppet_code/environments/production/modules/profile/manifests/tlsproxy/envoy.pp, line: 188, column: 13) on node deployment-deploy05.deployment-prep.eqiad1.wikimedia.cloud
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

I'm not seeing yet how this is configured differently for the legacy deployment-deploy04 node vs this new deployment-deploy05 node. I wonder if something changes after we switch puppetservers?

sudo puppet lookup --node deployment-deploy04.deployment-prep.eqiad1.wikimedia.cloud --compile --explain profile::tlsproxy::envoy::global_cert_name returns Found key: "profile::tlsproxy::envoy::global_cert_name" value: "deployment-deploy04.deployment-prep.eqiad1.wikimedia.cloud"

Oh man... sneaky and wrong! In operations/puppet.git's hieradata/cloud/eqiad1/deployment-prep/hosts/deployment-deploy04.yaml there is a profile::tlsproxy::envoy::global_cert_name: "%{facts.fqdn}" setting. Much sadness at yet another config location introduced by rOPUPefe792d91662: spiderpig: Set global_cert_name on deployment-deploy04. Let's hoist this up to the deployment-deploy prefix puppet settings.

https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/724a121f25fc0c8d182fc5e135d86828f89958ea%5E%21/#F0

diff --git a/deployment-prep/deployment-deploy.yaml b/deployment-prep/deployment-deploy.yaml
index 7441c77..709f897 100644
--- a/deployment-prep/deployment-deploy.yaml
+++ b/deployment-prep/deployment-deploy.yaml

@@ -10,3 +10,4 @@
 - '8.3'
 profile::mediawiki::scap_client::is_master: true
 profile::mediawiki::webserver::default_php_version: '8.3'
+profile::tlsproxy::envoy::global_cert_name: '%{facts.fqdn}'

Change #1276068 had a related patch set uploaded (by BryanDavis; author: Bryan Davis):

[operations/puppet@production] deployment-prep: Remove hiera for deployment-deploy04

https://gerrit.wikimedia.org/r/1276068

gerritbot added a project: Patch-For-Review.Apr 21 2026, 10:39 PM

Now that I have beaten the initial puppet run into submission, it is time to do the puppetserver switch dance from https://wikitech.wikimedia.org/wiki/Help:Project_puppetserver#Step_2:_Setup_a_puppet_client. In a more ideal world we would have been at this stage basically as soon as the ssh connection worked.
Future @bd808, start here in figuring out how to make OpenTofu drive this switch automagically ->

bd808@deployment-deploy05:~$ sudo -i puppet agent -tv
Info: Using environment 'production'
Error: certificate verify failed [self-signed certificate in certificate chain for CN=Puppet CA: deployment-puppetmaster03.deployment-prep.eqiad.wmflabs]
Info: Loading facts
Error: Could not retrieve catalog from remote server: certificate verify failed [self-signed certificate in certificate chain for CN=Puppet CA: deployment-puppetmaster03.deployment-prep.eqiad.wmflabs]
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
Error: Could not send report: certificate verify failed [self-signed certificate in certificate chain for CN=Puppet CA: deployment-puppetmaster03.deployment-prep.eqiad.wmflabs]
bd808@deployment-deploy05:~$ sudo rm -rf /var/lib/puppet/ssl
bd808@deployment-deploy05:~$ sudo -i run-puppet-agent
Info: Creating a new RSA SSL key for deployment-deploy05.deployment-prep.eqiad1.wikimedia.cloud
Info: csr_attributes file loading from /etc/puppet/csr_attributes.yaml
Info: Creating a new SSL certificate request for deployment-deploy05.deployment-prep.eqiad1.wikimedia.cloud
Info: Certificate Request fingerprint (SHA256): 52:09:80:52:B1:0C:2C:8D:80:63:87:AF:1E:A5:7B:99:3F:14:49:9E:94:DE:62:D1:0A:51:D0:14:52:77:0A:32
Info: Certificate for deployment-deploy05.deployment-prep.eqiad1.wikimedia.cloud has not been signed yet
Couldn't fetch certificate from CA server; you might still need to sign this agent's certificate (deployment-deploy05.deployment-prep.eqiad1.wikimedia.cloud).
Exiting now because the waitforcert setting is set to 0.

We need to manually sign the new cert because we don't set profile::puppetserver::autosign in hiera. It would be nice to add something like the /usr/local/sbin/validatecloudvpsfqdn.py script used by ::profile::puppetserver::wmcs to make this step go away.

bd808@deployment-puppetserver-1.deployment-prep.eqiad1:~$ sudo -i puppetserver ca list | grep deployment-deploy05
    deployment-deploy05.deployment-prep.eqiad1.wikimedia.cloud       (SHA256)  52:09:80:52:B1:0C:2C:8D:80:63:87:AF:1E:A5:7B:99:3F:14:49:9E:94:DE:62:D1:0A:51:D0:14:52:77:0A:32
bd808@deployment-puppetserver-1.deployment-prep.eqiad1:~$ sudo puppetserver ca sign --certname deployment-deploy05.deployment-prep.eqiad1.wikimedia.cloud
Successfully signed certificate request for deployment-deploy05.deployment-prep.eqiad1.wikimedia.cloud

Back to deployment-deploy05 now:

bd808@deployment-deploy05:~$ sudo -i run-puppet-agent
Info: csr_attributes file loading from /etc/puppet/csr_attributes.yaml
Info: Creating a new SSL certificate request for deployment-deploy05.deployment-prep.eqiad1.wikimedia.cloud
Info: Certificate Request fingerprint (SHA256): 52:09:80:52:B1:0C:2C:8D:80:63:87:AF:1E:A5:7B:99:3F:14:49:9E:94:DE:62:D1:0A:51:D0:14:52:77:0A:32
Info: Downloaded certificate for deployment-deploy05.deployment-prep.eqiad1.wikimedia.cloud from https://deployment-puppetserver-1.deployment-prep.eqiad1.wikimedia.cloud:8140/puppet-ca/v1
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin

... lots of things changing because of secrets and patches on the project puppetserver ...

Info: Stage[main]: Unscheduling all events on Stage[main]
Notice: Applied catalog in 94.78 seconds

There were some failures that streamed by, let's see if a second puppet run fixes things (missing graph dependencies) or if there is a real problem in there somewhere.

bd808@deployment-deploy05:~$ sudo -i run-puppet-agent
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for deployment-deploy05.deployment-prep.eqiad1.wikimedia.cloud
Info: Applying configuration version '(c7474b3aa7) gitpuppet - MediaWiki: Only proxy existing .php files, otherwise return nice 404'
Notice: /Stage[main]/Ssh::Client/File[/etc/ssh/ssh_known_hosts]/content:
--- /etc/ssh/ssh_known_hosts    2026-04-21 23:24:41.300654402 +0000
+++ /tmp/puppet-file20260421-41712-b7upp8       2026-04-21 23:25:56.632679041 +0000
@@ -1,6 +1,3 @@
-# HEADER: This file was autogenerated at 2026-04-21 23:24:41 +0000
-# HEADER: by puppet.  While it can still be managed manually, it
-# HEADER: is definitely not recommended.
 # This file is managed by puppet
 deployment-acme-chief05.deployment-prep.eqiad1.wikimedia.cloud,deployment-acme-chief05,172.16.4.3,fe80::f816:3eff:fe8e:e4dc ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNKxFW8xm8BypAq0/myF9ofyBgQ27G0WpM5Kc2bTm/nsiDgyJKc+QeN39JCMUCpiw6ggvV6EedLNLi6SjLxSCdE=
 deployment-acme-chief06.deployment-prep.eqiad1.wikimedia.cloud,deployment-acme-chief06,172.16.2.219,fe80::f816:3eff:fef4:9016 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLXu9iyX0uKDeAgBHri6gI+sn4UZMmQItQ4dPCLRNqtuNJSLPbT1urmCMfCLY15mDlxyoet3H+FBKKj4uW3BfHY=
@@ -14,6 +11,7 @@
 deployment-db11.deployment-prep.eqiad1.wikimedia.cloud,deployment-db11,172.16.5.150,fe80::f816:3eff:fec9:a169 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAR/qU6xjNMkhXNmsOcYc4sJVPnvCj2b5UrcHiKUxzRpY37Y/NDprrPODNFoO4x203u+qDRgLqs4fMte4dluL04=
 deployment-db14.deployment-prep.eqiad1.wikimedia.cloud,deployment-db14,172.16.5.170,fe80::f816:3eff:fe93:7c3c ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBESbM/xMyACA2BYwTLh+cc8WS2jp0ERFOR1AleCWwdkGO+EMRKV6MhVuEhB/hYwpX8l9pwC8hZ8Ce/XOFm/72K8=
 deployment-deploy04.deployment-prep.eqiad1.wikimedia.cloud,deployment-deploy04,172.16.1.63,fe80::f816:3eff:fe34:9d1 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHD98gLvb4/dh7Vf2QQL8rdvqYdOiMm/oIhqacNqS1qi4phxlBuoOUvvfD4Ss+epBeUvPrDWJAkDqvGAoZDgodY=
+deployment-deploy05.deployment-prep.eqiad1.wikimedia.cloud,deployment-deploy05,172.16.20.131,2a02:ec80:a000:1::2ce ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO/X8xrYKYJXGaXJHKMfOVU2uoG6SyB9KzPfIIjsv1qZUPyvQ09xIaxv9MTR7XQARQ8h6Lobm/FTLIjg44lZGTU=
 deployment-docker-charts01.deployment-prep.eqiad1.wikimedia.cloud,deployment-docker-charts01,172.16.4.223,fe80::f816:3eff:fe5d:b155 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDbCWNsNOV1rGnSEpdaxWE9GoQvmry9RMxWNJr9aopOB6zAb1bjwzzIeQNOm7tOltu4XTieOnLAjmuLg83YeqCc=
 deployment-docker-citoid02.deployment-prep.eqiad1.wikimedia.cloud,deployment-docker-citoid02,172.16.3.154,fe80::f816:3eff:fe39:eb44 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMXwspAqhtEFLbk8Bs/CgXvwqtdJTMsZ2hT63xK5YgTGGLybDYe1S/ynrsLHsTkcZb+UdZGLmpDrpzjv+z9JLkk=
 deployment-docker-cxserver02.deployment-prep.eqiad1.wikimedia.cloud,deployment-docker-cxserver02,172.16.1.68,fe80::f816:3eff:fe99:d9ed ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGDwOYa1IhiHkAvId+toECn5vZPdjMeqg9A0ulqcvrW31aeajdE4PBGj/AtfNS94cIN76H2H8z/cJ0oFeCeOgIo=
@@ -61,4 +59,3 @@
 deployment-webperf22.deployment-prep.eqiad1.wikimedia.cloud,deployment-webperf22,172.16.5.13,fe80::f816:3eff:fece:d2f6 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNOz0NEzFJSCeQDayftQjduYJW0fxRCyMtVas4zr5fQDVtdlidlWhqIqUZZeZrb2dcK8A1W5MKYlvmGTDSDZpxo=
 deployment-wikifeeds02.deployment-prep.eqiad1.wikimedia.cloud,deployment-wikifeeds02,172.16.4.35,fe80::f816:3eff:fe6a:3c52 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJRov6bw0g/1fTFuFysHF1jEshVWHHbAU6ZJAPKRxE+sKuIBQn3UqGzUgay1DYed+TPErViVwuDasJwv5/S2OJE=
 deployment-zookeeper-3.deployment-prep.eqiad1.wikimedia.cloud,deployment-zookeeper-3,172.16.6.11,fe80::f816:3eff:feed:a89d ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAn5JXibRrHVMLH63TyxRHP9ZuJ8cRoTRoCoOfdUZ/zwDMirB1JZBNA5b04FpCdbs6y5X/QDTMS3kOEaBSsAYVo=
-gerrit.wikimedia.org,gerrit.discovery.wmnet,208.80.153.116,2620:0:860:4:208:80:153:116,208.80.154.225,2620:0:861:ed1a::2,208.80.153.225,2620:0:860:ed1a::2,185.15.59.225,2a02:ec80:300:ed1a::2,198.35.26.225,2620:0:863:ed1a::2,103.102.166.225,2001:df2:e500:ed1a::2,185.15.58.225,2a02:ec80:600:ed1a::2,195.200.68.225,2a02:ec80:700:ed1a::2 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQCF8pwFLehzCXhbF1jfHWtd9d1LFq2NirplEBQYs7AOrGwQ/6ZZI0gvZFYiEiaw1o+F1CMfoHdny1VfWOJF3mJ1y9QMKAacc8/Z3tG39jBKRQCuxmYLO1SWymv7/Uvx9WQlkNRoTdTTa9OJFy6UqvLQEXKYaokfMIUHZ+oVFf1CgQ==

Notice: /Stage[main]/Ssh::Client/File[/etc/ssh/ssh_known_hosts]/content:

Notice: /Stage[main]/Ssh::Client/File[/etc/ssh/ssh_known_hosts]/content: content changed '{sha256}aa448e503c28f4db0a3b8b26fcf186ca09775dc7e36e3e4b80aada39b9a1ac24' to '{sha256}e63cf8c3b094438409be5f595d6be261b99d11faf1241494d121ee5d2d3f0d0b' (corrective)
Notice: /Stage[main]/Profile::Ci::Slave::Labs::Common/Profile::Gerrit::Sshkey[gerrit]/Sshkey[gerrit.wikimedia.org@ssh-rsa]/ensure: created (corrective)
Info: Computing checksum on file /etc/ssh/ssh_known_hosts
Error: Systemd start for mw-cgroup failed!
journalctl log for mw-cgroup:
Apr 21 23:24:42 deployment-deploy05 systemd[1]: Starting mw-cgroup.service - "Mw-cgroup"...
Apr 21 23:24:42 deployment-deploy05 mw-cgroup[40850]: /bin/bash: line 1: /sys/fs/cgroup/memory/release_agent: Permission denied
Apr 21 23:24:42 deployment-deploy05 systemd[1]: mw-cgroup.service: Main process exited, code=exited, status=1/FAILURE
Apr 21 23:24:42 deployment-deploy05 systemd[1]: mw-cgroup.service: Failed with result 'exit-code'.
Apr 21 23:24:42 deployment-deploy05 systemd[1]: Failed to start mw-cgroup.service - "Mw-cgroup".
Apr 21 23:26:04 deployment-deploy05 systemd[1]: Starting mw-cgroup.service - "Mw-cgroup"...
Apr 21 23:26:04 deployment-deploy05 mw-cgroup[42091]: /bin/bash: line 1: /sys/fs/cgroup/memory/release_agent: Permission denied
Apr 21 23:26:04 deployment-deploy05 systemd[1]: mw-cgroup.service: Main process exited, code=exited, status=1/FAILURE
Apr 21 23:26:04 deployment-deploy05 systemd[1]: mw-cgroup.service: Failed with result 'exit-code'.
Apr 21 23:26:04 deployment-deploy05 systemd[1]: Failed to start mw-cgroup.service - "Mw-cgroup".

Error: /Stage[main]/Mediawiki::Cgroup/Base::Service_unit[mw-cgroup]/Service[mw-cgroup]/ensure: change from 'stopped' to 'running' failed: Systemd start for mw-cgroup failed!
journalctl log for mw-cgroup:
Apr 21 23:24:42 deployment-deploy05 systemd[1]: Starting mw-cgroup.service - "Mw-cgroup"...
Apr 21 23:24:42 deployment-deploy05 mw-cgroup[40850]: /bin/bash: line 1: /sys/fs/cgroup/memory/release_agent: Permission denied
Apr 21 23:24:42 deployment-deploy05 systemd[1]: mw-cgroup.service: Main process exited, code=exited, status=1/FAILURE
Apr 21 23:24:42 deployment-deploy05 systemd[1]: mw-cgroup.service: Failed with result 'exit-code'.
Apr 21 23:24:42 deployment-deploy05 systemd[1]: Failed to start mw-cgroup.service - "Mw-cgroup".
Apr 21 23:26:04 deployment-deploy05 systemd[1]: Starting mw-cgroup.service - "Mw-cgroup"...
Apr 21 23:26:04 deployment-deploy05 mw-cgroup[42091]: /bin/bash: line 1: /sys/fs/cgroup/memory/release_agent: Permission denied
Apr 21 23:26:04 deployment-deploy05 systemd[1]: mw-cgroup.service: Main process exited, code=exited, status=1/FAILURE
Apr 21 23:26:04 deployment-deploy05 systemd[1]: mw-cgroup.service: Failed with result 'exit-code'.
Apr 21 23:26:04 deployment-deploy05 systemd[1]: Failed to start mw-cgroup.service - "Mw-cgroup".

Notice: /Stage[main]/Scap::Dsh/Scap::Dsh::Group[scap_targets]/File[/etc/dsh/group/scap_targets]/content:
--- /etc/dsh/group/scap_targets 2026-04-21 23:24:43.544655152 +0000
+++ /tmp/puppet-file20260421-41712-z534l        2026-04-21 23:26:05.708681937 +0000
@@ -5,6 +5,7 @@
 # Either directly in the declaration of the resource
 # or via hiera scap::dsh::groups::scap_targets
 deployment-deploy04.deployment-prep.eqiad1.wikimedia.cloud
+deployment-deploy05.deployment-prep.eqiad1.wikimedia.cloud
 deployment-echostore02.deployment-prep.eqiad1.wikimedia.cloud
 deployment-jobrunner05.deployment-prep.eqiad1.wikimedia.cloud
 deployment-mediawiki13.deployment-prep.eqiad1.wikimedia.cloud

Notice: /Stage[main]/Scap::Dsh/Scap::Dsh::Group[scap_targets]/File[/etc/dsh/group/scap_targets]/content: content changed '{sha256}b5df5d7e171a54ea09fa2ab116af549ec3f33bff8c9f046d94aaad0cf20161f0' to '{sha256}84a0fe980608bb386c2d4befd018788f9ebaef02069037f2c10c38280b7394b3'
Error: Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold --force-yes install libpcre2-8-0=10.42-1~wmf11+1' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
Package libpcre2-8-0 is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

W: --force-yes is deprecated, use one of the options starting with --allow instead.
E: Version '10.42-1~wmf11+1' for 'libpcre2-8-0' was not found
Error: /Stage[main]/Profile::Mediawiki::Php/Package[libpcre2-8-0]/ensure: change from '10.42-1' to '10.42-1~wmf11+1' failed: Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold --force-yes install libpcre2-8-0=10.42-1~wmf11+1' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
Package libpcre2-8-0 is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

W: --force-yes is deprecated, use one of the options starting with --allow instead.
E: Version '10.42-1~wmf11+1' for 'libpcre2-8-0' was not found
Error: Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold --force-yes install php-common=2:94+wmf11u1' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
Package php-common is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

W: --force-yes is deprecated, use one of the options starting with --allow instead.
E: Version '2:94+wmf11u1' for 'php-common' was not found
Error: /Stage[main]/Profile::Mediawiki::Php/Package[php-common]/ensure: change from 'purged' to '2:94+wmf11u1' failed: Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold --force-yes install php-common=2:94+wmf11u1' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
Package php-common is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

W: --force-yes is deprecated, use one of the options starting with --allow instead.
E: Version '2:94+wmf11u1' for 'php-common' was not found
Notice: /Stage[main]/Php/Package[php8.3-common]: Dependency Package[libpcre2-8-0] has failures: true
Notice: /Stage[main]/Php/Package[php8.3-common]: Dependency Package[php-common] has failures: true
Warning: /Stage[main]/Php/Package[php8.3-common]: Skipping because of failed dependencies
Warning: /Stage[main]/Php/Package[php8.3-opcache]: Skipping because of failed dependencies
Error: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install php8.3-cli' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package php8.3-cli
E: Couldn't find any package by glob 'php8.3-cli'
E: Couldn't find any package by regex 'php8.3-cli'
Error: /Stage[main]/Php/Package[php8.3-cli]/ensure: change from 'purged' to 'present' failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install php8.3-cli' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package php8.3-cli
E: Couldn't find any package by glob 'php8.3-cli'
E: Couldn't find any package by regex 'php8.3-cli'
Error: Cannot create /etc/php/8.3/cli/conf.d; parent directory /etc/php/8.3/cli does not exist
Error: /Stage[main]/Php/File[/etc/php/8.3/cli/conf.d]/ensure: change from 'absent' to 'directory' failed: Cannot create /etc/php/8.3/cli/conf.d; parent directory /etc/php/8.3/cli does not exist
Error: Could not set 'present' on ensure: No such file or directory - A directory component in /etc/php/8.3/cli/php.ini20260421-41712-et88ed.lock does not exist or is a dangling symbolic link (file: /srv/puppet_code/environments/production/modules/php/manifests/init.pp, line: 69)
Error: Could not set 'present' on ensure: No such file or directory - A directory component in /etc/php/8.3/cli/php.ini20260421-41712-et88ed.lock does not exist or is a dangling symbolic link (file: /srv/puppet_code/environments/production/modules/php/manifests/init.pp, line: 69)
Wrapped exception:
No such file or directory - A directory component in /etc/php/8.3/cli/php.ini20260421-41712-et88ed.lock does not exist or is a dangling symbolic link
Error: /Stage[main]/Php/File[/etc/php/8.3/cli/php.ini]/ensure: change from 'absent' to 'present' failed: Could not set 'present' on ensure: No such file or directory - A directory component in /etc/php/8.3/cli/php.ini20260421-41712-et88ed.lock does not exist or is a dangling symbolic link (file: /srv/puppet_code/environments/production/modules/php/manifests/init.pp, line: 69)
Error: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install php8.3-fpm' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package php8.3-fpm
E: Couldn't find any package by glob 'php8.3-fpm'
E: Couldn't find any package by regex 'php8.3-fpm'
Error: /Stage[main]/Php/Package[php8.3-fpm]/ensure: change from 'purged' to 'present' failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install php8.3-fpm' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package php8.3-fpm
E: Couldn't find any package by glob 'php8.3-fpm'
E: Couldn't find any package by regex 'php8.3-fpm'
Error: Cannot create /etc/php/8.3/fpm/conf.d; parent directory /etc/php/8.3/fpm does not exist
Error: /Stage[main]/Php/File[/etc/php/8.3/fpm/conf.d]/ensure: change from 'absent' to 'directory' failed: Cannot create /etc/php/8.3/fpm/conf.d; parent directory /etc/php/8.3/fpm does not exist
Error: Could not set 'present' on ensure: No such file or directory - A directory component in /etc/php/8.3/fpm/php.ini20260421-41712-drs5la.lock does not exist or is a dangling symbolic link (file: /srv/puppet_code/environments/production/modules/php/manifests/init.pp, line: 69)
Error: Could not set 'present' on ensure: No such file or directory - A directory component in /etc/php/8.3/fpm/php.ini20260421-41712-drs5la.lock does not exist or is a dangling symbolic link (file: /srv/puppet_code/environments/production/modules/php/manifests/init.pp, line: 69)
Wrapped exception:
No such file or directory - A directory component in /etc/php/8.3/fpm/php.ini20260421-41712-drs5la.lock does not exist or is a dangling symbolic link
Error: /Stage[main]/Php/File[/etc/php/8.3/fpm/php.ini]/ensure: change from 'absent' to 'present' failed: Could not set 'present' on ensure: No such file or directory - A directory component in /etc/php/8.3/fpm/php.ini20260421-41712-drs5la.lock does not exist or is a dangling symbolic link (file: /srv/puppet_code/environments/production/modules/php/manifests/init.pp, line: 69)
Notice: /Stage[main]/Php::Fpm/File[/etc/php/8.3/fpm/php-fpm.conf]: Dependency Package[php8.3-fpm] has failures: true
Warning: /Stage[main]/Php::Fpm/File[/etc/php/8.3/fpm/php-fpm.conf]: Skipping because of failed dependencies
Warning: /Stage[main]/Php::Fpm/File[/etc/php/8.3/fpm/pool.d]: Skipping because of failed dependencies
Warning: /Stage[main]/Php::Default_extensions/Php::Extension[calendar]/File[/etc/php/8.3/mods-available/calendar.ini]: Skipping because of failed dependencies
Notice: /Stage[main]/Php::Default_extensions/Php::Extension[calendar]/File[/etc/php/8.3/cli/conf.d/20-calendar.ini]: Dependency File[/etc/php/8.3/cli/conf.d] has failures: true
Warning: /Stage[main]/Php::Default_extensions/Php::Extension[calendar]/File[/etc/php/8.3/cli/conf.d/20-calendar.ini]: Skipping because of failed dependencies
Notice: /Stage[main]/Php::Default_extensions/Php::Extension[calendar]/File[/etc/php/8.3/fpm/conf.d/20-calendar.ini]: Dependency File[/etc/php/8.3/fpm/conf.d] has failures: true
Warning: /Stage[main]/Php::Default_extensions/Php::Extension[calendar]/File[/etc/php/8.3/fpm/conf.d/20-calendar.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Php::Default_extensions/Php::Extension[ctype]/File[/etc/php/8.3/mods-available/ctype.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Php::Default_extensions/Php::Extension[ctype]/File[/etc/php/8.3/cli/conf.d/20-ctype.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Php::Default_extensions/Php::Extension[ctype]/File[/etc/php/8.3/fpm/conf.d/20-ctype.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Php::Default_extensions/Php::Extension[exif]/File[/etc/php/8.3/mods-available/exif.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Php::Default_extensions/Php::Extension[exif]/File[/etc/php/8.3/cli/conf.d/20-exif.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Php::Default_extensions/Php::Extension[exif]/File[/etc/php/8.3/fpm/conf.d/20-exif.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Php::Default_extensions/Php::Extension[fileinfo]/File[/etc/php/8.3/mods-available/fileinfo.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Php::Default_extensions/Php::Extension[fileinfo]/File[/etc/php/8.3/cli/conf.d/20-fileinfo.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Php::Default_extensions/Php::Extension[fileinfo]/File[/etc/php/8.3/fpm/conf.d/20-fileinfo.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Php::Default_extensions/Php::Extension[ftp]/File[/etc/php/8.3/mods-available/ftp.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Php::Default_extensions/Php::Extension[ftp]/File[/etc/php/8.3/cli/conf.d/20-ftp.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Php::Default_extensions/Php::Extension[ftp]/File[/etc/php/8.3/fpm/conf.d/20-ftp.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Php::Default_extensions/Php::Extension[gettext]/File[/etc/php/8.3/mods-available/gettext.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Php::Default_extensions/Php::Extension[gettext]/File[/etc/php/8.3/cli/conf.d/20-gettext.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Php::Default_extensions/Php::Extension[gettext]/File[/etc/php/8.3/fpm/conf.d/20-gettext.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Php::Default_extensions/Php::Extension[iconv]/File[/etc/php/8.3/mods-available/iconv.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Php::Default_extensions/Php::Extension[iconv]/File[/etc/php/8.3/cli/conf.d/20-iconv.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Php::Default_extensions/Php::Extension[iconv]/File[/etc/php/8.3/fpm/conf.d/20-iconv.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Php::Default_extensions/Php::Extension[phar]/File[/etc/php/8.3/mods-available/phar.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Php::Default_extensions/Php::Extension[phar]/File[/etc/php/8.3/cli/conf.d/20-phar.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Php::Default_extensions/Php::Extension[phar]/File[/etc/php/8.3/fpm/conf.d/20-phar.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Php::Default_extensions/Php::Extension[posix]/File[/etc/php/8.3/mods-available/posix.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Php::Default_extensions/Php::Extension[posix]/File[/etc/php/8.3/cli/conf.d/20-posix.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Php::Default_extensions/Php::Extension[posix]/File[/etc/php/8.3/fpm/conf.d/20-posix.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Php::Default_extensions/Php::Extension[readline]/File[/etc/php/8.3/mods-available/readline.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Php::Default_extensions/Php::Extension[readline]/File[/etc/php/8.3/cli/conf.d/20-readline.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Php::Default_extensions/Php::Extension[readline]/File[/etc/php/8.3/fpm/conf.d/20-readline.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Php::Default_extensions/Php::Extension[shmop]/File[/etc/php/8.3/mods-available/shmop.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Php::Default_extensions/Php::Extension[shmop]/File[/etc/php/8.3/cli/conf.d/20-shmop.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Php::Default_extensions/Php::Extension[shmop]/File[/etc/php/8.3/fpm/conf.d/20-shmop.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Php::Default_extensions/Php::Extension[sockets]/File[/etc/php/8.3/mods-available/sockets.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Php::Default_extensions/Php::Extension[sockets]/File[/etc/php/8.3/cli/conf.d/20-sockets.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Php::Default_extensions/Php::Extension[sockets]/File[/etc/php/8.3/fpm/conf.d/20-sockets.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Php::Default_extensions/Php::Extension[sysvmsg]/File[/etc/php/8.3/mods-available/sysvmsg.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Php::Default_extensions/Php::Extension[sysvmsg]/File[/etc/php/8.3/cli/conf.d/20-sysvmsg.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Php::Default_extensions/Php::Extension[sysvmsg]/File[/etc/php/8.3/fpm/conf.d/20-sysvmsg.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Php::Default_extensions/Php::Extension[sysvsem]/File[/etc/php/8.3/mods-available/sysvsem.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Php::Default_extensions/Php::Extension[sysvsem]/File[/etc/php/8.3/cli/conf.d/20-sysvsem.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Php::Default_extensions/Php::Extension[sysvsem]/File[/etc/php/8.3/fpm/conf.d/20-sysvsem.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Php::Default_extensions/Php::Extension[sysvshm]/File[/etc/php/8.3/mods-available/sysvshm.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Php::Default_extensions/Php::Extension[sysvshm]/File[/etc/php/8.3/cli/conf.d/20-sysvshm.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Php::Default_extensions/Php::Extension[sysvshm]/File[/etc/php/8.3/fpm/conf.d/20-sysvshm.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Php::Default_extensions/Php::Extension[tokenizer]/File[/etc/php/8.3/mods-available/tokenizer.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Php::Default_extensions/Php::Extension[tokenizer]/File[/etc/php/8.3/cli/conf.d/20-tokenizer.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Php::Default_extensions/Php::Extension[tokenizer]/File[/etc/php/8.3/fpm/conf.d/20-tokenizer.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Php::Default_extensions/Php::Extension[pdo]/File[/etc/php/8.3/mods-available/pdo.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Php::Default_extensions/Php::Extension[pdo]/File[/etc/php/8.3/cli/conf.d/10-pdo.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Php::Default_extensions/Php::Extension[pdo]/File[/etc/php/8.3/fpm/conf.d/10-pdo.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Php::Default_extensions/Php::Extension[opcache]/File[/etc/php/8.3/mods-available/opcache.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Php::Default_extensions/Php::Extension[opcache]/File[/etc/php/8.3/cli/conf.d/10-opcache.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Php::Default_extensions/Php::Extension[opcache]/File[/etc/php/8.3/fpm/conf.d/10-opcache.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Fpm::Pool[www-8.3]/File[/etc/php/8.3/fpm/pool.d/www.conf]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[bcmath]/File[/etc/php/8.3/mods-available/bcmath.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[bcmath]/Package[php8.3-bcmath]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[bcmath]/File[/etc/php/8.3/cli/conf.d/20-bcmath.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[bcmath]/File[/etc/php/8.3/fpm/conf.d/20-bcmath.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[bz2]/File[/etc/php/8.3/mods-available/bz2.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[bz2]/Package[php8.3-bz2]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[bz2]/File[/etc/php/8.3/cli/conf.d/20-bz2.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[bz2]/File[/etc/php/8.3/fpm/conf.d/20-bz2.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[curl]/File[/etc/php/8.3/mods-available/curl.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[curl]/Package[php8.3-curl]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[curl]/File[/etc/php/8.3/cli/conf.d/20-curl.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[curl]/File[/etc/php/8.3/fpm/conf.d/20-curl.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[gd]/File[/etc/php/8.3/mods-available/gd.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[gd]/Package[php8.3-gd]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[gd]/File[/etc/php/8.3/cli/conf.d/20-gd.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[gd]/File[/etc/php/8.3/fpm/conf.d/20-gd.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[gmp]/File[/etc/php/8.3/mods-available/gmp.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[gmp]/Package[php8.3-gmp]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[gmp]/File[/etc/php/8.3/cli/conf.d/20-gmp.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[gmp]/File[/etc/php/8.3/fpm/conf.d/20-gmp.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[intl]/File[/etc/php/8.3/mods-available/intl.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[intl]/Package[php8.3-intl]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[intl]/File[/etc/php/8.3/cli/conf.d/20-intl.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[intl]/File[/etc/php/8.3/fpm/conf.d/20-intl.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[mbstring]/File[/etc/php/8.3/mods-available/mbstring.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[mbstring]/Package[php8.3-mbstring]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[mbstring]/File[/etc/php/8.3/cli/conf.d/20-mbstring.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[mbstring]/File[/etc/php/8.3/fpm/conf.d/20-mbstring.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[apcu]/File[/etc/php/8.3/mods-available/apcu.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[apcu]/Package[php8.3-apcu]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[apcu]/File[/etc/php/8.3/cli/conf.d/20-apcu.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[apcu]/File[/etc/php/8.3/fpm/conf.d/20-apcu.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[msgpack]/File[/etc/php/8.3/mods-available/msgpack.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[msgpack]/Package[php8.3-msgpack]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[msgpack]/File[/etc/php/8.3/cli/conf.d/20-msgpack.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[msgpack]/File[/etc/php/8.3/fpm/conf.d/20-msgpack.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[redis]/File[/etc/php/8.3/mods-available/redis.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[redis]/Package[php8.3-redis]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[redis]/File[/etc/php/8.3/cli/conf.d/20-redis.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[redis]/File[/etc/php/8.3/fpm/conf.d/20-redis.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[luasandbox]/File[/etc/php/8.3/mods-available/luasandbox.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[luasandbox]/Package[php8.3-luasandbox]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[luasandbox]/File[/etc/php/8.3/cli/conf.d/20-luasandbox.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[luasandbox]/File[/etc/php/8.3/fpm/conf.d/20-luasandbox.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[wikidiff2]/File[/etc/php/8.3/mods-available/wikidiff2.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[wikidiff2]/Package[php8.3-wikidiff2]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[wikidiff2]/File[/etc/php/8.3/cli/conf.d/20-wikidiff2.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[wikidiff2]/File[/etc/php/8.3/fpm/conf.d/20-wikidiff2.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[yaml]/File[/etc/php/8.3/mods-available/yaml.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[yaml]/Package[php8.3-yaml]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[yaml]/File[/etc/php/8.3/cli/conf.d/20-yaml.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[yaml]/File[/etc/php/8.3/fpm/conf.d/20-yaml.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[uuid]/File[/etc/php/8.3/mods-available/uuid.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[uuid]/Package[php8.3-uuid]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[uuid]/File[/etc/php/8.3/cli/conf.d/20-uuid.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[uuid]/File[/etc/php/8.3/fpm/conf.d/20-uuid.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[xml]/File[/etc/php/8.3/mods-available/xml.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[xml]/Package[php8.3-xml]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[xml]/File[/etc/php/8.3/cli/conf.d/15-xml.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[xml]/File[/etc/php/8.3/fpm/conf.d/15-xml.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[mysqli]/File[/etc/php/8.3/mods-available/mysqli.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[mysqli]/Package[php8.3-mysql]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[mysqli]/File[/etc/php/8.3/cli/conf.d/20-mysqli.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[mysqli]/File[/etc/php/8.3/fpm/conf.d/20-mysqli.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[dba]/File[/etc/php/8.3/mods-available/dba.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[dba]/Package[php8.3-dba]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[dba]/File[/etc/php/8.3/cli/conf.d/20-dba.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[dba]/File[/etc/php/8.3/fpm/conf.d/20-dba.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[memcached]/File[/etc/php/8.3/mods-available/memcached.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[memcached]/Package[php8.3-memcached]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[memcached]/File[/etc/php/8.3/cli/conf.d/25-memcached.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[memcached]/File[/etc/php/8.3/fpm/conf.d/25-memcached.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[igbinary]/File[/etc/php/8.3/mods-available/igbinary.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[igbinary]/Package[php8.3-igbinary]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[igbinary]/File[/etc/php/8.3/cli/conf.d/20-igbinary.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[igbinary]/File[/etc/php/8.3/fpm/conf.d/20-igbinary.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[pdo_mysql]/File[/etc/php/8.3/mods-available/pdo_mysql.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[pdo_mysql]/File[/etc/php/8.3/cli/conf.d/20-pdo_mysql.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[pdo_mysql]/File[/etc/php/8.3/fpm/conf.d/20-pdo_mysql.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[mysqlnd]/File[/etc/php/8.3/mods-available/mysqlnd.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[mysqlnd]/File[/etc/php/8.3/cli/conf.d/10-mysqlnd.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[mysqlnd]/File[/etc/php/8.3/fpm/conf.d/10-mysqlnd.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[dom]/File[/etc/php/8.3/mods-available/dom.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[dom]/File[/etc/php/8.3/cli/conf.d/20-dom.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[dom]/File[/etc/php/8.3/fpm/conf.d/20-dom.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[simplexml]/File[/etc/php/8.3/mods-available/simplexml.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[simplexml]/File[/etc/php/8.3/cli/conf.d/20-simplexml.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[simplexml]/File[/etc/php/8.3/fpm/conf.d/20-simplexml.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[xmlreader]/File[/etc/php/8.3/mods-available/xmlreader.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[xmlreader]/File[/etc/php/8.3/cli/conf.d/20-xmlreader.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[xmlreader]/File[/etc/php/8.3/fpm/conf.d/20-xmlreader.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[xmlwriter]/File[/etc/php/8.3/mods-available/xmlwriter.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[xmlwriter]/File[/etc/php/8.3/cli/conf.d/20-xmlwriter.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[xmlwriter]/File[/etc/php/8.3/fpm/conf.d/20-xmlwriter.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[xsl]/File[/etc/php/8.3/mods-available/xsl.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[xsl]/File[/etc/php/8.3/cli/conf.d/20-xsl.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[xsl]/File[/etc/php/8.3/fpm/conf.d/20-xsl.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[xhprof]/File[/etc/php/8.3/mods-available/xhprof.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[xhprof]/Package[php8.3-xhprof]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[xhprof]/File[/etc/php/8.3/cli/conf.d/30-xhprof.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[xhprof]/File[/etc/php/8.3/fpm/conf.d/30-xhprof.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[excimer]/File[/etc/php/8.3/mods-available/excimer.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[excimer]/Package[php8.3-excimer]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[excimer]/File[/etc/php/8.3/cli/conf.d/20-excimer.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[excimer]/File[/etc/php/8.3/fpm/conf.d/20-excimer.ini]: Skipping because of failed dependencies
Notice: /Stage[main]/Profile::Mediawiki::Php/Alternatives::Select[php]/Exec[update_alternative_php]: Dependency Package[php8.3-cli] has failures: true
Warning: /Stage[main]/Profile::Mediawiki::Php/Alternatives::Select[php]/Exec[update_alternative_php]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[wmerrors]/File[/etc/php/8.3/mods-available/wmerrors.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[wmerrors]/Package[php8.3-wmerrors]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/Php::Extension[wmerrors]/File[/etc/php/8.3/fpm/conf.d/20-wmerrors.ini]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/File[/etc/php/php7-fatal-error.php]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Mediawiki::Php/File[/etc/php/error-params.php]: Skipping because of failed dependencies
Warning: /Stage[main]/Php::Fpm/Service[php8.3-fpm]: Skipping because of failed dependencies
Notice: /Stage[main]/Envoyproxy/Systemd::Service[envoyproxy.service]/Service[envoyproxy.service]/ensure: ensure changed 'stopped' to 'running' (corrective)
Info: /Stage[main]/Envoyproxy/Systemd::Service[envoyproxy.service]/Service[envoyproxy.service]: Unscheduling refresh on Service[envoyproxy.service]
Info: Stage[main]: Unscheduling all events on Stage[main]
Notice: Applied catalog in 19.62 seconds

Well crap. Our Puppet config to install php 8.3 is extremely hard coded to specific package versions and those versions are in https://apt.wikimedia.org/wikimedia/dists/bullseye-wikimedia/component/php83/. There is no Bookworm equivalent component module.

bd808@deployment-deploy05:~$ sudo apt update
Hit:1 http://apt.wikimedia.org/wikimedia bookworm-wikimedia InRelease
Hit:2 http://deb.debian.org/debian bookworm-backports InRelease
Hit:3 http://security.debian.org/debian-security bookworm-security InRelease
Hit:4 http://deb.debian.org/debian-debug bookworm-debug InRelease
Hit:5 http://deb.debian.org/debian bookworm InRelease
Hit:6 http://deb.debian.org/debian bookworm-updates InRelease
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
All packages are up to date.
W: Skipping acquire of configured file 'component/php83/source/Sources' as repository 'http://apt.wikimedia.org/wikimedia bookworm-wikimedia InRelease' doesn't have the component 'component/php83' (component misspelt in sources.list?)
W: Skipping acquire of configured file 'component/php83/binary-amd64/Packages' as repository 'http://apt.wikimedia.org/wikimedia bookworm-wikimedia InRelease' doesn't have the component 'component/php83' (component misspelt in sources.list?)
W: Skipping acquire of configured file 'component/php83/i18n/Translation-en' as repository 'http://apt.wikimedia.org/wikimedia bookworm-wikimedia InRelease' doesn't have the component 'component/php83' (component misspelt in sources.list?)

#wikimedia-sre-collab

[23:47]  <    bd808> mutante: we have a problem with the Jenkins switch. We need bookworm to get the jdk runtime version, but in bookworm we don't have the local php8.3 apt component that has the php packages we need to build a deploy server for Beta.
[23:47]  <    bd808> https://phabricator.wikimedia.org/T421244#11847986
[23:48]  <    bd808> so.... we need to either get all that php stuff packaged in apt, or we need to find a new way to run the deployment steps that Jenkins does for us now in Beta.
[23:48]  <    bd808> thcipriani: ^^

@thcipriani and I chatted a bit about what now. We are thinking T256168: Move beta cluster automatic deployment to a dedicated infrastructure is probably the "best" pivot.

Dzahn added a parent task: T418521: setup 2 contint machines for jenkins.Apr 23 2026, 3:23 PM

• bd808 mentioned this in T256168: Move beta cluster automatic deployment to a dedicated infrastructure.Apr 23 2026, 4:34 PM

bd808 opened https://gitlab.wikimedia.org/cloudvps-repos/deployment-prep/tofu-provisioning/-/merge_requests/9

deployment-deploy05: decomm host

bd808 merged https://gitlab.wikimedia.org/cloudvps-repos/deployment-prep/tofu-provisioning/-/merge_requests/9

deployment-deploy05: decomm host

Declined in favor of T256168: Move beta cluster automatic deployment to a dedicated infrastructure per our finding of PHP issues for a quick switch. Blocking on T397075: Package Wikimedia's PHP 8.3 component for bookworm does not seem like a good idea when we know that eventually we need to break with Jenkins anyway.

How about using the sury.org packages as is recommended in T397075#11655976 ?

We could add the sury repo in beta?

• bd808 mentioned this in T424288: Found non-revoked Puppet certificates for 1 deleted instances on deployment-puppetserver-1.Apr 24 2026, 4:37 PM

• bd808 mentioned this in T424115: Puppet agent failure detected on instance deployment-deploy05 in project deployment-prep.Apr 24 2026, 5:17 PM

dancy mentioned this in T429413: Eliminate sudo rm -rf /var/lib/puppet/ssl step in new deployment-prep WMCS project (and others).Jun 16 2026, 10:07 PM

• bd808 mentioned this in T433998: Planning for a plan to migrate Beta Cluster again.Aug 4 2026, 5:21 PM

• bd808 mentioned this in T435393: Beta Cluster: Switch to PHP 8.5, includes writing Puppet changes to support PHP 8.5 on baremetal.Aug 19 2026, 7:30 PM

In T421244#11847209, @bd808 wrote:

Booleans were coerced to strings somewhere???

yamlencode({"a": true, "c": false}) works as expected in tofu console. Did I do something weird in the provider when I worked on clean round tripping?

This bug ended up being related to https://gitlab.wikimedia.org/cloudvps-repos/deployment-prep/tofu-provisioning/-/commit/e80983d096d50f7c2a1d443168eb293214753f50. The map(any) typing for hiera settings was coercing all values to strings after seeing the first value in the map as a string.

Replace deployment-deploy04 with a Bookworm instance with Java 21Closed, DeclinedPublicFeatureActions

Description

Details

Related ObjectsSearch...

Event Timeline

Replace deployment-deploy04 with a Bookworm instance with Java 21
Closed, DeclinedPublicFeature
Actions

Related Objects
Search...