Alright currently we have only one host in eqiad wdqs-internal-scholarly; we should bump that to 2, and leave the rest of the hosts to public-main which actually needs the compute:

wdqs1033: wdqs-main
wdqs1034: wdqs-main
wdqs1035: wdqs-internal-scholarly

(while spelunking, I noticed that we originally planned wdqs1022 to move to be a wdqs-internal-scholarly host, but at some point we forgot that, so this will implicitly correct the minor imbalance)

Change #1345041 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] wdqs: restore internal scholarly transfer peers

https://gerrit.wikimedia.org/r/1345041

Change #1345041 merged by Ryan Kemper:

[operations/puppet@production] wdqs: restore internal scholarly transfer peers

https://gerrit.wikimedia.org/r/1345041

Change #1345043 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] wdqs: configure eqiad expansion hosts

https://gerrit.wikimedia.org/r/1345043

Cookbook cookbooks.sre.hosts.reimage was started by ryankemper@cumin2003 for host wdqs1033.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by ryankemper@cumin2003 for host wdqs1033.eqiad.wmnet with OS bookworm completed:

• wdqs1033 (PASS)
  • Downtimed on Icinga/Alertmanager
  • Disabled Puppet
  • Host successfully migrated to the new VLAN
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced UEFI HTTP Boot for next reboot
  • Host rebooted via Redfish
  • Host up (Debian installer)
  • Host up (new fresh bookworm OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • Removed previous downtime on Alertmanager (old OS)
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202609290447_ryankemper_1569548_wdqs1033.out
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is optimal
  • Icinga downtime removed
  • Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage was started by ryankemper@cumin2003 for host wdqs1034.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage was started by ryankemper@cumin2003 for host wdqs1035.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by ryankemper@cumin2003 for host wdqs1034.eqiad.wmnet with OS bookworm completed:

• wdqs1034 (PASS)
  • Downtimed on Icinga/Alertmanager
  • Disabled Puppet
  • Host successfully migrated to the new VLAN
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced UEFI HTTP Boot for next reboot
  • Host rebooted via Redfish
  • Host up (Debian installer)
  • Host up (new fresh bookworm OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • Removed previous downtime on Alertmanager (old OS)
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202609290534_ryankemper_1579454_wdqs1034.out
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is optimal
  • Icinga downtime removed
  • Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage started by ryankemper@cumin2003 for host wdqs1035.eqiad.wmnet with OS bookworm completed:

• wdqs1035 (PASS)
  • Downtimed on Icinga/Alertmanager
  • Disabled Puppet
  • Host successfully migrated to the new VLAN
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced UEFI HTTP Boot for next reboot
  • Host rebooted via Redfish
  • Host up (Debian installer)
  • Host up (new fresh bookworm OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • Removed previous downtime on Alertmanager (old OS)
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202609290538_ryankemper_1579668_wdqs1035.out
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is optimal
  • Icinga downtime removed
  • Updated Netbox data from PuppetDB

Change #1345043 merged by Ryan Kemper:

[operations/puppet@production] wdqs: configure eqiad expansion hosts

https://gerrit.wikimedia.org/r/1345043

Mentioned in SAL (#wikimedia-operations) [2026-09-29T16:54:27Z] <ryankemper@deploy1003> Started deploy [wdqs/wdqs@cf64fe9]: T420707: deploy to fresh wdqs host

Mentioned in SAL (#wikimedia-operations) [2026-09-29T16:54:32Z] <ryankemper@deploy1003> Finished deploy [wdqs/wdqs@cf64fe9]: T420707: deploy to fresh wdqs host (duration: 00m 04s)

Mentioned in SAL (#wikimedia-operations) [2026-09-29T16:54:36Z] <ryankemper@deploy1003> Started deploy [wdqs/wdqs@cf64fe9]: T420707: deploy to fresh wdqs host

Mentioned in SAL (#wikimedia-operations) [2026-09-29T16:54:40Z] <ryankemper@deploy1003> Finished deploy [wdqs/wdqs@cf64fe9]: T420707: deploy to fresh wdqs host (duration: 00m 04s)

Mentioned in SAL (#wikimedia-operations) [2026-09-29T16:54:47Z] <ryankemper@deploy1003> Started deploy [wdqs/wdqs@cf64fe9]: T420707: deploy to fresh wdqs host

Mentioned in SAL (#wikimedia-operations) [2026-09-29T16:54:51Z] <ryankemper@deploy1003> Finished deploy [wdqs/wdqs@cf64fe9]: T420707: deploy to fresh wdqs host (duration: 00m 04s)

Mentioned in SAL (#wikimedia-operations) [2026-09-29T17:00:56Z] <ryankemper@cumin2003> START - Cookbook sre.wdqs.data-transfer (T420707, T420707: initial data for new host) xfer wdqs-all from wdqs2021.codfw.wmnet -> wdqs1033.eqiad.wmnet, repooling source-only afterwards

Mentioned in SAL (#wikimedia-operations) [2026-09-29T17:01:13Z] <ryankemper@cumin2003> START - Cookbook sre.wdqs.data-transfer (T420707, T420707: initial data for new host) xfer wdqs-all from wdqs2013.codfw.wmnet -> wdqs1034.eqiad.wmnet, repooling source-only afterwards

Mentioned in SAL (#wikimedia-operations) [2026-09-29T17:01:42Z] <ryankemper@cumin2003> START - Cookbook sre.wdqs.data-transfer (T420707, T420707: initial data for new host) xfer scholarly_articles from wdqs2023.codfw.wmnet -> wdqs1035.eqiad.wmnet, repooling source-only afterwards

Mentioned in SAL (#wikimedia-operations) [2026-09-29T17:59:18Z] <ryankemper@cumin2003> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T420707, T420707: initial data for new host) xfer scholarly_articles from wdqs2023.codfw.wmnet -> wdqs1035.eqiad.wmnet, repooling source-only afterwards

Mentioned in SAL (#wikimedia-operations) [2026-09-29T18:01:32Z] <ryankemper@cumin2003> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T420707, T420707: initial data for new host) xfer wdqs-all from wdqs2013.codfw.wmnet -> wdqs1034.eqiad.wmnet, repooling source-only afterwards

Mentioned in SAL (#wikimedia-operations) [2026-09-29T18:08:48Z] <ryankemper@cumin2003> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T420707, T420707: initial data for new host) xfer wdqs-all from wdqs2021.codfw.wmnet -> wdqs1033.eqiad.wmnet, repooling source-only afterwards