Page MenuHomePhabricator
Create Task
Maniphest T421714

Data platform: Re-IP eqiad private baremetal hosts to new per-rack vlans/subnets
Open, MediumPublic
Actions

Description

eqiad rows C and D have been migrated to new Nokia switches, and more importantly to the new network design.

You can find all the information on https://wikitech.wikimedia.org/wiki/Vlan_migration

All new servers are by default getting in those new vlans, but to not have to wait for a full 5+ years server refresh cycle, we're now asking service owners to re-image their existing baremetal servers using the --move-vlan cookbook parameter, at their own pace/convenience. This will change the server's IP.
There will of course be some special cases (like Ganeti, or DBs) and that's ok to not convert 100% of the servers, but the most we can get, the better.

Please contact netops for any help.

cumin1003:~$ sudo cumin 'A:owner-data-platform and P{P:netbox::host%location ~ "[C|D].*eqiad"} and P{F:fqdn ~ ".wmnet$"} and not A:vms and not P{F:netmask = "255.255.255.0"}'
10 hosts will be targeted:
an-redacteddb1001.eqiad.wmnet,dbstore1007.eqiad.wmnet,druid[1012-1013].eqiad.wmnet,dse-k8s-worker[1003-1004].eqiad.wmnet,ganeti-jumbo1001.eqiad.wmnet,krb1002.eqiad.wmnet,wdqs[1029-1030].eqiad.wmnet
  • an-redacteddb1001.eqiad.wmnet
  • dbstore1007.eqiad.wmnet
  • druid[1012-1013].eqiad.wmnet
  • dse-k8s-worker[1003-1004].eqiad.wmnet
  • ganeti-jumbo1001.eqiad.wmnet
  • cirrussearch1080
  • cirrussearch1081
  • cirrussearch1082
  • cirrussearch1083
  • cirrussearch1087
  • cirrussearch1088
  • cirrussearch1103
  • cirrussearch11[18-22]
  • cloudelastic[1009-1010].eqiad.wmnet
  • relforge[1008-1009].eqiad.wmnet - per conversation with Search platform, these are in use and can't be reimaged yet. We'll ask again once we're done with all the other hosts.
  • wcqs1003.eqiad.wmnet
  • wdqs[1013-1014,1017,1022,1027].eqiad.wmnet
  • wdqs[1029-1030].eqiad.wmnet - // see subticket

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
opensearch: handle IP changes for software firewallBkingoperations/puppetproduction+2 -3
Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
ayounsi added a comment.Apr 1 2026, 6:46 AM
In T421714#11774619, @BTullis wrote:

I can also take the druid[1012-1013].eqiad.wmnet hosts.

I think that @MoritzMuehlenhoff will likely take krb1002.eqiad.wmnet

That one is also tracked in T421706: Infrastructure foundations : Re-IP eqiad private baremetal hosts to new per-rack vlans/subnets

MoritzMuehlenhoff updated the task description. (Show Details)Apr 1 2026, 7:24 AM
In T421714#11776340, @ayounsi wrote:
In T421714#11774619, @BTullis wrote:

I can also take the druid[1012-1013].eqiad.wmnet hosts.

I think that @MoritzMuehlenhoff will likely take krb1002.eqiad.wmnet

That one is also tracked in T421706: Infrastructure foundations : Re-IP eqiad private baremetal hosts to new per-rack vlans/subnets

I've remove it from this task. We can migrate krb1002 as part of common maintenance window when we add the hardware refresh for krb2001.

ops-monitoring-bot added a comment.Apr 1 2026, 8:43 AM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1003 for host an-druid1003.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Apr 1 2026, 9:31 AM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1003 for host an-druid1003.eqiad.wmnet with OS bookworm completed:

  • an-druid1003 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604010903_btullis_763821_an-druid1003.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Apr 1 2026, 9:57 AM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1003 for host an-druid1004.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Apr 1 2026, 10:47 AM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1003 for host an-druid1004.eqiad.wmnet with OS bookworm completed:

  • an-druid1004 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604011019_btullis_841629_an-druid1004.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
BTullis updated the task description. (Show Details)Apr 1 2026, 11:21 AM
bking updated the task description. (Show Details)Apr 1 2026, 1:50 PM
bking mentioned this in T421718: Search Platform: Re-IP eqiad private baremetal hosts to new per-rack vlans/subnets.Apr 1 2026, 1:52 PM
bking merged a task: T421718: Search Platform: Re-IP eqiad private baremetal hosts to new per-rack vlans/subnets.
bking added subscribers: EBernhardson, dcausse.
ops-monitoring-bot added a comment.Apr 1 2026, 1:59 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host wcqs1003.eqiad.wmnet with OS bullseye

MoritzMuehlenhoff unsubscribed.Apr 1 2026, 2:07 PM
ops-monitoring-bot added a comment.Apr 1 2026, 2:58 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host wcqs1003.eqiad.wmnet with OS bullseye completed:

  • wcqs1003 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604011428_bking_3639818_wcqs1003.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Apr 1 2026, 3:20 PM

Mentioned in SAL (#wikimedia-operations) [2026-04-01T15:20:55Z] <bking@cumin2002> START - Cookbook sre.wdqs.data-transfer (T421714, prepare newly-reimaged host) xfer wikidata from wcqs1001.eqiad.wmnet -> wcqs1003.eqiad.wmnet, repooling both afterwards

Stashbot added a comment.Apr 1 2026, 3:22 PM

Mentioned in SAL (#wikimedia-operations) [2026-04-01T15:22:13Z] <bking@cumin2002> END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) (T421714, prepare newly-reimaged host) xfer wikidata from wcqs1001.eqiad.wmnet -> wcqs1003.eqiad.wmnet, repooling both afterwards

Stashbot added a comment.Apr 1 2026, 3:22 PM

Mentioned in SAL (#wikimedia-operations) [2026-04-01T15:22:16Z] <bking@cumin2002> START - Cookbook sre.wdqs.data-transfer (T421714, prepare newly-reimaged host) xfer commons from wcqs1001.eqiad.wmnet -> wcqs1003.eqiad.wmnet, repooling both afterwards

Stashbot added a comment.Apr 1 2026, 3:22 PM

Mentioned in SAL (#wikimedia-operations) [2026-04-01T15:22:22Z] <bking@cumin2002> END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) (T421714, prepare newly-reimaged host) xfer commons from wcqs1001.eqiad.wmnet -> wcqs1003.eqiad.wmnet, repooling both afterwards

Stashbot added a comment.Apr 1 2026, 3:23 PM

Mentioned in SAL (#wikimedia-operations) [2026-04-01T15:23:38Z] <bking@cumin2002> START - Cookbook sre.wdqs.data-transfer (T421714, prepare newly-reimaged host) xfer commons from wcqs1001.eqiad.wmnet -> wcqs1003.eqiad.wmnet, repooling both afterwards

Stashbot added a comment.Apr 1 2026, 3:23 PM

Mentioned in SAL (#wikimedia-operations) [2026-04-01T15:23:41Z] <bking@cumin2002> END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) (T421714, prepare newly-reimaged host) xfer commons from wcqs1001.eqiad.wmnet -> wcqs1003.eqiad.wmnet, repooling both afterwards

Stashbot added a comment.Apr 1 2026, 3:24 PM

Mentioned in SAL (#wikimedia-operations) [2026-04-01T15:24:07Z] <bking@cumin2002> START - Cookbook sre.wdqs.data-transfer (T421714, prepare newly-reimaged host) xfer commons from wcqs1001.eqiad.wmnet -> wcqs1003.eqiad.wmnet, repooling both afterwards

bking updated the task description. (Show Details)Apr 1 2026, 3:59 PM
Stashbot added a comment.Apr 1 2026, 4:36 PM

Mentioned in SAL (#wikimedia-operations) [2026-04-01T16:36:16Z] <bking@cumin2002> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T421714, prepare newly-reimaged host) xfer commons from wcqs1001.eqiad.wmnet -> wcqs1003.eqiad.wmnet, repooling both afterwards

bking updated the task description. (Show Details)Apr 1 2026, 5:29 PM
ops-monitoring-bot added a comment.Apr 1 2026, 5:32 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host cloudelastic1009.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 1 2026, 6:59 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host cloudelastic1009.eqiad.wmnet with OS bullseye completed:

  • cloudelastic1009 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604011810_bking_3684606_cloudelastic1009.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Apr 1 2026, 7:31 PM

Change #1266372 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] opensearch: handle IP changes for software firewall

https://gerrit.wikimedia.org/r/1266372

gerritbot added a project: Patch-For-Review.Apr 1 2026, 7:31 PM
gmodena subscribed.Apr 1 2026, 7:36 PM
bking updated the task description. (Show Details)Apr 1 2026, 7:46 PM
bking updated the task description. (Show Details)Apr 1 2026, 7:50 PM
ops-monitoring-bot added a comment.Apr 1 2026, 7:52 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host cloudelastic1010.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 1 2026, 9:07 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host cloudelastic1010.eqiad.wmnet with OS bullseye completed:

  • cloudelastic1010 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604012023_bking_3733229_cloudelastic1010.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
bking updated the task description. (Show Details)Apr 1 2026, 9:08 PM
ops-monitoring-bot added a comment.Apr 1 2026, 9:33 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host wdqs1027.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 1 2026, 11:02 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host wdqs1027.eqiad.wmnet with OS bullseye completed:

  • wdqs1027 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604012238_bking_3772654_wdqs1027.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Apr 2 2026, 7:47 AM

Mentioned in SAL (#wikimedia-operations) [2026-04-02T07:47:40Z] <bking@cumin2002> START - Cookbook sre.wdqs.data-transfer (, T421714) xfer wdqs-all from wdqs2016.codfw.wmnet -> wdqs1027.eqiad.wmnet, repooling both afterwards

gerritbot added a comment.Apr 2 2026, 2:09 PM

Change #1266372 merged by Bking:

[operations/puppet@production] opensearch: handle IP changes for software firewall

https://gerrit.wikimedia.org/r/1266372

Maintenance_bot removed a project: Patch-For-Review.Apr 2 2026, 2:31 PM
ops-monitoring-bot added a comment.Apr 7 2026, 6:43 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host cirrussearch1080.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 7 2026, 7:07 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host cirrussearch1082.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 7 2026, 7:26 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host cirrussearch1080.eqiad.wmnet with OS bullseye completed:

  • cirrussearch1080 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604071908_bking_3714754_cirrussearch1080.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Apr 7 2026, 8:02 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host cirrussearch1082.eqiad.wmnet with OS bullseye completed:

  • cirrussearch1082 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604071943_bking_3731080_cirrussearch1082.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
bking updated the task description. (Show Details)Apr 7 2026, 8:29 PM
bking updated the task description. (Show Details)
ops-monitoring-bot added a comment.Apr 7 2026, 9:00 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host cirrussearch1083.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 7 2026, 9:52 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host cirrussearch1083.eqiad.wmnet with OS bullseye completed:

  • cirrussearch1083 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604072133_bking_3808066_cirrussearch1083.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
bking updated the task description. (Show Details)Apr 7 2026, 9:59 PM
ops-monitoring-bot added a comment.Apr 8 2026, 3:05 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host cirrussearch1081.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 8 2026, 3:19 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host cirrussearch1087.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 8 2026, 4:00 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host cirrussearch1081.eqiad.wmnet with OS bullseye completed:

  • cirrussearch1081 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604081535_bking_312875_cirrussearch1081.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Apr 8 2026, 4:07 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host cirrussearch1087.eqiad.wmnet with OS bullseye completed:

  • cirrussearch1087 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604081549_bking_321458_cirrussearch1087.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
bking updated the task description. (Show Details)Apr 8 2026, 4:59 PM
ops-monitoring-bot added a comment.Apr 8 2026, 5:00 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host cirrussearch1088.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 8 2026, 5:35 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host cirrussearch1089.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 8 2026, 5:36 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host cirrussearch1089.eqiad.wmnet with OS bullseye executed with errors:

  • cirrussearch1089 (FAIL)
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cirrussearch1089.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Apr 8 2026, 5:36 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host cirrussearch1103.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 8 2026, 5:49 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host cirrussearch1088.eqiad.wmnet with OS bullseye completed:

  • cirrussearch1088 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604081729_bking_399042_cirrussearch1088.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Apr 8 2026, 6:25 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host cirrussearch1103.eqiad.wmnet with OS bullseye completed:

  • cirrussearch1103 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604081804_bking_422082_cirrussearch1103.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
bking updated the task description. (Show Details)Apr 8 2026, 6:26 PM
ops-monitoring-bot added a comment.Apr 8 2026, 6:32 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host cirrussearch1103.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 8 2026, 7:14 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host cirrussearch1103.eqiad.wmnet with OS bullseye completed:

  • cirrussearch1103 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604081854_bking_462543_cirrussearch1103.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Apr 13 2026, 1:19 PM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1003 for host an-redacteddb1001.eqiad.wmnet with OS bookworm

BTullis updated the task description. (Show Details)Apr 13 2026, 2:18 PM
ops-monitoring-bot added a comment.Apr 13 2026, 2:20 PM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1003 for host an-redacteddb1001.eqiad.wmnet with OS bookworm completed:

  • an-redacteddb1001 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604131359_btullis_2391205_an-redacteddb1001.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Gehel edited projects, added: Data-Platform-SRE (2026-04-24 - 2026-05-15); removed: Data-Platform-SRE (2026-03-27 - 2026-04-17).Apr 24 2026, 9:16 AM
Gehel moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2026-04-24 - 2026-05-15) board.
Gehel edited projects, added: Data-Platform-SRE (2026-06-05 - 2026-06-26); removed: Data-Platform-SRE (2026-04-24 - 2026-05-15).Jun 5 2026, 1:28 PM
Gehel moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2026-06-05 - 2026-06-26) board.
MLechvien-WMF mentioned this in T421721: Machine Learning: Re-IP eqiad private baremetal hosts to new per-rack vlans/subnets.Jun 26 2026, 7:06 AM
bking updated the task description. (Show Details)Jun 26 2026, 2:32 PM
Gehel edited projects, added: Data-Platform-SRE (2026-07-03 - 2026-07-31); removed: Data-Platform-SRE (2026-06-05 - 2026-06-26).Jul 3 2026, 9:46 AM
Gehel moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2026-07-03 - 2026-07-31) board.
bking updated the task description. (Show Details)Jul 9 2026, 4:15 PM
bking updated the task description. (Show Details)Jul 10 2026, 2:08 PM
Gehel edited projects, added: Data-Platform-SRE (2026-08-07 - 2026-08-28); removed: Data-Platform-SRE (2026-07-03 - 2026-07-31).Aug 11 2026, 6:55 AM
Gehel moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2026-08-07 - 2026-08-28) board.
Gehel edited projects, added: Data-Platform-SRE (2026-08-28 - 2026-09-18); removed: Data-Platform-SRE (2026-08-07 - 2026-08-28).Aug 28 2026, 1:00 PM
Gehel moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2026-08-28 - 2026-09-18) board.
bking changed the task status from In Progress to Open.Aug 31 2026, 10:18 PM
bking removed bking as the assignee of this task.
bking created subtask T436571: Re-IP Search Platform-owned EQIAD hosts to new per-rack vlans/subnets.
bking updated the task description. (Show Details)
bking updated Other Assignee, removed: BTullis.
bking updated the task description. (Show Details)
bking added a comment.Aug 31 2026, 10:22 PM

Most of this is done, however there are still a few hosts left :

 sudo cumin 'A:owner-data-platform and P{P:netbox::host%location ~ "[C|D].*eqiad"} and P{F:fqdn ~ ".wmnet$"} and not A:vms and not P{F:netmask = "255.255.255.0"}'
6 hosts will be targeted:
cirrussearch1120.eqiad.wmnet,dbstore1007.eqiad.wmnet,druid[1012-1013].eqiad.wmnet,wdqs[1029-1030].eqiad.wmnet
DRY-RUN mode enabled, aborting

I've created subticket T436571 for cirrussearch1120.eqiad.wmnet,wdqs[1029-1030].eqiad.wmnet . We can talk about the remaining hosts at stand-up tomorrow, but I'm going to unassign the main ticket and move it back to the backlog as there has not been any movement in some time.

bking moved this task from In Progress to Backlog - operations on the Data-Platform-SRE (2026-08-28 - 2026-09-18) board.Aug 31 2026, 10:22 PM
Gehel added a project: Essential-Work.Wed, Sep 16, 9:42 AM
Gehel edited projects, added: Data-Platform-SRE (2026-09-18 - 2026-10-09); removed: Data-Platform-SRE (2026-08-28 - 2026-09-18).Fri, Sep 18, 1:16 PM
Gehel moved this task from Backlog - project to Backlog - operations on the Data-Platform-SRE (2026-09-18 - 2026-10-09) board.
bking changed the status of subtask T436571: Re-IP Search Platform-owned EQIAD hosts to new per-rack vlans/subnets from Open to In Progress.Tue, Sep 22, 6:38 PM
bking closed subtask T436571: Re-IP Search Platform-owned EQIAD hosts to new per-rack vlans/subnets as Resolved.Tue, Sep 22, 9:22 PM
Gehel edited projects, added: Data-Platform-SRE (2026-10-09 - 2026-10-30); removed: Data-Platform-SRE (2026-09-18 - 2026-10-09).Fri, Oct 9, 8:41 AM
Gehel moved this task from Backlog - project to Backlog - operations on the Data-Platform-SRE (2026-10-09 - 2026-10-30) board.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits