Page MenuHomePhabricator

Collaboration Services: Re-IP eqiad private baremetal hosts to new per-rack vlans/subnets
Closed, ResolvedPublic

Description

eqiad rows C and D have been migrated to new Nokia switches, and more importantly to the new network design.

You can find all the information on https://wikitech.wikimedia.org/wiki/Vlan_migration

All new servers are by default getting in those new vlans, but to not have to wait for a full 5+ years server refresh cycle, we're now asking service owners to re-image their existing baremetal servers using the --move-vlan cookbook parameter, at their own pace/convenience. This will change the server's IP.
There will of course be some special cases (like Ganeti, or DBs) and that's ok to not covert 100% of the servers, but the most we can get, the better.

Please contact netops for any help.

cumin1003:~$ sudo cumin 'A:owner-collaboration-services and P{P:netbox::host%location ~ "[C|D].*eqiad"} and P{F:fqdn ~ ".wmnet$"} and not A:vms and not P{F:netmask = "255.255.255.0"}'
2 hosts will be targeted:
gitlab-runner[1003-1004].eqiad.wmnet
  • gitlab-runner[1003-1004].eqiad.wmnet

Event Timeline

Mentioned in SAL (#wikimedia-operations) [2026-04-06T18:39:08Z] <mutante> gitlab-runner1004 - reimaging with --move-vlan T421717

Cookbook cookbooks.sre.hosts.reimage was started by dzahn@cumin2002 for host gitlab-runner1004.eqiad.wmnet with OS bookworm

Change #1268265 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] docker_registry: update IP of gitlab-runner1004 for jwt-auth'ed hosts

https://gerrit.wikimedia.org/r/1268265

Change #1268265 merged by Dzahn:

[operations/puppet@production] docker_registry: update IP of gitlab-runner1004 for jwt-auth'ed hosts

https://gerrit.wikimedia.org/r/1268265

Cookbook cookbooks.sre.hosts.reimage started by dzahn@cumin2002 for host gitlab-runner1004.eqiad.wmnet with OS bookworm completed:

  • gitlab-runner1004 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604061909_dzahn_2996862_gitlab-runner1004.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB

confirmed gitlab-runner1004 is online after reimaging and IP change, as 2620:0:861:144:10:64:189:2

in the trusted runner group

https://gitlab.wikimedia.org/admin/runners/1484


continue with gitlab-runner1003

Cookbook cookbooks.sre.hosts.reimage was started by dzahn@cumin2002 for host gitlab-runner1003.eqiad.wmnet with OS bookworm

Change #1268288 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] docker_registry: update IP of gitlab-runner1003 for jwt-auth'ed hosts

https://gerrit.wikimedia.org/r/1268288

Change #1268288 merged by Dzahn:

[operations/puppet@production] docker_registry: update IP of gitlab-runner1003 for jwt-auth'ed hosts

https://gerrit.wikimedia.org/r/1268288

Mentioned in SAL (#wikimedia-operations) [2026-04-06T23:25:14Z] <mutante> gitlab: reimaging trusted runners with --move-vlan parameter which changed their IPs - verified was showing up as online after the change and using the new IPs (T421717)

Cookbook cookbooks.sre.hosts.reimage started by dzahn@cumin2002 for host gitlab-runner1003.eqiad.wmnet with OS bookworm completed:

  • gitlab-runner1003 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604062343_dzahn_3133281_gitlab-runner1003.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Dzahn updated the task description. (Show Details)

done!

[cumin2002:~] $ sudo cumin 'A:owner-collaboration-services and P{P:netbox::host%location ~ "[C|D].*eqiad"} and P{F:fqdn ~ ".wmnet$"} and not A:vms and not P{F:netmask = "255.255.255.0"}'

No hosts found that matches the query