Page MenuHomePhabricator
Create Task
Maniphest T428229

eqsin: re-image rack 604 servers on new vlan
Closed, ResolvedPublic
Actions

Description

Forking from T427393: EQSIN: Setup VRRP on both routers for the new subnets

This is a requirement for the switch replacement as the new network design have per rack subnets.
That will make the switch replacement much more smooth and minimize downtime as reimages/re-IP won't be necessary *after* the work.

TL;DR; please run the reimage cookbook with --move-vlan on them.

If they can't be reimaged, then the move-vlan cookbook with --inplace (but only if a re-image isn't possible).

  • cp5018
  • cp5020
  • cp5024
  • cp5026
  • cp5028
  • cp5030
  • cp5032
  • dns5004
  • ganeti5004
  • ganeti5006
  • lvs5005

https://netbox.wikimedia.org/dcim/devices/?rack_id=78&status=active&role_id=1

Details

Other Assignee
BCornwall
Related Changes in Gerrit:
Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
 OpenPapaulT418439 EQSIN:New switch setup/configuration
 ResolvedNoneT428229 eqsin: re-image rack 604 servers on new vlan

Event Timeline

ayounsi created this task.Jun 5 2026, 6:30 AM
ayounsi triaged this task as High priority.
Restricted Application added a subscriber: • Aklapper. · View Herald TranscriptJun 5 2026, 6:30 AM
ayounsi added a parent task: T418439: EQSIN:New switch setup/configuration.Jun 5 2026, 6:30 AM
ayounsi mentioned this in T427393: EQSIN: Setup VRRP on both routers for the new subnets.
BCornwall updated Other Assignee, added: BCornwall.Jun 5 2026, 10:22 PM
Papaul subscribed.Jun 9 2026, 3:51 PM

@BCornwall hello just wanted to follow up on this to know when do you think you or your team will be done with the re-image. Our goal is to have all the re-image done before the new switches arrived which is some times close to the end of this month. Thanks.

BCornwall added a comment.Jun 9 2026, 4:05 PM

Hi! We'll be working on these shortly, I was planning on starting today.

gerritbot added a comment.Jun 9 2026, 5:07 PM

Change #1299579 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] common: Update cp5018's IP address

https://gerrit.wikimedia.org/r/1299579

gerritbot added a project: Patch-For-Review.Jun 9 2026, 5:07 PM
Papaul added a comment.Jun 9 2026, 6:33 PM

Thank you

ops-monitoring-bot added a comment.Jun 10 2026, 5:41 AM

Draining ganeti5004.eqsin.wmnet of running VMs

ops-monitoring-bot added a comment.Jun 10 2026, 5:43 AM

Draining ganeti5004.eqsin.wmnet of running VMs

ops-monitoring-bot added a comment.Jun 10 2026, 9:20 AM

VM prometheus5003.eqsin.wmnet switching disk type to plain

Stashbot added a comment.Jun 10 2026, 10:37 AM

Mentioned in SAL (#wikimedia-operations) [2026-06-10T10:37:27Z] <moritzm> failover Ganeti master in eqsin to ganeti5007 T428229

Stashbot added a comment.Jun 10 2026, 10:51 AM

Mentioned in SAL (#wikimedia-operations) [2026-06-10T10:51:31Z] <moritzm> remove ganeti5004 from eqsin cluster for reimage T428229

ops-monitoring-bot added a comment.Jun 10 2026, 11:34 AM

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host ganeti5004.eqsin.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jun 10 2026, 12:46 PM

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host ganeti5004.eqsin.wmnet with OS bookworm completed:

  • ganeti5004 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202606101224_jmm_1713265_ganeti5004.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Jun 10 2026, 1:54 PM

Change #1300152 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/software/homer/deploy@master] wmf-plugin: temp excpetion for new vlan names in eqsin

https://gerrit.wikimedia.org/r/1300152

gerritbot added a comment.Jun 10 2026, 2:26 PM

Change #1300152 merged by Cathal Mooney:

[operations/software/homer/deploy@master] wmf-plugin: temp excpetion for new vlan names in eqsin

https://gerrit.wikimedia.org/r/1300152

cmooney mentioned this in rOSHPe7b4a6c24a6b: wmf-plugin: temp excpetion for new vlan names in eqsin.Jun 10 2026, 2:27 PM
gerritbot added a comment.Jun 10 2026, 3:15 PM

Change #1299579 merged by BCornwall:

[operations/puppet@production] common: Update cp5018's IP address

https://gerrit.wikimedia.org/r/1299579

gerritbot added a comment.Jun 10 2026, 3:24 PM

Change #1300180 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/puppet@production] ganeti5004: set up custom bgp neighbors for private1-604-eqsin vlan

https://gerrit.wikimedia.org/r/1300180

gerritbot added a comment.Jun 10 2026, 4:12 PM

Change #1300180 merged by Cathal Mooney:

[operations/puppet@production] ganeti5004: set up custom bgp neighbors for private1-604-eqsin vlan

https://gerrit.wikimedia.org/r/1300180

Maintenance_bot removed a project: Patch-For-Review.Jun 10 2026, 4:31 PM
BCornwall updated the task description. (Show Details)Jun 10 2026, 6:17 PM
gerritbot added a comment.Jun 10 2026, 6:25 PM

Change #1300227 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] common: Update cp5020's IP address

https://gerrit.wikimedia.org/r/1300227

gerritbot added a project: Patch-For-Review.Jun 10 2026, 6:25 PM
gerritbot added a comment.Jun 10 2026, 6:56 PM

Change #1300227 merged by BCornwall:

[operations/puppet@production] common: Update cp5020's IP address

https://gerrit.wikimedia.org/r/1300227

Maintenance_bot removed a project: Patch-For-Review.Jun 10 2026, 7:31 PM
gerritbot added a comment.Jun 10 2026, 8:50 PM

Change #1300246 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] common: Update cp5024's IP address

https://gerrit.wikimedia.org/r/1300246

gerritbot added a project: Patch-For-Review.Jun 10 2026, 8:51 PM
gerritbot added a comment.Jun 10 2026, 8:53 PM

Change #1300246 merged by BCornwall:

[operations/puppet@production] common: Update cp5024's IP address

https://gerrit.wikimedia.org/r/1300246

Maintenance_bot removed a project: Patch-For-Review.Jun 10 2026, 9:31 PM
gerritbot added a comment.Jun 11 2026, 8:43 AM

Change #1300702 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] ganeti5006: set up custom bgp neighbors for private1-604-eqsin vlan

https://gerrit.wikimedia.org/r/1300702

gerritbot added a project: Patch-For-Review.Jun 11 2026, 8:43 AM
ops-monitoring-bot added a comment.Jun 11 2026, 11:11 AM

Draining ganeti5006.eqsin.wmnet of running VMs

MoritzMuehlenhoff updated the task description. (Show Details)Jun 11 2026, 11:12 AM
ops-monitoring-bot added a comment.Jun 11 2026, 11:12 AM

Draining ganeti5006.eqsin.wmnet of running VMs

Stashbot added a comment.Jun 11 2026, 12:21 PM

Mentioned in SAL (#wikimedia-operations) [2026-06-11T12:21:28Z] <moritzm> remove ganeti5006 from eqsin cluster for reimage T428229

ops-monitoring-bot added a comment.Jun 11 2026, 12:26 PM

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host ganeti5006.eqsin.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jun 11 2026, 1:46 PM

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host ganeti5006.eqsin.wmnet with OS bookworm completed:

  • ganeti5006 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202606111322_jmm_2033402_ganeti5006.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Jun 11 2026, 2:03 PM

Change #1300702 merged by Muehlenhoff:

[operations/puppet@production] ganeti5006: set up custom bgp neighbors for private1-604-eqsin vlan

https://gerrit.wikimedia.org/r/1300702

Maintenance_bot removed a project: Patch-For-Review.Jun 11 2026, 2:31 PM
MoritzMuehlenhoff updated the task description. (Show Details)Jun 11 2026, 2:38 PM
BCornwall updated the task description. (Show Details)Jun 11 2026, 5:34 PM
ops-monitoring-bot added a comment.Jun 12 2026, 12:02 PM

VM prometheus5003.eqsin.wmnet switching disk type to drbd

gerritbot added a comment.Jun 15 2026, 5:13 PM

Change #1302209 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] common: Update cp5026's IP address

https://gerrit.wikimedia.org/r/1302209

gerritbot added a project: Patch-For-Review.Jun 15 2026, 5:13 PM
gerritbot added a comment.Jun 15 2026, 5:42 PM

Change #1302209 merged by BCornwall:

[operations/puppet@production] common: Update cp5026's IP address

https://gerrit.wikimedia.org/r/1302209

Maintenance_bot removed a project: Patch-For-Review.Jun 15 2026, 6:31 PM
gerritbot added a comment.Jun 15 2026, 7:27 PM

Change #1302249 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] common: Update cp5028's IP address

https://gerrit.wikimedia.org/r/1302249

gerritbot added a project: Patch-For-Review.Jun 15 2026, 7:27 PM
gerritbot added a comment.Jun 15 2026, 7:33 PM

Change #1302249 merged by BCornwall:

[operations/puppet@production] common: Update cp5028's IP address

https://gerrit.wikimedia.org/r/1302249

Maintenance_bot removed a project: Patch-For-Review.Jun 15 2026, 8:32 PM
BCornwall updated the task description. (Show Details)Jun 16 2026, 5:42 PM
gerritbot added a comment.Jun 16 2026, 5:52 PM

Change #1302926 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] common: Update dns5004's IP address

https://gerrit.wikimedia.org/r/1302926

gerritbot added a project: Patch-For-Review.Jun 16 2026, 5:52 PM
gerritbot added a comment.Jun 16 2026, 5:55 PM

Change #1302926 merged by BCornwall:

[operations/puppet@production] common: Update dns5004's IP address

https://gerrit.wikimedia.org/r/1302926

Maintenance_bot removed a project: Patch-For-Review.Jun 16 2026, 6:31 PM
BCornwall updated the task description. (Show Details)Jun 16 2026, 7:51 PM
gerritbot added a comment.Jun 17 2026, 4:20 PM

Change #1303484 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] lvs5005: Set lowest bgp priority during reimage

https://gerrit.wikimedia.org/r/1303484

gerritbot added a project: Patch-For-Review.Jun 17 2026, 4:20 PM
gerritbot added a comment.Jun 17 2026, 4:24 PM

Change #1303484 merged by BCornwall:

[operations/puppet@production] lvs5005: Set lowest bgp priority during reimage

https://gerrit.wikimedia.org/r/1303484

Maintenance_bot removed a project: Patch-For-Review.Jun 17 2026, 4:31 PM
gerritbot added a comment.Jun 17 2026, 4:33 PM

Change #1303485 merged by BCornwall:

[operations/puppet@production] common: Update lvs5005's IP address

https://gerrit.wikimedia.org/r/1303485

Stashbot added a comment.Jun 17 2026, 7:30 PM

Mentioned in SAL (#wikimedia-operations) [2026-06-17T19:30:54Z] <brett@cumin2002> START - Cookbook sre.loadbalancer.admin config_reloading P{lvs5005*} and A:liberica (T428229)

Stashbot added a comment.Jun 17 2026, 7:31 PM

Mentioned in SAL (#wikimedia-operations) [2026-06-17T19:31:18Z] <brett@cumin2002> END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs5005*} and A:liberica (T428229)

BCornwall closed this task as Resolved.Jun 17 2026, 7:31 PM
BCornwall updated the task description. (Show Details)
Papaul added a comment.Jun 18 2026, 12:52 AM

@BCornwall @MoritzMuehlenhoff thanks to all of you getting this done

ops-monitoring-bot added a comment.Jun 18 2026, 12:56 PM

VM prometheus5003.eqsin.wmnet switching disk type to drbd

ops-monitoring-bot added a comment.Jun 18 2026, 1:33 PM

VM prometheus5003.eqsin.wmnet switching disk type to drbd

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits