Page MenuHomePhabricator
Create Task
Maniphest T319184

Move WMCS servers to 1 single NIC
Closed, ResolvedPublic
Actions

Assigned To
cmooney
Authored By
• aborrero
Oct 3 2022, 11:53 AM
Tags
Referenced Files
F35565980: image.png
Oct 13 2022, 12:17 PM
F35565783: Screenshot from 2022-10-13 12-22-21.png
Oct 13 2022, 10:24 AM
F35565728: Screenshot from 2022-10-13 11-09-13.png
Oct 13 2022, 10:24 AM
Subscribers
• aborrero
Aklapper
Andrew
ayounsi
BTullis
cmooney
dcaro
View All 9 Subscribers

Description

This ticket tracks work to move WMCS servers to a 1 NIC setup.

codfw

eqiad

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
openstack: nova-compute: remove support for legacy NIC setupoperations/puppetproduction+7 -62
hieradata: Move cloudvirt1042 to OVS and single NIC setupoperations/puppetproduction+1 -5
cloudvirt1041: move to modern single NIC setupoperations/puppetproduction+1 -5
cloudvirt1042: move to modern single NIC setupoperations/puppetproduction+1 -5
cloudvirt1040: move to modern single NIC setupoperations/puppetproduction+1 -5
cloudvirt1039: move to modern single NIC setupoperations/puppetproduction+1 -5
cloudvirt1037: move to modern single NIC setupoperations/puppetproduction+1 -5
cloudvirt1037: move to modern single NIC setupoperations/puppetproduction+1 -5
cloudvirt1036: move to modern single NIC setupoperations/puppetproduction+1 -5
cloudvirt1035: move to modern single NIC setupoperations/puppetproduction+1 -5
cloudvirt1034: move to modern single NIC setupoperations/puppetproduction+1 -5
cloudvirt1033: move to single NIC setupoperations/puppetproduction+1 -5
cloudvirt1031: move to modern NIC setupoperations/puppetproduction+0 -5
ceph: osd: introduce support for single NIC setupoperations/puppetproduction+192 -33
cloudvirt1043: move to modern NIC setupoperations/puppetproduction+0 -5
cloudvirt1044: move to modern NIC setupoperations/puppetproduction+0 -5
cloudvirt1045: move to modern NIC setupoperations/puppetproduction+0 -5
cloudvirt1046: move to modern NIC setupoperations/puppetproduction+0 -5
cloudvirt1047: move to modern NIC setupoperations/puppetproduction+0 -5
cloudvirt1048: move to modern NIC setupoperations/puppetproduction+0 -5
cloudvirt1049: move to modern NIC setupoperations/puppetproduction+0 -5
cloudvirt1050: move to modern NIC setupoperations/puppetproduction+0 -5
cloudvirt1051: move to modern NIC setupoperations/puppetproduction+0 -5
cloudvirt1052: move to modern NIC setupoperations/puppetproduction+0 -5
cloudvirt1053: move to modern NIC setupoperations/puppetproduction+0 -5
cloudvirt: eqiad1: prepare hiera configuration for modern NIC setupoperations/puppetproduction+5 -6
cloudvirts: introduce modern NIC setup and use it by defaultoperations/puppetproduction+139 -20
cloudgw: cleanup unused code to support multiple NICsoperations/puppetproduction+4 -65
cloudgw1002: move to the single-NIC setupoperations/puppetproduction+7 -22
cloudgw1001: prepare for reimage into the new vlan NIC name with a single NICoperations/puppetproduction+4 -6
cloudgw: introduce more robust vlan interface namingoperations/puppetproduction+58 -14
cloudvirt2003-dev: move to a single NIC setupoperations/puppetproduction+3 -3
cloudvirt2002-dev: move to a single NIC setupoperations/puppetproduction+3 -3
cloudgw2002-dev: move to a single NIC setupoperations/puppetproduction+1 -1
cloudgw: codfw1dev: don't hardcode interface namesoperations/puppetproduction+3 -2
cloudgw: don't configure anything on base dataplace interfaceoperations/puppetproduction+0 -7
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
ops-monitoring-bot added a comment.Feb 21 2024, 12:52 PM

Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1002 for host cloudvirt1033.eqiad.wmnet with OS bookworm completed:

  • cloudvirt1033 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202402211224_aborrero_1262466_cloudvirt1033.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Feb 21 2024, 1:19 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-02-21T13:19:54Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (T319184)

Stashbot added a comment.Feb 21 2024, 1:20 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-02-21T13:20:05Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) (T319184)

• aborrero updated the task description. (Show Details)Feb 21 2024, 1:36 PM
• aborrero closed subtask T357631: openstack: nova refuses to admit a compute node after a reimage as Resolved.
Stashbot added a comment.Feb 21 2024, 1:43 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-02-21T13:43:44Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.set_maintenance (T319184)

Stashbot added a comment.Feb 21 2024, 1:44 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-02-21T13:44:15Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0) (T319184)

gerritbot added a comment.Feb 21 2024, 1:45 PM

Change 1005513 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloudvirt1033: move to single NIC setup

https://gerrit.wikimedia.org/r/1005513

gerritbot added a comment.Feb 21 2024, 1:49 PM

Change 1005513 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloudvirt1033: move to single NIC setup

https://gerrit.wikimedia.org/r/1005513

ops-monitoring-bot added a comment.Feb 21 2024, 1:50 PM

Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin1002 for host cloudvirt1033.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Feb 21 2024, 2:40 PM

Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1002 for host cloudvirt1033.eqiad.wmnet with OS bookworm completed:

  • cloudvirt1033 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202402211410_aborrero_1277933_cloudvirt1033.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Feb 22 2024, 12:32 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-02-22T12:32:37Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1034.eqiad.wmnet' (T319184)

gerritbot added a comment.Feb 22 2024, 12:38 PM

Change 1005750 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloudvirt1034: move to modern single NIC setup

https://gerrit.wikimedia.org/r/1005750

Stashbot added a comment.Feb 22 2024, 12:53 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-02-22T12:53:12Z] <aborrero@cloudcumin1001> END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1034.eqiad.wmnet' (T319184)

Stashbot added a comment.Feb 22 2024, 12:57 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-02-22T12:57:57Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1034.eqiad.wmnet' (T319184)

Stashbot added a comment.Feb 22 2024, 12:58 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-02-22T12:58:41Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1034.eqiad.wmnet' (T319184)

ops-monitoring-bot added a comment.Feb 22 2024, 1:01 PM

Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin1002 for host cloudvirt1034.eqiad.wmnet with OS bookworm

gerritbot added a comment.Feb 22 2024, 1:02 PM

Change 1005750 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloudvirt1034: move to modern single NIC setup

https://gerrit.wikimedia.org/r/1005750

• aborrero updated the task description. (Show Details)Feb 22 2024, 1:42 PM
ops-monitoring-bot added a comment.Feb 22 2024, 1:46 PM

Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1002 for host cloudvirt1034.eqiad.wmnet with OS bookworm completed:

  • cloudvirt1034 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202402221320_aborrero_1465831_cloudvirt1034.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
• aborrero moved this task from Doing to Next on the User-aborrero board.Mar 5 2024, 3:31 PM
• aborrero moved this task from Next to Doing on the User-aborrero board.Apr 2 2024, 11:28 AM
gerritbot added a comment.Apr 2 2024, 11:30 AM

Change #1016318 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloudvirt1035: move to modern single NIC setup

https://gerrit.wikimedia.org/r/1016318

Stashbot added a comment.Apr 2 2024, 11:32 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-04-02T11:32:41Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1035.eqiad.wmnet' (T319184)

Stashbot added a comment.Apr 2 2024, 11:45 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-04-02T11:45:34Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1035.eqiad.wmnet' (T319184)

ops-monitoring-bot added a comment.Apr 2 2024, 11:49 AM

Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin1002 for host cloudvirt1035.eqiad.wmnet with OS bookworm

gerritbot added a comment.Apr 2 2024, 11:52 AM

Change #1016318 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloudvirt1035: move to modern single NIC setup

https://gerrit.wikimedia.org/r/1016318

• aborrero updated the task description. (Show Details)Apr 2 2024, 11:59 AM
ops-monitoring-bot added a comment.Apr 2 2024, 12:39 PM

Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1002 for host cloudvirt1035.eqiad.wmnet with OS bookworm completed:

  • cloudvirt1035 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202404021209_aborrero_254247_cloudvirt1035.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Apr 2 2024, 2:40 PM

Change #1016363 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloudvirt1036: move to modern single NIC setup

https://gerrit.wikimedia.org/r/1016363

Stashbot added a comment.Apr 2 2024, 2:43 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-04-02T14:43:30Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1036.eqiad.wmnet' (T319184)

Stashbot added a comment.Apr 2 2024, 3:00 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-04-02T15:00:13Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1036.eqiad.wmnet' (T319184)

ops-monitoring-bot added a comment.Apr 2 2024, 3:31 PM

Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin1002 for host cloudvirt1036.eqiad.wmnet with OS bookworm

gerritbot added a comment.Apr 2 2024, 3:50 PM

Change #1016363 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloudvirt1036: move to modern single NIC setup

https://gerrit.wikimedia.org/r/1016363

• aborrero updated the task description. (Show Details)Apr 2 2024, 3:51 PM
ops-monitoring-bot added a comment.Apr 2 2024, 4:22 PM

Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1002 for host cloudvirt1036.eqiad.wmnet with OS bookworm completed:

  • cloudvirt1036 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202404021551_aborrero_294759_cloudvirt1036.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Apr 3 2024, 9:09 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-04-03T09:09:41Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1037.eqiad.wmnet' (T319184)

Stashbot added a comment.Apr 3 2024, 9:26 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-04-03T09:26:28Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1037.eqiad.wmnet' (T319184)

ops-monitoring-bot added a comment.Apr 3 2024, 9:27 AM

Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin1002 for host cloudvirt1037.eqiad.wmnet with OS bookworm

gerritbot added a comment.Apr 3 2024, 9:28 AM

Change #1016720 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloudvirt1037: move to modern single NIC setup

https://gerrit.wikimedia.org/r/1016720

gerritbot added a comment.Apr 3 2024, 9:32 AM

Change #1016720 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloudvirt1037: move to modern single NIC setup

https://gerrit.wikimedia.org/r/1016720

ops-monitoring-bot added a comment.Apr 3 2024, 10:17 AM

Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1002 for host cloudvirt1037.eqiad.wmnet with OS bookworm completed:

  • cloudvirt1037 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202404030948_aborrero_441921_cloudvirt1037.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
• aborrero updated the task description. (Show Details)Apr 3 2024, 10:25 AM
Stashbot added a comment.Apr 3 2024, 10:25 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-04-03T10:25:36Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1038.eqiad.wmnet' (T319184)

Stashbot added a comment.Apr 3 2024, 10:37 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-04-03T10:37:06Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1038.eqiad.wmnet' (T319184)

ops-monitoring-bot added a comment.Apr 3 2024, 10:47 AM

Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin1002 for host cloudvirt1038.eqiad.wmnet with OS bookworm

gerritbot added a comment.Apr 3 2024, 10:47 AM

Change #1016743 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloudvirt1037: move to modern single NIC setup

https://gerrit.wikimedia.org/r/1016743

gerritbot added a comment.Apr 3 2024, 10:58 AM

Change #1016743 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloudvirt1037: move to modern single NIC setup

https://gerrit.wikimedia.org/r/1016743

ops-monitoring-bot added a comment.Apr 3 2024, 11:33 AM

Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1002 for host cloudvirt1038.eqiad.wmnet with OS bookworm completed:

  • cloudvirt1038 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202404031107_aborrero_454180_cloudvirt1038.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
• aborrero updated the task description. (Show Details)Apr 3 2024, 11:33 AM
Stashbot added a comment.Apr 3 2024, 11:34 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-04-03T11:34:24Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1039.eqiad.wmnet' (T319184)

Stashbot added a comment.Apr 3 2024, 11:48 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-04-03T11:48:20Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1039.eqiad.wmnet' (T319184)

ops-monitoring-bot added a comment.Apr 3 2024, 11:50 AM

Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin1002 for host cloudvirt1039.eqiad.wmnet with OS bookworm

gerritbot added a comment.Apr 3 2024, 11:53 AM

Change #1016751 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloudvirt1039: move to modern single NIC setup

https://gerrit.wikimedia.org/r/1016751

gerritbot added a comment.Apr 3 2024, 11:54 AM

Change #1016751 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloudvirt1039: move to modern single NIC setup

https://gerrit.wikimedia.org/r/1016751

• aborrero updated the task description. (Show Details)Apr 3 2024, 11:55 AM
ops-monitoring-bot added a comment.Apr 3 2024, 12:34 PM

Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1002 for host cloudvirt1039.eqiad.wmnet with OS bookworm completed:

  • cloudvirt1039 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202404031211_aborrero_464637_cloudvirt1039.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Apr 3 2024, 2:09 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-04-03T14:09:19Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1040.eqiad.wmnet' (T319184)

Stashbot added a comment.Apr 3 2024, 2:16 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-04-03T14:16:11Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1040.eqiad.wmnet' (T319184)

ops-monitoring-bot added a comment.Apr 3 2024, 2:16 PM

Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin1002 for host cloudvirt1040.eqiad.wmnet with OS bookworm

gerritbot added a comment.Apr 3 2024, 2:17 PM

Change #1016795 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloudvirt1040: move to modern single NIC setup

https://gerrit.wikimedia.org/r/1016795

gerritbot added a comment.Apr 3 2024, 2:21 PM

Change #1016795 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloudvirt1040: move to modern single NIC setup

https://gerrit.wikimedia.org/r/1016795

• aborrero updated the task description. (Show Details)Apr 3 2024, 2:24 PM
ops-monitoring-bot added a comment.Apr 3 2024, 3:01 PM

Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1002 for host cloudvirt1040.eqiad.wmnet with OS bookworm completed:

  • cloudvirt1040 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202404031434_aborrero_485212_cloudvirt1040.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
• aborrero moved this task from Doing to Next on the User-aborrero board.Apr 4 2024, 11:45 AM
Stashbot added a comment.May 15 2024, 8:57 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-05-15T08:57:20Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1041.eqiad.wmnet' (T319184)

Stashbot added a comment.May 15 2024, 9:08 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-05-15T09:08:49Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1041.eqiad.wmnet' (T319184)

Maintenance_bot removed a project: Patch-For-Review.May 15 2024, 9:31 AM
ops-monitoring-bot added a comment.May 15 2024, 10:52 AM

Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin1002 for host cloudvirt1041.eqiad.wmnet with OS bookworm

gerritbot added a comment.May 15 2024, 10:54 AM

Change #1031836 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloudvirt1041: move to modern single NIC setup

https://gerrit.wikimedia.org/r/1031836

gerritbot added a project: Patch-For-Review.May 15 2024, 10:54 AM

Change #1031839 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloudvirt1042: move to modern single NIC setup

https://gerrit.wikimedia.org/r/1031839

ops-monitoring-bot added a comment.May 15 2024, 11:05 AM

Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1002 for host cloudvirt1041.eqiad.wmnet with OS bookworm executed with errors:

  • cloudvirt1041 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details,You can also try typing "install-console" cloudvirt1041.eqiad.wmnet to get a root shellbut depending on the failure this may not work.
ops-monitoring-bot added a comment.May 15 2024, 11:10 AM

Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin1002 for host cloudvirt1041.eqiad.wmnet with OS bookworm

gerritbot added a comment.May 15 2024, 11:28 AM

Change #1031836 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloudvirt1041: move to modern single NIC setup

https://gerrit.wikimedia.org/r/1031836

ops-monitoring-bot added a comment.May 15 2024, 11:52 AM

Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1002 for host cloudvirt1041.eqiad.wmnet with OS bookworm executed with errors:

  • cloudvirt1041 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details,You can also try typing "install-console" cloudvirt1041.eqiad.wmnet to get a root shellbut depending on the failure this may not work.
• aborrero changed the task status from Open to Stalled.May 22 2024, 1:42 PM
• aborrero moved this task from Next to Blocked/waiting on the User-aborrero board.
• aborrero moved this task from Soon! to Blocked on the cloud-services-team board.

blocking until T364984: cloudvirt1041: can't boot after reimage is fixed, so we don't risk having another cloudvirt offline.

Andrew closed subtask T364984: cloudvirt1041: can't boot after reimage as Resolved.May 30 2024, 4:40 PM
cmooney awarded a token.May 30 2024, 4:41 PM
Aklapper changed the task status from Stalled to Open.Jun 1 2024, 8:05 AM

Subtask resolved thus reopening

• aborrero moved this task from Blocked/waiting to Next on the User-aborrero board.Jun 3 2024, 9:21 AM
• aborrero updated the task description. (Show Details)Jun 10 2024, 2:49 PM
gerritbot added a comment.Jun 19 2024, 12:01 PM

Change #1047486 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] hieradata: Move cloudvirt1042 to OVS and single NIC setup

https://gerrit.wikimedia.org/r/1047486

gerritbot added a comment.Jun 19 2024, 12:02 PM

Change #1047486 merged by Majavah:

[operations/puppet@production] hieradata: Move cloudvirt1042 to OVS and single NIC setup

https://gerrit.wikimedia.org/r/1047486

• aborrero updated the task description. (Show Details)Jun 24 2024, 2:58 PM
gerritbot added a comment.Jun 24 2024, 3:05 PM

Change #1049200 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] openstack: nova-compute: remove support for legacy NIC setup

https://gerrit.wikimedia.org/r/1049200

gerritbot added a comment.Jul 2 2024, 2:26 PM

Change #1049200 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] openstack: nova-compute: remove support for legacy NIC setup

https://gerrit.wikimedia.org/r/1049200

• aborrero changed the task status from Open to Stalled.Jul 2 2024, 2:27 PM
• aborrero moved this task from Next to Radar/observer on the User-aborrero board.

marking as stalled, because the work on ceph nodes wont be progressing for a while.

cmooney closed this task as Resolved.Jul 10 2025, 11:30 AM
cmooney claimed this task.
cmooney mentioned this in T399180: Cloudcephosd: migrate to single network uplink.

I am going to close this one (please ping me if that is hasty!) as I've opened a separate task specifically to look at migration for the cloudcephosd* hosts.

T399180: Cloudcephosd: migrate to single network uplink

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits