Page MenuHomePhabricator
Create Task
Maniphest T319184

Move WMCS servers to 1 single NIC
Open, MediumPublic
Actions

Assigned To
None
Authored By
aborrero
Oct 3 2022, 11:53 AM
Tags
Referenced Files
F35565980: image.png
Oct 13 2022, 12:17 PM
F35565783: Screenshot from 2022-10-13 12-22-21.png
Oct 13 2022, 10:24 AM
F35565728: Screenshot from 2022-10-13 11-09-13.png
Oct 13 2022, 10:24 AM
Subscribers
aborrero
Aklapper
Andrew
ayounsi
BTullis
cmooney
dcaro
View All 9 Subscribers

Description

This ticket tracks work to move WMCS servers to a 1 NIC setup.

codfw

eqiad

Details

SubjectRepoBranchLines +/-
cloudvirt1040: move to modern single NIC setupoperations/puppetproduction+1 -5
cloudvirt1039: move to modern single NIC setupoperations/puppetproduction+1 -5
cloudvirt1037: move to modern single NIC setupoperations/puppetproduction+1 -5
cloudvirt1037: move to modern single NIC setupoperations/puppetproduction+1 -5
cloudvirt1036: move to modern single NIC setupoperations/puppetproduction+1 -5
cloudvirt1035: move to modern single NIC setupoperations/puppetproduction+1 -5
cloudvirt1034: move to modern single NIC setupoperations/puppetproduction+1 -5
cloudvirt1033: move to single NIC setupoperations/puppetproduction+1 -5
cloudvirt1031: move to modern NIC setupoperations/puppetproduction+0 -5
ceph: osd: introduce support for single NIC setupoperations/puppetproduction+192 -33
cloudvirt1043: move to modern NIC setupoperations/puppetproduction+0 -5
cloudvirt1044: move to modern NIC setupoperations/puppetproduction+0 -5
cloudvirt1045: move to modern NIC setupoperations/puppetproduction+0 -5
cloudvirt1046: move to modern NIC setupoperations/puppetproduction+0 -5
cloudvirt1047: move to modern NIC setupoperations/puppetproduction+0 -5
cloudvirt1048: move to modern NIC setupoperations/puppetproduction+0 -5
cloudvirt1049: move to modern NIC setupoperations/puppetproduction+0 -5
cloudvirt1050: move to modern NIC setupoperations/puppetproduction+0 -5
cloudvirt1051: move to modern NIC setupoperations/puppetproduction+0 -5
cloudvirt1052: move to modern NIC setupoperations/puppetproduction+0 -5
cloudvirt1053: move to modern NIC setupoperations/puppetproduction+0 -5
cloudvirt: eqiad1: prepare hiera configuration for modern NIC setupoperations/puppetproduction+5 -6
cloudvirts: introduce modern NIC setup and use it by defaultoperations/puppetproduction+139 -20
cloudgw: cleanup unused code to support multiple NICsoperations/puppetproduction+4 -65
cloudgw1002: move to the single-NIC setupoperations/puppetproduction+7 -22
cloudgw1001: prepare for reimage into the new vlan NIC name with a single NICoperations/puppetproduction+4 -6
cloudgw: introduce more robust vlan interface namingoperations/puppetproduction+58 -14
cloudvirt2003-dev: move to a single NIC setupoperations/puppetproduction+3 -3
cloudvirt2002-dev: move to a single NIC setupoperations/puppetproduction+3 -3
cloudgw2002-dev: move to a single NIC setupoperations/puppetproduction+1 -1
cloudgw: codfw1dev: don't hardcode interface namesoperations/puppetproduction+3 -2
cloudgw: don't configure anything on base dataplace interfaceoperations/puppetproduction+0 -7
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
aborrero added a comment.Oct 9 2023, 9:26 AM

This is the patch to enable the single NIC setup on ceph nodes:

https://gerrit.wikimedia.org/r/c/operations/puppet/+/856675/

Is marked as abandoned, but should work just fine.

cmooney mentioned this in T349756: Audit of WMCS Servers Using Single & Dual Switchports.Dec 15 2023, 5:11 PM
aborrero changed the task status from Stalled to Open.Feb 14 2024, 3:30 PM
aborrero added a project: User-aborrero.

In a 2024-02-14 network sync meeting we decided to continue moving older cloudvirts into the new single NIC setup. I plan to work on this soon.

aborrero moved this task from Backlog to Next on the User-aborrero board.Feb 14 2024, 3:30 PM
aborrero updated the task description. (Show Details)
gerritbot added a comment.Feb 15 2024, 11:50 AM

Change 1003616 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloudvirt1031: move to modern NIC setup

https://gerrit.wikimedia.org/r/1003616

gerritbot added a project: Patch-For-Review.Feb 15 2024, 11:50 AM
Stashbot added a comment.Feb 15 2024, 12:05 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-02-15T12:05:25Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1031.eqiad.wmnet' (T319184)

gerritbot added a comment.Feb 15 2024, 12:10 PM

Change 1003616 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloudvirt1031: move to modern NIC setup

https://gerrit.wikimedia.org/r/1003616

ops-monitoring-bot added a comment.Feb 15 2024, 12:11 PM

Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin1002 for host cloudvirt1031.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Feb 15 2024, 1:00 PM

Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1002 for host cloudvirt1031.eqiad.wmnet with OS bookworm completed:

  • cloudvirt1031 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202402151232_aborrero_124213_cloudvirt1031.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
aborrero updated the task description. (Show Details)Feb 15 2024, 3:53 PM
aborrero moved this task from Next to Doing on the User-aborrero board.Feb 16 2024, 12:06 PM
Stashbot added a comment.Feb 19 2024, 9:49 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-02-19T09:49:01Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.pre-reimage prepare cloudvirt1032.eqiad.wmnet for reimage (drain, remove nova agent, etc) (T319184)

Stashbot added a comment.Feb 19 2024, 9:49 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-02-19T09:49:07Z] <aborrero@cloudcumin1001> END (FAIL) - Cookbook wmcs.openstack.cloudvirt.pre-reimage (exit_code=99) prepare cloudvirt1032.eqiad.wmnet for reimage (drain, remove nova agent, etc) (T319184)

Stashbot added a comment.Feb 19 2024, 9:52 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-02-19T09:52:10Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.pre-reimage prepare cloudvirt1032.eqiad.wmnet for reimage (drain, remove nova agent, etc) (T319184)

Stashbot added a comment.Feb 19 2024, 10:09 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-02-19T10:09:12Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudvirt.pre-reimage (exit_code=0) prepare cloudvirt1032.eqiad.wmnet for reimage (drain, remove nova agent, etc) (T319184)

ops-monitoring-bot added a comment.Feb 19 2024, 11:11 AM

Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin1002 for host cloudvirt1032.eqiad.wmnet with OS bookworm

Stashbot added a comment.Feb 19 2024, 12:00 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-02-19T12:00:50Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.post-reimage preparing cloudvirt cloudvirt1032.eqiad.wmnet for duty (nova discovery, canary VM) Pending aggregates though. (T319184)

Stashbot added a comment.Feb 19 2024, 12:00 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-02-19T12:00:56Z] <aborrero@cloudcumin1001> END (FAIL) - Cookbook wmcs.openstack.cloudvirt.post-reimage (exit_code=99) preparing cloudvirt cloudvirt1032.eqiad.wmnet for duty (nova discovery, canary VM) Pending aggregates though. (T319184)

Stashbot added a comment.Feb 19 2024, 12:02 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-02-19T12:02:28Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.post-reimage preparing cloudvirt cloudvirt1032.eqiad.wmnet for duty (nova discovery, canary VM) Pending aggregates though. (T319184)

Stashbot added a comment.Feb 19 2024, 12:02 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-02-19T12:02:32Z] <aborrero@cloudcumin1001> END (FAIL) - Cookbook wmcs.openstack.cloudvirt.post-reimage (exit_code=99) preparing cloudvirt cloudvirt1032.eqiad.wmnet for duty (nova discovery, canary VM) Pending aggregates though. (T319184)

ops-monitoring-bot added a comment.Feb 19 2024, 12:03 PM

Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1002 for host cloudvirt1032.eqiad.wmnet with OS bookworm completed:

  • cloudvirt1032 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202402191134_aborrero_830992_cloudvirt1032.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Feb 19 2024, 12:32 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-02-19T12:32:49Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.post-reimage preparing cloudvirt cloudvirt1032.eqiad.wmnet for duty (nova discovery, canary VM) Pending aggregates though. (T319184)

Stashbot added a comment.Feb 19 2024, 12:33 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-02-19T12:33:11Z] <aborrero@cloudcumin1001> END (FAIL) - Cookbook wmcs.openstack.cloudvirt.post-reimage (exit_code=99) preparing cloudvirt cloudvirt1032.eqiad.wmnet for duty (nova discovery, canary VM) Pending aggregates though. (T319184)

Stashbot added a comment.Feb 20 2024, 11:30 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-02-20T11:30:00Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.post-reimage preparing cloudvirt cloudvirt1032.eqiad.wmnet for duty (nova discovery, canary VM) Pending aggregates though. (T319184)

Stashbot added a comment.Feb 20 2024, 11:30 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-02-20T11:30:28Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudvirt.post-reimage (exit_code=0) preparing cloudvirt cloudvirt1032.eqiad.wmnet for duty (nova discovery, canary VM) Pending aggregates though. (T319184)

aborrero updated the task description. (Show Details)Feb 20 2024, 12:01 PM
Stashbot added a comment.Feb 21 2024, 11:44 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-02-21T11:44:22Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1033.eqiad.wmnet' (T319184)

Stashbot added a comment.Feb 21 2024, 12:03 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-02-21T12:03:14Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1033.eqiad.wmnet' (T319184)

ops-monitoring-bot added a comment.Feb 21 2024, 12:04 PM

Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin1002 for host cloudvirt1033.eqiad.wmnet with OS bookworm

Stashbot added a comment.Feb 21 2024, 12:50 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-02-21T12:50:30Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1033.eqiad.wmnet' (T319184)

Stashbot added a comment.Feb 21 2024, 12:50 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-02-21T12:50:56Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1033.eqiad.wmnet' (T319184)

ops-monitoring-bot added a comment.Feb 21 2024, 12:52 PM

Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1002 for host cloudvirt1033.eqiad.wmnet with OS bookworm completed:

  • cloudvirt1033 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202402211224_aborrero_1262466_cloudvirt1033.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Feb 21 2024, 1:19 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-02-21T13:19:54Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (T319184)

Stashbot added a comment.Feb 21 2024, 1:20 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-02-21T13:20:05Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) (T319184)

aborrero updated the task description. (Show Details)Feb 21 2024, 1:36 PM
aborrero closed subtask T357631: openstack: nova refuses to admit a compute node after a reimage as Resolved.
Stashbot added a comment.Feb 21 2024, 1:43 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-02-21T13:43:44Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.set_maintenance (T319184)

Stashbot added a comment.Feb 21 2024, 1:44 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-02-21T13:44:15Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0) (T319184)

gerritbot added a comment.Feb 21 2024, 1:45 PM

Change 1005513 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloudvirt1033: move to single NIC setup

https://gerrit.wikimedia.org/r/1005513

gerritbot added a comment.Feb 21 2024, 1:49 PM

Change 1005513 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloudvirt1033: move to single NIC setup

https://gerrit.wikimedia.org/r/1005513

ops-monitoring-bot added a comment.Feb 21 2024, 1:50 PM

Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin1002 for host cloudvirt1033.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Feb 21 2024, 2:40 PM

Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1002 for host cloudvirt1033.eqiad.wmnet with OS bookworm completed:

  • cloudvirt1033 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202402211410_aborrero_1277933_cloudvirt1033.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Feb 22 2024, 12:32 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-02-22T12:32:37Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1034.eqiad.wmnet' (T319184)

gerritbot added a comment.Feb 22 2024, 12:38 PM

Change 1005750 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloudvirt1034: move to modern single NIC setup

https://gerrit.wikimedia.org/r/1005750

Stashbot added a comment.Feb 22 2024, 12:53 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-02-22T12:53:12Z] <aborrero@cloudcumin1001> END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1034.eqiad.wmnet' (T319184)

Stashbot added a comment.Feb 22 2024, 12:57 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-02-22T12:57:57Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1034.eqiad.wmnet' (T319184)

Stashbot added a comment.Feb 22 2024, 12:58 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-02-22T12:58:41Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1034.eqiad.wmnet' (T319184)

ops-monitoring-bot added a comment.Feb 22 2024, 1:01 PM

Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin1002 for host cloudvirt1034.eqiad.wmnet with OS bookworm

gerritbot added a comment.Feb 22 2024, 1:02 PM

Change 1005750 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloudvirt1034: move to modern single NIC setup

https://gerrit.wikimedia.org/r/1005750

aborrero updated the task description. (Show Details)Feb 22 2024, 1:42 PM
ops-monitoring-bot added a comment.Feb 22 2024, 1:46 PM

Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1002 for host cloudvirt1034.eqiad.wmnet with OS bookworm completed:

  • cloudvirt1034 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202402221320_aborrero_1465831_cloudvirt1034.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
aborrero moved this task from Doing to Next on the User-aborrero board.Mar 5 2024, 3:31 PM
aborrero moved this task from Next to Doing on the User-aborrero board.Tue, Apr 2, 11:28 AM
gerritbot added a comment.Tue, Apr 2, 11:30 AM

Change #1016318 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloudvirt1035: move to modern single NIC setup

https://gerrit.wikimedia.org/r/1016318

Stashbot added a comment.Tue, Apr 2, 11:32 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-04-02T11:32:41Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1035.eqiad.wmnet' (T319184)

Stashbot added a comment.Tue, Apr 2, 11:45 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-04-02T11:45:34Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1035.eqiad.wmnet' (T319184)

ops-monitoring-bot added a comment.Tue, Apr 2, 11:49 AM

Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin1002 for host cloudvirt1035.eqiad.wmnet with OS bookworm

gerritbot added a comment.Tue, Apr 2, 11:52 AM

Change #1016318 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloudvirt1035: move to modern single NIC setup

https://gerrit.wikimedia.org/r/1016318

aborrero updated the task description. (Show Details)Tue, Apr 2, 11:59 AM
ops-monitoring-bot added a comment.Tue, Apr 2, 12:39 PM

Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1002 for host cloudvirt1035.eqiad.wmnet with OS bookworm completed:

  • cloudvirt1035 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202404021209_aborrero_254247_cloudvirt1035.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Tue, Apr 2, 2:40 PM

Change #1016363 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloudvirt1036: move to modern single NIC setup

https://gerrit.wikimedia.org/r/1016363

Stashbot added a comment.Tue, Apr 2, 2:43 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-04-02T14:43:30Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1036.eqiad.wmnet' (T319184)

Stashbot added a comment.Tue, Apr 2, 3:00 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-04-02T15:00:13Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1036.eqiad.wmnet' (T319184)

ops-monitoring-bot added a comment.Tue, Apr 2, 3:31 PM

Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin1002 for host cloudvirt1036.eqiad.wmnet with OS bookworm

gerritbot added a comment.Tue, Apr 2, 3:50 PM

Change #1016363 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloudvirt1036: move to modern single NIC setup

https://gerrit.wikimedia.org/r/1016363

aborrero updated the task description. (Show Details)Tue, Apr 2, 3:51 PM
ops-monitoring-bot added a comment.Tue, Apr 2, 4:22 PM

Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1002 for host cloudvirt1036.eqiad.wmnet with OS bookworm completed:

  • cloudvirt1036 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202404021551_aborrero_294759_cloudvirt1036.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Wed, Apr 3, 9:09 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-04-03T09:09:41Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1037.eqiad.wmnet' (T319184)

Stashbot added a comment.Wed, Apr 3, 9:26 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-04-03T09:26:28Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1037.eqiad.wmnet' (T319184)

ops-monitoring-bot added a comment.Wed, Apr 3, 9:27 AM

Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin1002 for host cloudvirt1037.eqiad.wmnet with OS bookworm

gerritbot added a comment.Wed, Apr 3, 9:28 AM

Change #1016720 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloudvirt1037: move to modern single NIC setup

https://gerrit.wikimedia.org/r/1016720

gerritbot added a comment.Wed, Apr 3, 9:32 AM

Change #1016720 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloudvirt1037: move to modern single NIC setup

https://gerrit.wikimedia.org/r/1016720

ops-monitoring-bot added a comment.Wed, Apr 3, 10:17 AM

Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1002 for host cloudvirt1037.eqiad.wmnet with OS bookworm completed:

  • cloudvirt1037 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202404030948_aborrero_441921_cloudvirt1037.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
aborrero updated the task description. (Show Details)Wed, Apr 3, 10:25 AM
Stashbot added a comment.Wed, Apr 3, 10:25 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-04-03T10:25:36Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1038.eqiad.wmnet' (T319184)

Stashbot added a comment.Wed, Apr 3, 10:37 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-04-03T10:37:06Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1038.eqiad.wmnet' (T319184)

ops-monitoring-bot added a comment.Wed, Apr 3, 10:47 AM

Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin1002 for host cloudvirt1038.eqiad.wmnet with OS bookworm

gerritbot added a comment.Wed, Apr 3, 10:47 AM

Change #1016743 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloudvirt1037: move to modern single NIC setup

https://gerrit.wikimedia.org/r/1016743

gerritbot added a comment.Wed, Apr 3, 10:58 AM

Change #1016743 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloudvirt1037: move to modern single NIC setup

https://gerrit.wikimedia.org/r/1016743

ops-monitoring-bot added a comment.Wed, Apr 3, 11:33 AM

Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1002 for host cloudvirt1038.eqiad.wmnet with OS bookworm completed:

  • cloudvirt1038 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202404031107_aborrero_454180_cloudvirt1038.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
aborrero updated the task description. (Show Details)Wed, Apr 3, 11:33 AM
Stashbot added a comment.Wed, Apr 3, 11:34 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-04-03T11:34:24Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1039.eqiad.wmnet' (T319184)

Stashbot added a comment.Wed, Apr 3, 11:48 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-04-03T11:48:20Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1039.eqiad.wmnet' (T319184)

ops-monitoring-bot added a comment.Wed, Apr 3, 11:50 AM

Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin1002 for host cloudvirt1039.eqiad.wmnet with OS bookworm

gerritbot added a comment.Wed, Apr 3, 11:53 AM

Change #1016751 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloudvirt1039: move to modern single NIC setup

https://gerrit.wikimedia.org/r/1016751

gerritbot added a comment.Wed, Apr 3, 11:54 AM

Change #1016751 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloudvirt1039: move to modern single NIC setup

https://gerrit.wikimedia.org/r/1016751

aborrero updated the task description. (Show Details)Wed, Apr 3, 11:55 AM
ops-monitoring-bot added a comment.Wed, Apr 3, 12:34 PM

Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1002 for host cloudvirt1039.eqiad.wmnet with OS bookworm completed:

  • cloudvirt1039 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202404031211_aborrero_464637_cloudvirt1039.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Wed, Apr 3, 2:09 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-04-03T14:09:19Z] <aborrero@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1040.eqiad.wmnet' (T319184)

Stashbot added a comment.Wed, Apr 3, 2:16 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-04-03T14:16:11Z] <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1040.eqiad.wmnet' (T319184)

ops-monitoring-bot added a comment.Wed, Apr 3, 2:16 PM

Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin1002 for host cloudvirt1040.eqiad.wmnet with OS bookworm

gerritbot added a comment.Wed, Apr 3, 2:17 PM

Change #1016795 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloudvirt1040: move to modern single NIC setup

https://gerrit.wikimedia.org/r/1016795

gerritbot added a comment.Wed, Apr 3, 2:21 PM

Change #1016795 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloudvirt1040: move to modern single NIC setup

https://gerrit.wikimedia.org/r/1016795

aborrero updated the task description. (Show Details)Wed, Apr 3, 2:24 PM
ops-monitoring-bot added a comment.Wed, Apr 3, 3:01 PM

Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1002 for host cloudvirt1040.eqiad.wmnet with OS bookworm completed:

  • cloudvirt1040 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202404031434_aborrero_485212_cloudvirt1040.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
aborrero moved this task from Doing to Next on the User-aborrero board.Thu, Apr 4, 11:45 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL