Page MenuHomePhabricator
Create Task
Maniphest T316284

Replace cloudnet100[34] with cloudnet100[56]
Closed, ResolvedPublic
Actions

Description

These new hosts are racked but not yet puppetized or in service.

Details

SubjectRepoBranchLines +/-
cloudnet1003: decom hostoperations/puppetproduction+3 -4
cloudnet: merge host hiera overrides back into the profileoperations/puppetproduction+17 -30
Make cloudnet100[56] into cloudnet nodesoperations/puppetproduction+0 -0
cloudnet1005/1006: prepare for single NIC setupoperations/puppetproduction+25 -0
cloudnet1005/1006: give them proper roleoperations/puppetproduction+1 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

Andrew created this task.Aug 25 2022, 7:21 PM
Restricted Application edited projects, added cloud-services-team (Kanban); removed Patch-For-Review, cloud-services-team (Hardware). · View Herald TranscriptAug 25 2022, 7:21 PM
Andrew reassigned this task from Andrew to aborrero.Sep 9 2022, 5:53 PM
• Cmjohnson moved this task from Backlog to Hardware Failure / Troubleshoot on the ops-eqiad board.Sep 14 2022, 4:12 PM
aborrero triaged this task as Low priority.Sep 27 2022, 12:56 PM
aborrero edited projects, added cloud-services-team (Hardware); removed SRE, ops-eqiad, DC-Ops, cloud-services-team (Kanban).
gerritbot added a comment.Sep 27 2022, 5:15 PM

Change 835657 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] Make cloudnet100[56] into cloudnet nodes

https://gerrit.wikimedia.org/r/835657

gerritbot added a project: Patch-For-Review.Sep 27 2022, 5:15 PM
aborrero added a subscriber: ayounsi.Sep 28 2022, 2:56 PM

We agreed with @cmooney and @ayounsi to make them single NIC before putting them into service.

aborrero closed subtask T318824: Make cloudnet200[56}-dev single NIC as Resolved.Sep 30 2022, 12:24 PM
gerritbot added a comment.Oct 3 2022, 10:07 AM

Change 837631 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloudnet1005/1006: prepare for single NIC setup

https://gerrit.wikimedia.org/r/837631

aborrero mentioned this in T319184: Move WMCS servers to 1 single NIC.Oct 3 2022, 11:57 AM
gerritbot added a comment.Oct 4 2022, 8:42 AM

Change 837631 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloudnet1005/1006: prepare for single NIC setup

https://gerrit.wikimedia.org/r/837631

gerritbot added a comment.Oct 4 2022, 10:48 AM

Change 838117 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloudnet1005/1006: give them proper role

https://gerrit.wikimedia.org/r/838117

gerritbot added a comment.Oct 4 2022, 10:52 AM

Change 838117 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloudnet1005/1006: give them proper role

https://gerrit.wikimedia.org/r/838117

ops-monitoring-bot added a comment.Oct 4 2022, 10:53 AM

Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin1001 for host cloudnet1005.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Oct 4 2022, 10:54 AM

Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin1001 for host cloudnet1006.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Oct 4 2022, 10:55 AM

Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1001 for host cloudnet1006.eqiad.wmnet with OS bullseye executed with errors:

  • cloudnet1006 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Oct 4 2022, 10:58 AM

Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin1001 for host cloudnet1006.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Oct 4 2022, 10:58 AM

Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1001 for host cloudnet1006.eqiad.wmnet with OS bullseye executed with errors:

  • cloudnet1006 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Oct 4 2022, 11:04 AM

Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin1001 for host cloudnet1006.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Oct 4 2022, 12:10 PM

Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1001 for host cloudnet1005.eqiad.wmnet with OS bullseye executed with errors:

  • cloudnet1005 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Oct 4 2022, 12:21 PM

Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1001 for host cloudnet1006.eqiad.wmnet with OS bullseye executed with errors:

  • cloudnet1006 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Oct 4 2022, 12:37 PM

Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin1001 for host cloudnet1005.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Oct 4 2022, 12:37 PM

Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin1001 for host cloudnet1006.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Oct 4 2022, 2:27 PM

Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1001 for host cloudnet1005.eqiad.wmnet with OS bullseye executed with errors:

  • cloudnet1005 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202210041237_aborrero_2721337_cloudnet1005.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Oct 4 2022, 2:29 PM

Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1001 for host cloudnet1006.eqiad.wmnet with OS bullseye executed with errors:

  • cloudnet1006 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202210041237_aborrero_2721275_cloudnet1006.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Oct 5 2022, 11:33 AM

Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin1001 for host cloudnet1005.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Oct 5 2022, 11:33 AM

Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin1001 for host cloudnet1006.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Oct 5 2022, 12:13 PM

Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1001 for host cloudnet1005.eqiad.wmnet with OS bullseye completed:

  • cloudnet1005 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202210051133_aborrero_2971986_cloudnet1005.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Oct 5 2022, 12:16 PM

Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1001 for host cloudnet1006.eqiad.wmnet with OS bullseye completed:

  • cloudnet1006 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202210051133_aborrero_2971853_cloudnet1006.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Failed to get Netbox script results, try manually: https://netbox.discovery.wmnet/api/extras/job-results/3824464/
ops-monitoring-bot added a comment.Oct 5 2022, 12:16 PM

Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1001 for host cloudnet1006.eqiad.wmnet with OS bullseye executed with errors:

  • cloudnet1006 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202210051133_aborrero_2971853_cloudnet1006.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Failed to get Netbox script results, try manually: https://netbox.discovery.wmnet/api/extras/job-results/3824464/
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Oct 5 2022, 12:30 PM

Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin1001 for host cloudnet1006.eqiad.wmnet with OS bullseye

gerritbot added a comment.Oct 5 2022, 12:43 PM

Change 838793 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloudnet1003: decom host

https://gerrit.wikimedia.org/r/838793

ops-monitoring-bot added a comment.Oct 5 2022, 2:15 PM

Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1001 for host cloudnet1006.eqiad.wmnet with OS bullseye completed:

  • cloudnet1006 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202210051229_aborrero_2984550_cloudnet1006.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Oct 5 2022, 2:28 PM

Mentioned in SAL (#wikimedia-cloud) [2022-10-05T14:28:15Z] <arturo> adding cloudinstances2b-gw router to l3 agents on cloudnet1005/1006 (T316284)

aborrero added a comment.Oct 5 2022, 2:32 PM

For the record:

aborrero@cloudcontrol1005:~ $ sudo wmcs-openstack network agent list | grep 1006
| 2c9a8a8d-2ff4-47a9-80d4-645ac4c5ec50 | Linux bridge agent | cloudnet1006       | None              | :-)   | UP    | neutron-linuxbridge-agent |
| 3f54b3c2-503f-4667-8263-859a259b3b21 | L3 agent           | cloudnet1006       | nova              | :-)   | UP    | neutron-l3-agent          |
| 97b30d69-fd14-4061-a7df-601186626a3c | Metadata agent     | cloudnet1006       | None              | :-)   | UP    | neutron-metadata-agent    |
| e4f71e5d-e182-487d-8c5f-eb15f1ff2bf6 | DHCP agent         | cloudnet1006       | nova              | :-)   | UP    | neutron-dhcp-agent        |
aborrero@cloudcontrol1005:~ 2s $ sudo wmcs-openstack network agent list | grep 1005
| 29547916-33cd-45d8-b33c-4947921ba728 | Linux bridge agent | cloudnet1005       | None              | :-)   | UP    | neutron-linuxbridge-agent |
| 40a4af74-77fb-4341-8589-ae3e8a5fce5a | DHCP agent         | cloudnet1005       | nova              | :-)   | UP    | neutron-dhcp-agent        |
| 6a88c860-29fb-4a85-8aea-6a8877c2e035 | L3 agent           | cloudnet1005       | nova              | :-)   | UP    | neutron-l3-agent          |
| 723d796e-e702-43c9-a0bc-f9645e3ad7d0 | Metadata agent     | cloudnet1005       | None              | :-)   | UP    | neutron-metadata-agent    |
aborrero@cloudcontrol1005:~ 2s $ sudo wmcs-openstack router list
+--------------------------------------+---------------------+--------+-------+---------+-------------+------+
| ID                                   | Name                | Status | State | Project | Distributed | HA   |
+--------------------------------------+---------------------+--------+-------+---------+-------------+------+
| d93771ba-2711-4f88-804a-8df6fd03978a | cloudinstances2b-gw | ACTIVE | UP    | admin   | False       | True |
+--------------------------------------+---------------------+--------+-------+---------+-------------+------+
aborrero@cloudcontrol1005:~ 2s $ sudo wmcs-openstack network agent add router
usage: openstack network agent add router [-h] [--l3] <agent-id> <router>
openstack network agent add router: error: the following arguments are required: <agent-id>, <router>
aborrero@cloudcontrol1005:~ $ sudo wmcs-openstack network agent add router --l3 3f54b3c2-503f-4667-8263-859a259b3b21 d93771ba-2711-4f88-804a-8df6fd03978a
aborrero@cloudcontrol1005:~ 3s $ sudo wmcs-openstack network agent add router --l3 6a88c860-29fb-4a85-8aea-6a8877c2e035 d93771ba-2711-4f88-804a-8df6fd03978a
ops-monitoring-bot added a comment.Oct 5 2022, 2:38 PM

Icinga downtime and Alertmanager silence (ID=811efa94-5012-4cee-9c58-75a3650086e8) set by aborrero@cumin1001 for 1:00:00 on 1 host(s) and their services with reason: decom

cloudnet1003.eqiad.wmnet
ops-monitoring-bot added a comment.Oct 5 2022, 2:39 PM

Icinga downtime and Alertmanager silence (ID=a7a84472-3569-4c03-a2d9-0da483f24b3e) set by aborrero@cumin1001 for 1:00:00 on 1 host(s) and their services with reason: decom

cloudnet1004.eqiad.wmnet
Papaul unsubscribed.Oct 5 2022, 2:43 PM
gerritbot added a comment.Oct 5 2022, 2:45 PM

Change 835657 abandoned by Andrew Bogott:

[operations/puppet@production] Make cloudnet100[56] into cloudnet nodes

Reason:

this was done elsewhere

https://gerrit.wikimedia.org/r/835657

ops-monitoring-bot added a comment.Oct 5 2022, 2:51 PM

Icinga downtime and Alertmanager silence (ID=83a14b71-79b5-4be2-ac1b-f073f85678b0) set by aborrero@cumin1001 for 15 days, 0:00:00 on 2 host(s) and their services with reason: migrating

cloudnet[1005-1006].eqiad.wmnet
aborrero added a comment.Oct 6 2022, 9:45 AM

@cmooney reported a problem with cloudnet bridges and vlan interfaces not being attached to them.

I confirmed this is true and spotted a potential cause:

aborrero@cloudnet1005:~ $ sudo systemctl status networking
● networking.service - Raise network interfaces
     Loaded: loaded (/lib/systemd/system/networking.service; enabled; vendor preset: enabled)
     Active: active (exited) since Thu 2022-10-06 09:36:27 UTC; 6min ago
       Docs: man:interfaces(5)
   Main PID: 1017 (code=exited, status=0/SUCCESS)
      Tasks: 0 (limit: 76785)
     Memory: 0B
        CPU: 0
     CGroup: /system.slice/networking.service

Oct 06 09:36:26 cloudnet1005 systemd[1]: Starting Raise network interfaces...
Oct 06 09:36:26 cloudnet1005 ifup[1046]: interface vlan1107 does not exist!
Oct 06 09:36:27 cloudnet1005 ifup[1098]: interface vlan1105 does not exist!
Oct 06 09:36:27 cloudnet1005 systemd[1]: Finished Raise network interfaces.

So apparently when the system boots, the vlan interfaces are not added to the bridge because they are not UP yet?

TheresNoTime subscribed.Oct 6 2022, 10:05 AM
Stashbot added a comment.Oct 6 2022, 11:50 AM

Mentioned in SAL (#wikimedia-cloud) [2022-10-06T11:50:31Z] <arturo> set neutron l3 agents on cloudnet1005/1006 as down root@cloudcontrol1005:~# neutron agent-update --admin-state-down <uuid> (T316284)

Stashbot added a comment.Oct 6 2022, 11:54 AM

Mentioned in SAL (#wikimedia-cloud) [2022-10-06T11:54:14Z] <arturo> rebooting cloudnet1005/1006 to see if they have the right network config (T316284)

ops-monitoring-bot added a comment.Oct 6 2022, 12:34 PM

Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin1001 for host cloudnet1006.eqiad.wmnet with OS bullseye

aborrero mentioned this in T319539: neutron: cloudnet nodes use VRRP over VXLAN to instrument HA and they require to be on the same subnet.Oct 6 2022, 12:39 PM
ops-monitoring-bot added a comment.Oct 6 2022, 12:42 PM

Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1001 for host cloudnet1006.eqiad.wmnet with OS bullseye executed with errors:

  • cloudnet1006 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Oct 6 2022, 12:43 PM

Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin1001 for host cloudnet1006.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Oct 6 2022, 12:58 PM

Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1001 for host cloudnet1006.eqiad.wmnet with OS bullseye executed with errors:

  • cloudnet1006 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Oct 6 2022, 12:59 PM

Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin1001 for host cloudnet1006.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Oct 6 2022, 1:06 PM

Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1001 for host cloudnet1006.eqiad.wmnet with OS bullseye executed with errors:

  • cloudnet1006 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Oct 6 2022, 1:06 PM

Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin1001 for host cloudnet1006.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Oct 6 2022, 1:15 PM

Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1001 for host cloudnet1006.eqiad.wmnet with OS bullseye executed with errors:

  • cloudnet1006 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Oct 6 2022, 1:16 PM

Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin1001 for host cloudnet1006.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Oct 6 2022, 2:28 PM

Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1001 for host cloudnet1006.eqiad.wmnet with OS bullseye completed:

  • cloudnet1006 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run failed, asking the operator what to do
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202210061316_aborrero_3257462_cloudnet1006.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Oct 6 2022, 2:28 PM

Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1001 for host cloudnet1006.eqiad.wmnet with OS bullseye executed with errors:

  • cloudnet1006 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run failed, asking the operator what to do
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202210061316_aborrero_3257462_cloudnet1006.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Oct 6 2022, 2:45 PM

Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin1001 for host cloudnet1005.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Oct 6 2022, 3:28 PM

Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1001 for host cloudnet1005.eqiad.wmnet with OS bullseye completed:

  • cloudnet1005 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Unable to disable Puppet, the host may have been unreachable
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202210061445_aborrero_3276126_cloudnet1005.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
aborrero added a subtask: T319683: decommission cloudnet1004.eqiad.wmnet.Oct 6 2022, 3:41 PM
aborrero added a subtask: T319682: decommission cloudnet1003.eqiad.wmnet.
Stashbot added a comment.Oct 6 2022, 3:55 PM

Mentioned in SAL (#wikimedia-cloud) [2022-10-06T15:55:51Z] <arturo> cloudnet1005 & cloudnet1006 now in service. Secom cloudnet1003 & cloudnet1004. Drop neutron agents, etc. (T316284)

aborrero closed this task as Resolved.Oct 7 2022, 9:00 AM
aborrero closed subtask T319524: Cloud VPS: neutron network: the ifupdown bridge setup can be fragile as Resolved.
jbond mentioned this in T234207: Investigate improvements to how puppet manages network interfaces.Oct 11 2022, 11:12 AM
gerritbot added a comment.Oct 11 2022, 12:01 PM

Change 841478 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloudnet: merge host hiera overrides back into the profile

https://gerrit.wikimedia.org/r/841478

gerritbot added a comment.Oct 13 2022, 8:16 AM

Change 841478 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloudnet: merge host hiera overrides back into the profile

https://gerrit.wikimedia.org/r/841478

Jclark-ctr closed subtask T319682: decommission cloudnet1003.eqiad.wmnet as Resolved.Oct 14 2022, 1:58 PM
Jclark-ctr closed subtask T319683: decommission cloudnet1004.eqiad.wmnet as Resolved.
bd808 mentioned this in T320975: Toolforge hosted IRC bots occasionally disconnecting.Oct 26 2022, 10:07 PM
gerritbot added a comment.Jun 29 2023, 3:39 PM

Change 838793 abandoned by Arturo Borrero Gonzalez:

[operations/puppet@production] cloudnet1003: decom host

Reason:

no longer relevant

https://gerrit.wikimedia.org/r/838793

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL