Page MenuHomePhabricator
Create Task
Maniphest T342456

Q1:rack/setup/install cloudcontrol200[6-8]-dev, cloudnet200[7-8]-dev
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of cloudcontrol200[6-8]-dev, cloudnet200[7-8]-dev

Hostname / Racking / Installation Details

Hostnames: cloudcontrol200[6-8]-dev, cloudnet200[7-8]-dev
Racking Proposal: codfw B1 https://netbox.wikimedia.org/dcim/racks/51/
Networking Setup:1x 10G connection to cloudsw / cloud-hosts1-codfw vlan
Partitioning/Raid: HW Raid: Y/N, Partman recipe and/or desired Raid Level: Software RAID
OS Distro: Bullseye (default unless otherwise specified)
Sub-team Technical Contact: Who should our on-sites contact with any questions involving system racking and setup? @aborrero @Andrew

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

cloudcontrol2006-dev: RACK: B1 U: 41 Port: 40
  • - receive in system on procurement task T341239 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
cloudcontrol2007-dev: RACK: B1 U: 40 Port: 39
  • - receive in system on procurement task T341239 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
cloudcontrol2008-dev:RACK: B1 U: 39 Port: 38
  • - receive in system on procurement task T341239 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
cloudnet2007-dev: RACK: B1 U: 38 Port: 37
  • - receive in system on procurement task T341239 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
cloudnet2008-dev:RACK: B1 U: 37 Port: 36
  • - receive in system on procurement task T341239 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.

Details

SubjectRepoBranchLines +/-
Add new cloud nodes to site.pp and netboot.cfgoperations/puppetproduction+17 -2
Customize query in gerrit

Related Objects
Search...

Event Timeline

RobH created this task.Jul 21 2023, 5:59 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 21 2023, 5:59 PM
RobH added a parent task: Unknown Object (Task).Jul 21 2023, 5:59 PM
RobH moved this task from Backlog to Racking Tasks on the ops-codfw board.
RobH mentioned this in Unknown Object (Task).
RobH unsubscribed.
Maintenance_bot added a project: SRE.Jul 21 2023, 6:29 PM
Jhancock.wm subscribed.Jul 25 2023, 4:50 PM

@aborrero I wanted to check with you about the cabling information on these servers. (planning out the racking ahead of delivery)

Type of Connections:1/2, Speed:1G/10G. Vlan:Public/Private/Other(Specify) AAAA records:Y/N, Additional IP records (Cassandra)? 2, 10G. cloud-hosts1-codfw

Do cloudNet and cloudControl have different requirements? Or do all 5 servers need 3 different connections?

To further clarify, is this going in the codfw cloud rack, B1? (assuming there was a typo here)

aborrero added a comment.Jul 26 2023, 11:56 AM

hey @Jhancock.wm they need to be connected to a cloudsw device. The only such device we have in codfw is in rack B1 https://netbox.wikimedia.org/dcim/devices/4567/ so all these hosts need to be racked in there and connected there. (Ideally we would have more switches, but...)

The network connection is just 1x cable to the switch, in the cloud-hosts vlan (https://netbox.wikimedia.org/ipam/vlans/113/).

We will later do some additional vlan trunking once we put them in service.

I will update the task description to reflect this.

aborrero updated the task description. (Show Details)Jul 26 2023, 11:57 AM
aborrero added a project: User-aborrero.
aborrero added a parent task: T342750: cloud: introduce a kubernetes undercloud to run openstack (via openstack-helm).Jul 26 2023, 12:00 PM
aborrero moved this task from Backlog to Radar on the User-aborrero board.Jul 27 2023, 10:26 AM
Jhancock.wm claimed this task.Jul 31 2023, 9:53 PM
Jhancock.wm updated the task description. (Show Details)
Jhancock.wm updated the task description. (Show Details)Jul 31 2023, 10:06 PM
Jhancock.wm updated the task description. (Show Details)Jul 31 2023, 11:14 PM
Papaul updated the task description. (Show Details)Aug 1 2023, 12:16 AM
Papaul updated the task description. (Show Details)Aug 1 2023, 1:00 AM
Papaul updated the task description. (Show Details)Aug 1 2023, 3:46 PM
Papaul updated the task description. (Show Details)Aug 1 2023, 6:39 PM
gerritbot added a comment.Aug 1 2023, 6:51 PM

Change 944306 had a related patch set uploaded (by Papaul; author: Papaul):

[operations/puppet@production] Add new cloud nodes to site.pp and netboot.cfg

https://gerrit.wikimedia.org/r/944306

gerritbot added a project: Patch-For-Review.Aug 1 2023, 6:51 PM
gerritbot added a comment.Aug 1 2023, 6:56 PM

Change 944306 merged by Papaul:

[operations/puppet@production] Add new cloud nodes to site.pp and netboot.cfg

https://gerrit.wikimedia.org/r/944306

ops-monitoring-bot added a comment.Aug 1 2023, 6:56 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host cloudcontrol2006-dev.codfw.wmnet with OS bullseye

Maintenance_bot removed a project: Patch-For-Review.Aug 1 2023, 7:10 PM
ops-monitoring-bot added a comment.Aug 1 2023, 7:28 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host cloudcontrol2007-dev.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Aug 1 2023, 7:52 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host cloudcontrol2006-dev.codfw.wmnet with OS bullseye completed:

  • cloudcontrol2006-dev (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202308011934_pt1979_2594731_cloudcontrol2006-dev.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Aug 1 2023, 7:53 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host cloudcontrol2008-dev.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Aug 1 2023, 8:17 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host cloudcontrol2007-dev.codfw.wmnet with OS bullseye completed:

  • cloudcontrol2007-dev (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202308011951_pt1979_2627686_cloudcontrol2007-dev.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Aug 1 2023, 8:19 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host cloudnet2007-dev.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Aug 1 2023, 8:28 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host cloudnet2008-dev.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Aug 1 2023, 8:42 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host cloudcontrol2008-dev.codfw.wmnet with OS bullseye completed:

  • cloudcontrol2008-dev (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202308012017_pt1979_2653839_cloudcontrol2008-dev.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Aug 1 2023, 9:46 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host cloudnet2007-dev.codfw.wmnet with OS bullseye completed:

  • cloudnet2007-dev (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202308012043_pt1979_2682819_cloudnet2007-dev.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Aug 1 2023, 9:46 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host cloudnet2008-dev.codfw.wmnet with OS bullseye completed:

  • cloudnet2008-dev (WARN)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202308012052_pt1979_2695959_cloudnet2008-dev.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • Failed to run the sre.puppet.sync-netbox-hiera cookbook, run it manually
Papaul closed this task as Resolved.Aug 1 2023, 10:00 PM
Papaul updated the task description. (Show Details)
Papaul subscribed.

@Andrew this is complete

ops-monitoring-bot added a comment.Aug 1 2023, 10:29 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host cloudvirt2004-dev.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Aug 2 2023, 12:30 AM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host cloudvirt2004-dev.codfw.wmnet with OS bullseye completed:

  • cloudvirt2004-dev (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202308020014_pt1979_2824853_cloudvirt2004-dev.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
aborrero added a subtask: T346661: cloud: prepare codfw for expansion (racks, switches, ceph).Sep 18 2023, 4:01 PM
aborrero removed a parent task: T342750: cloud: introduce a kubernetes undercloud to run openstack (via openstack-helm).Sep 18 2023, 4:03 PM
rook mentioned this in T348461: bare metal deploy poc.Oct 9 2023, 5:13 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL