Page MenuHomePhabricator
Create Task
Maniphest T306854

Q4: (Need By: TBD) rack/setup/install cloudcontrol2005-dev, clouddb2002-dev, cloudgw2003-dev
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of cloudcontrol2005-dev, clouddb2002-dev, cloudgw2003-dev

Hostname / Racking / Installation Details

Hostnames: cloudcontrol2005-dev, clouddb2002-dev, cloudgw2003-dev
Racking Proposal: Use WMCS racks. Cloudcontrol must be placed in a row without other cloudcontrol nodes (which are in C1 and D1). Likewise, cloudgw must not share the same row with cloudgw2002-dev (which is in B5).
Networking/Subnet/VLAN/IP:
cloudcontrol2005-dev: 1 10G connection to public1
clouddb2002-dev: 1 10G connection to cloud-hosts1-codfw
cloudgw2003-dev: 2 10G connections. eth0/eno1 on cloud-hosts1-b-codfw (vlan 2118) 10.192.20.0/24 (untagged), eth1/eno2 on a trunk with cloud-instances-transport1-b-codfw (vlan 2120) and cloud-gw-transport-codfw (vlan 2107) with no address allocated)
Partitioning/Raid: RAID10 (hardware raid) and then 'partman/standard.cfg partman/hwraid-1dev.cfg'
OS Distro: Bullseye

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

cloudcontrol2005-dev: ge-1/0/14
  • - receive in system on procurement task T303441 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
clouddb2002-dev: ge-1/0/15
  • - receive in system on procurement task T303441 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
cloudgw2003-dev: ge-1/0/[16-17]
  • - receive in system on procurement task T303441 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
wikimediacloud.org: replace cloudcontrol2001-dev with cloudcontrol2005-devoperations/dnsmaster+2 -2
acme_chief: permit access to cloudcontrol2005-devoperations/puppetproduction+1 -1
Replace cloudcontrol2001-dev with cloudcontrol2005-dev.operations/puppetproduction+26 -26
Make cloudcontrol2005-dev a cloudcontrol nodeoperations/puppetproduction+22 -10
clouddb2002-dev: make a db nodeoperations/puppetproduction+1 -1
Replae labs-hosts1-b-codfw with cloud-hosts1-boperations/puppetproduction+1 -1
Add new cloud nodes to site.pp and netboot.cfgoperations/puppetproduction+1 -0
Add new Cloud node to site.pp and to netboot.cfgoperations/puppetproduction+15 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

RobH created this task.Apr 25 2022, 11:46 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 25 2022, 11:46 PM
RobH mentioned this in Unknown Object (Task).Apr 25 2022, 11:46 PM
RobH added a parent task: Unknown Object (Task).
RobH unsubscribed.
Maintenance_bot added a project: SRE.Apr 26 2022, 12:30 AM
Papaul moved this task from Backlog to Racking Tasks on the ops-codfw board.Apr 27 2022, 12:27 PM
Papaul mentioned this in T305469: codfw: Dedicate Rack B1 for cloudX-dev servers.May 12 2022, 3:20 PM
Papaul updated the task description. (Show Details)Jun 28 2022, 4:26 PM
Papaul updated the task description. (Show Details)Jun 28 2022, 4:56 PM
Andrew updated the task description. (Show Details)Jun 28 2022, 5:04 PM
Andrew subscribed.Jun 28 2022, 5:51 PM

For these boxes, please make one big hardware raid10 out of all the drives, and then use partman/standard.cfg partman/hwraid-1dev.cfg.

Andrew updated the task description. (Show Details)Jun 28 2022, 5:52 PM
gerritbot added a comment.Jun 28 2022, 6:56 PM

Change 809239 had a related patch set uploaded (by Papaul; author: Papaul):

[operations/puppet@production] Add new Cloud node to site.pp and to netboot.cfg

https://gerrit.wikimedia.org/r/809239

gerritbot added a project: Patch-For-Review.Jun 28 2022, 6:56 PM

Change 809240 had a related patch set uploaded (by Papaul; author: Papaul):

[operations/puppet@production] Add new cloud nodes to site.pp and netboot.cfg

https://gerrit.wikimedia.org/r/809240

gerritbot added a comment.Jun 28 2022, 7:05 PM

Change 809239 merged by Papaul:

[operations/puppet@production] Add new Cloud node to site.pp and to netboot.cfg

https://gerrit.wikimedia.org/r/809239

gerritbot added a comment.Jun 28 2022, 7:06 PM

Change 809240 merged by Papaul:

[operations/puppet@production] Add new cloud nodes to site.pp and netboot.cfg

https://gerrit.wikimedia.org/r/809240

ops-monitoring-bot added a comment.Jun 28 2022, 7:22 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host cloudcontrol2005-dev.wikimedia.org with OS bullseye

Maintenance_bot removed a project: Patch-For-Review.Jun 28 2022, 7:30 PM
ops-monitoring-bot added a comment.Jun 28 2022, 7:56 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host cloudcontrol2005-dev.wikimedia.org with OS bullseye completed:

  • cloudcontrol2005-dev (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202206281922_pt1979_1421692_cloudcontrol2005-dev.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
ops-monitoring-bot added a comment.Jun 28 2022, 9:08 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host clouddb2002-dev.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jun 28 2022, 9:30 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host clouddb2002-dev.codfw.wmnet with OS bullseye executed with errors:

  • clouddb2002-dev (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Jun 28 2022, 9:31 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host clouddb2002-dev.codfw.wmnet with OS bullseye

gerritbot added a comment.Jun 28 2022, 10:05 PM

Change 809304 had a related patch set uploaded (by Papaul; author: Papaul):

[operations/puppet@production] Replae labs-hosts1-b-codfw with cloud-hosts1-b

https://gerrit.wikimedia.org/r/809304

gerritbot added a project: Patch-For-Review.Jun 28 2022, 10:05 PM
gerritbot added a comment.Jun 28 2022, 10:08 PM

Change 809304 merged by Papaul:

[operations/puppet@production] Replae labs-hosts1-b-codfw with cloud-hosts1-b

https://gerrit.wikimedia.org/r/809304

ops-monitoring-bot added a comment.Jun 28 2022, 10:20 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host clouddb2002-dev.codfw.wmnet with OS bullseye executed with errors:

  • clouddb2002-dev (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
Papaul added a subscriber: ayounsi.Jun 28 2022, 10:22 PM

@ayounsi
On clouddb2002 i was getting the error mesage below

 Failed to retrieve the preconfiguration file              │
   │ The file needed for preconfiguration could not be retrieved from      │
   │ http://apt.wikimedia.org/autoinstall/subnets/labs-hosts1-b-codfw.cfg. │
   │ The installation will proceed in non-automated mode.

the file subnets/labs-hosts1-b-codfw.cfg was replaced with cloud-hosts1-codfw.cfg and never got change in the netboot.cfg file
46         10.192.20.1) echo subnets/labs-hosts1-b-codfw.cfg ;; \

i did a patch to change it to use cloud-hosts1-codfw.cfg

ops-monitoring-bot added a comment.Jun 28 2022, 10:27 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host clouddb2002-dev.codfw.wmnet with OS bullseye

Maintenance_bot removed a project: Patch-For-Review.Jun 28 2022, 10:31 PM
ops-monitoring-bot added a comment.Jun 28 2022, 11:17 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host clouddb2002-dev.codfw.wmnet with OS bullseye completed:

  • clouddb2002-dev (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202206282227_pt1979_1460329_clouddb2002-dev.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
ops-monitoring-bot added a comment.Jun 28 2022, 11:20 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host cloudgw2003-dev.codfw.wmnet with OS bullseye

Papaul updated the task description. (Show Details)Jun 28 2022, 11:28 PM
ops-monitoring-bot added a comment.Jun 29 2022, 12:17 AM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host cloudgw2003-dev.codfw.wmnet with OS bullseye completed:

  • cloudgw2003-dev (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202206282320_pt1979_1468378_cloudgw2003-dev.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
Papaul closed this task as Resolved.Jun 29 2022, 12:19 AM
Papaul updated the task description. (Show Details)

@Andrew thanks for getting me the partman recipe info. This is complete.

gerritbot added a comment.Aug 11 2022, 6:45 PM

Change 822432 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] clouddb2002-dev: make a db node

https://gerrit.wikimedia.org/r/822432

gerritbot added a project: Patch-For-Review.Aug 11 2022, 6:45 PM
gerritbot added a comment.Aug 11 2022, 6:48 PM

Change 822432 merged by Andrew Bogott:

[operations/puppet@production] clouddb2002-dev: make a db node

https://gerrit.wikimedia.org/r/822432

Maintenance_bot removed a project: Patch-For-Review.Aug 11 2022, 7:30 PM
gerritbot added a comment.Aug 12 2022, 2:09 PM

Change 822637 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] Make cloudcontrol2005-dev a cloudcontrol node

https://gerrit.wikimedia.org/r/822637

gerritbot added a project: Patch-For-Review.Aug 12 2022, 2:09 PM

Change 822638 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] Replace cloudcontrol2001-dev with cloudcontrol2005-dev.

https://gerrit.wikimedia.org/r/822638

gerritbot added a comment.Aug 12 2022, 2:24 PM

Change 822637 merged by Andrew Bogott:

[operations/puppet@production] Make cloudcontrol2005-dev a cloudcontrol node

https://gerrit.wikimedia.org/r/822637

gerritbot added a comment.Aug 12 2022, 2:51 PM

Change 822638 merged by Andrew Bogott:

[operations/puppet@production] Replace cloudcontrol2001-dev with cloudcontrol2005-dev.

https://gerrit.wikimedia.org/r/822638

gerritbot added a comment.Aug 12 2022, 2:53 PM

Change 822643 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/dns@master] wikimediacloud.org: replace cloudcontrol2001-dev with cloudcontrol2005-dev

https://gerrit.wikimedia.org/r/822643

gerritbot added a comment.Aug 12 2022, 2:56 PM

Change 822643 merged by Andrew Bogott:

[operations/dns@master] wikimediacloud.org: replace cloudcontrol2001-dev with cloudcontrol2005-dev

https://gerrit.wikimedia.org/r/822643

gerritbot added a comment.Aug 12 2022, 3:11 PM

Change 822646 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] acme_chief: permit access to cloudcontrol2005-dev

https://gerrit.wikimedia.org/r/822646

gerritbot added a comment.Aug 12 2022, 3:12 PM

Change 822646 merged by Andrew Bogott:

[operations/puppet@production] acme_chief: permit access to cloudcontrol2005-dev

https://gerrit.wikimedia.org/r/822646

ayounsi unsubscribed.Aug 12 2022, 3:13 PM
Maintenance_bot removed a project: Patch-For-Review.Aug 12 2022, 3:30 PM
Andrew added a comment.Aug 12 2022, 4:17 PM

cloudcontrol2005-dev and clouddb2002-dev are now in service.

I don't feel confident setting up cloudgw2003-dev so will probably wait on Arturo for that one.

• aborrero closed subtask T318704: Replace cloudgw2001-dev with cloudgw2003-dev as Resolved.Oct 27 2022, 4:28 PM
• lmata mentioned this in T322321: Decommission netmon1002.Nov 8 2022, 5:41 PM
Andrew added a subtask: T369308: Decommission clouddb2002-dev.codfw.wmnet.Nov 21 2024, 5:21 PM
Papaul closed subtask T369308: Decommission clouddb2002-dev.codfw.wmnet as Resolved.Nov 26 2024, 4:45 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits