Page MenuHomePhabricator
Create Task
Maniphest T278182

(Need By: TBD) rack/setup/install cp501[3-6]
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of cp501[3-6]

Hostname / Racking / Installation Details

Hostnames: cp5013, cp5014, cp5015, cp5016
Racking Proposal: Same split as rest of cluster: odd-numbered hosts in rack 603 and even-numbered hosts in rack 604
Networking/Subnet/VLAN/IP: Onboard 1GbE disabled, use 1x 10G port from add-in card connected to top-of-rack switch, to vlan private1-eqsin (520)
Partitioning/Raid: Partman has a recipe for this config already, but we will have to add a new line to netboot.cfg for the two different kinds of hardware now in eqsin
OS Distro: Buster

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

cp5013:

  • - receive in system on procurement task T275801 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - firmware update (idrac, bios)
  • - operations/puppet update - this should include updates to install_server dhcp and netboot, and site.pp role(insetup) or cp systems use role(insetup::nofirm). https://gerrit.wikimedia.org/r/683425
  • - OS installation & initital puppet run via wmf-auto-reimage or wmf-auto-reimage-host
  • - host state in netbox set to staged

cp5014:

  • - receive in system on procurement task T275801 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - firmware update (idrac, bios)
  • - operations/puppet update - this should include updates to install_server dhcp and netboot, and site.pp role(insetup) or cp systems use role(insetup::nofirm). https://gerrit.wikimedia.org/r/683425
  • - OS installation & initital puppet run via wmf-auto-reimage or wmf-auto-reimage-host
  • - host state in netbox set to staged

cp5015:

  • - receive in system on procurement task T275801 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - firmware update (idrac, bios)
  • - operations/puppet update - this should include updates to install_server dhcp and netboot, and site.pp role(insetup) or cp systems use role(insetup::nofirm). https://gerrit.wikimedia.org/r/683425
  • - OS installation & initital puppet run via wmf-auto-reimage or wmf-auto-reimage-host
  • - host state in netbox set to staged

cp5016:

  • - receive in system on procurement task T275801 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - firmware update (idrac, bios)
  • - operations/puppet update - this should include updates to install_server dhcp and netboot, and site.pp role(insetup) or cp systems use role(insetup::nofirm). https://gerrit.wikimedia.org/r/683425
  • - OS installation & initital puppet run via wmf-auto-reimage or wmf-auto-reimage-host
  • - host state in netbox set to staged

Once the system(s) above have had all checkbox steps completed, this task can be resolved.

Details

SubjectRepoBranchLines +/-
Add missing cache::nodes for cp501[3456]operations/puppetproduction+4 -0
Puppetize cp501[3456]operations/puppetproduction+28 -8
updating cp501[3456] roleoperations/puppetproduction+1 -1
cp501[3-6] base install paramsoperations/puppetproduction+24 -0
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenNoneT274888 cp_upload @ eqsin cascading failures, February 2021
 ResolvedBBlackT275046 provision more machines for eqsin caches
Unknown Object (Task)
Unknown Object (Task)
 ResolvedRobHT278182 (Need By: TBD) rack/setup/install cp501[3-6]

Event Timeline

RobH created this task.Mar 22 2021, 8:43 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 22 2021, 8:43 PM
RobH mentioned this in Unknown Object (Task).Mar 22 2021, 8:43 PM
RobH added a parent task: Unknown Object (Task).
Maintenance_bot added a project: SRE.Mar 22 2021, 8:45 PM
RobH moved this task from Backlog to Racking Tasks on the ops-eqsin board.Mar 22 2021, 8:51 PM
RobH added a parent task: Unknown Object (Task).Mar 30 2021, 6:12 PM
RobH updated the task description. (Show Details)Mar 30 2021, 6:23 PM
gerritbot added a comment.Apr 27 2021, 6:03 PM

Change 683026 had a related patch set uploaded (by BBlack; author: BBlack):

[operations/puppet@production] Puppetize cp501[3456]

https://gerrit.wikimedia.org/r/683026

gerritbot added a project: Patch-For-Review.Apr 27 2021, 6:03 PM
RobH updated the task description. (Show Details)Apr 28 2021, 6:35 PM
gerritbot added a comment.Apr 28 2021, 7:31 PM

Change 683425 had a related patch set uploaded (by RobH; author: RobH):

[operations/puppet@production] cp501[3-6] base install params

https://gerrit.wikimedia.org/r/683425

gerritbot added a comment.Apr 28 2021, 7:46 PM

Change 683425 merged by RobH:

[operations/puppet@production] cp501[3-6] base install params

https://gerrit.wikimedia.org/r/683425

RobH updated the task description. (Show Details)Apr 28 2021, 7:55 PM
ops-monitoring-bot added a comment.Apr 28 2021, 8:35 PM

Script wmf-auto-reimage was launched by robh on cumin1001.eqiad.wmnet for hosts:

['cp5013.eqsin.wmnet', 'cp5014.eqsin.wmnet', 'cp5015.eqsin.wmnet', 'cp5016.eqsin.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202104282035_robh_16679.log.

ops-monitoring-bot added a comment.Apr 28 2021, 9:03 PM

Completed auto-reimage of hosts:

['cp5014.eqsin.wmnet', 'cp5015.eqsin.wmnet', 'cp5013.eqsin.wmnet', 'cp5016.eqsin.wmnet']

Of which those FAILED:

['cp5014.eqsin.wmnet', 'cp5015.eqsin.wmnet', 'cp5013.eqsin.wmnet', 'cp5016.eqsin.wmnet']
ops-monitoring-bot added a comment.Apr 28 2021, 9:21 PM

Script wmf-auto-reimage was launched by robh on cumin1001.eqiad.wmnet for hosts:

cp5013.eqsin.wmnet

The log can be found in /var/log/wmf-auto-reimage/202104282121_robh_24148_cp5013_eqsin_wmnet.log.

ops-monitoring-bot added a comment.Apr 28 2021, 9:44 PM

Completed auto-reimage of hosts:

['cp5013.eqsin.wmnet']

Of which those FAILED:

['cp5013.eqsin.wmnet']
gerritbot added a comment.Apr 28 2021, 9:51 PM

Change 683438 had a related patch set uploaded (by RobH; author: RobH):

[operations/puppet@production] updating cp501[3456] role

https://gerrit.wikimedia.org/r/683438

gerritbot added a comment.Apr 28 2021, 9:52 PM

Change 683438 merged by RobH:

[operations/puppet@production] updating cp501[3456] role

https://gerrit.wikimedia.org/r/683438

ops-monitoring-bot added a comment.Apr 28 2021, 9:53 PM

Script wmf-auto-reimage was launched by robh on cumin1001.eqiad.wmnet for hosts:

cp5013.eqsin.wmnet

The log can be found in /var/log/wmf-auto-reimage/202104282152_robh_30911_cp5013_eqsin_wmnet.log.

ops-monitoring-bot added a comment.Apr 28 2021, 10:34 PM

Completed auto-reimage of hosts:

['cp5013.eqsin.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Apr 28 2021, 11:09 PM

Script wmf-auto-reimage was launched by robh on cumin1001.eqiad.wmnet for hosts:

['cp5014.eqsin.wmnet', 'cp5015.eqsin.wmnet', 'cp5016.eqsin.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202104282308_robh_14085.log.

RobH updated the task description. (Show Details)Apr 28 2021, 11:09 PM
ops-monitoring-bot added a comment.Apr 28 2021, 11:52 PM

Completed auto-reimage of hosts:

['cp5014.eqsin.wmnet', 'cp5015.eqsin.wmnet', 'cp5016.eqsin.wmnet']

and were ALL successful.

RobH closed this task as Resolved.Apr 28 2021, 11:53 PM
RobH updated the task description. (Show Details)

These are all set to staged with the insetup_noferm role applied.

BBlack subscribed.EditedApr 29 2021, 9:58 PM

Note - https://gerrit.wikimedia.org/r/c/operations/puppet/+/683026 has the production roles and config, but we'll need to reimage them into this rather than just applying it, in order to get the nvme storage and partman set up consistently.

gerritbot added a comment.May 4 2021, 10:10 PM

Change 683026 merged by BBlack:

[operations/puppet@production] Puppetize cp501[3456]

https://gerrit.wikimedia.org/r/683026

Maintenance_bot removed a project: Patch-For-Review.May 4 2021, 11:11 PM
gerritbot added a comment.May 14 2021, 12:49 PM

Change 691170 had a related patch set uploaded (by BBlack; author: BBlack):

[operations/puppet@production] Add missing cache::nodes for cp501[3456]

https://gerrit.wikimedia.org/r/691170

gerritbot added a project: Patch-For-Review.May 14 2021, 12:49 PM
gerritbot added a comment.May 14 2021, 12:53 PM

Change 691170 merged by BBlack:

[operations/puppet@production] Add missing cache::nodes for cp501[3456]

https://gerrit.wikimedia.org/r/691170

Maintenance_bot removed a project: Patch-For-Review.May 14 2021, 1:11 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL