Page MenuHomePhabricator
Create Task
Maniphest T276637

(Need By: TBD) rack/setup/install moss-be100[12]
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of <enter the FQDN/hostname of the hosts being setup here>

Hostname / Racking / Installation Details

Hostnames: moss-be100[12]
Racking Proposal: As row and rack diverse as possible
Networking/Subnet/VLAN/IP: 10G private VLAN
Partitioning/Raid: Existing recipe and raid configuration as ms-be
OS Distro: Buster

Per host setup checklist

moss-be1001:

  • - receive in system on procurement task T275177 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - firmware update: idrac, bios, raid, network
  • - operations/puppet update - this should include updates to install_server dhcp and netboot, and site.pp role(insetup) or cp systems use role(insetup::nofirm). https://gerrit.wikimedia.org/r/c/operations/puppet/+/697990
  • - OS installation & initital puppet run via wmf-auto-reimage or wmf-auto-reimage-host
  • - host state in netbox set to staged

moss-be1002:

  • - receive in system on procurement task T275177 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - firmware update: idrac, bios, raid, network
  • - operations/puppet update - this should include updates to install_server dhcp and netboot, and site.pp role(insetup) or cp systems use role(insetup::nofirm). https://gerrit.wikimedia.org/r/c/operations/puppet/+/697990
  • - OS installation & initital puppet run via wmf-auto-reimage or wmf-auto-reimage-host
  • - host state in netbox set to staged

Once the system(s) above have had all checkbox steps completed, this task can be resolved.

Details

SubjectRepoBranchLines +/-
moss-be100[12] setup infooperations/puppetproduction+15 -0
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
Unknown Object (Task)
 ResolvedRobHT276637 (Need By: TBD) rack/setup/install moss-be100[12]

Event Timeline

RobH created this task.Mar 5 2021, 11:38 PM
RobH added a parent task: Unknown Object (Task).
RobH mentioned this in Unknown Object (Task).
RobH moved this task from Backlog to Racking Tasks on the ops-eqiad board.Mar 5 2021, 11:43 PM
RobH unsubscribed.
Maintenance_bot added a project: SRE.Mar 5 2021, 11:45 PM
fgiunchedi mentioned this in T279621: Set up Misc Object Storage Service (moss).Apr 8 2021, 7:55 AM
Jclark-ctr added a comment.May 5 2021, 10:00 PM

moss-be1001 B4 U30 port13 id5345
moss-be1002 C2 U21 port33 id5344

Jclark-ctr reassigned this task from Jclark-ctr to • Cmjohnson.May 5 2021, 10:00 PM
Jclark-ctr updated the task description. (Show Details)
Jclark-ctr subscribed.
• Cmjohnson added a comment.May 11 2021, 3:32 PM

@Jclark-ctr moss-be1001 cables are wrong, the ports you have them connected to are already labeled for cloudcephosd1016 but I see that the server is not connected to the switch and also listed as decommission in Netbox (unsure about that status as well). I am confused about what's going on with this switch and available ports. Can you let me know which ports are available?

• Cmjohnson reassigned this task from • Cmjohnson to Jclark-ctr.May 11 2021, 3:33 PM
• Cmjohnson subscribed.
• Cmjohnson updated the task description. (Show Details)May 20 2021, 5:26 PM
• Cmjohnson reassigned this task from Jclark-ctr to RobH.May 20 2021, 5:40 PM
• Cmjohnson updated the task description. (Show Details)
• Cmjohnson added a subscriber: RobH.

@RobH if you have time to do the installs that would be great, assign back to me if you're busy.

gerritbot added a comment.Jun 3 2021, 3:18 PM

Change 697990 had a related patch set uploaded (by RobH; author: RobH):

[operations/puppet@production] moss-be100[12] setup info

https://gerrit.wikimedia.org/r/697990

gerritbot added a project: Patch-For-Review.Jun 3 2021, 3:18 PM

Change 697990 merged by RobH:

[operations/puppet@production] moss-be100[12] setup info

https://gerrit.wikimedia.org/r/697990

ops-monitoring-bot added a comment.Jun 3 2021, 3:27 PM

Script wmf-auto-reimage was launched by robh on cumin1001.eqiad.wmnet for hosts:

['moss-be1001.eqiad.wmnet', 'moss-be1002.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202106031527_robh_2852.log.

RobH removed a project: Patch-For-Review.Jun 3 2021, 3:29 PM
RobH updated the task description. (Show Details)
RobH added a comment.Jun 3 2021, 3:36 PM
In T276637#7132069, @ops-monitoring-bot wrote:

Script wmf-auto-reimage was launched by robh on cumin1001.eqiad.wmnet for hosts:

['moss-be1001.eqiad.wmnet', 'moss-be1002.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202106031527_robh_2852.log.

First imaging attempt failed due to the raid disks not having been setup in a bunch of individual raid0s in advance; fixing it now just echoing result so the above script start without end has context.

ops-monitoring-bot added a comment.Jun 3 2021, 5:10 PM

Script wmf-auto-reimage was launched by robh on cumin1001.eqiad.wmnet for hosts:

['moss-be1001.eqiad.wmnet', 'moss-be1002.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202106031710_robh_31308.log.

ops-monitoring-bot added a comment.Jun 3 2021, 5:25 PM

Completed auto-reimage of hosts:

['moss-be1001.eqiad.wmnet', 'moss-be1002.eqiad.wmnet']

Of which those FAILED:

['moss-be1001.eqiad.wmnet', 'moss-be1002.eqiad.wmnet']
RobH added a comment.Jun 3 2021, 5:27 PM
17:23:22 | moss-be1001.eqiad.wmnet | Unable to run wmf-auto-reimage-host: The host moss-be1001.eqiad.wmnet should have rebooted into the newly installed Operating System but appears to have rebooted instead into the Debian installer again. Manual intervention required.
17:25:52 | moss-be1002.eqiad.wmnet | Unable to run wmf-auto-reimage-host: The host moss-be1002.eqiad.wmnet should have rebooted into the newly installed Operating System but appears to have rebooted instead into the Debian installer again. Manual intervention required.

fixing and will rerun

ops-monitoring-bot added a comment.Jun 3 2021, 5:31 PM

Script wmf-auto-reimage was launched by robh on cumin1001.eqiad.wmnet for hosts:

['moss-be1001.eqiad.wmnet', 'moss-be1002.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202106031731_robh_16029.log.

ops-monitoring-bot added a comment.Jun 3 2021, 5:58 PM

Completed auto-reimage of hosts:

['moss-be1001.eqiad.wmnet', 'moss-be1002.eqiad.wmnet']

and were ALL successful.

RobH closed this task as Resolved.Jun 3 2021, 7:05 PM
RobH updated the task description. (Show Details)
RobH added a subscriber: fgiunchedi.

@fgiunchedi these are now ready for your use!

fgiunchedi added a comment.Jun 4 2021, 8:32 AM

Ack, thank you @RobH !

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL