Page MenuHomePhabricator
Create Task
Maniphest T232367

(2019-09-15) rack/setup/install ms-be105[1-6].eqiad.wmnet
Closed, ResolvedPublic
Actions

Description

This task will track the setup and installation of 6 new ms-be hosts ordered via T228461.

Hostnames: ms-be105[1-6].eqiad.wmnet
Racking Proposal: 10G racks, 2x hosts in rows A+C+D and 3x in row B to refresh old hw. Rack diversity with existing ms-be hw is preferred but not required.
Networking/Subnet/VLAN/IP: Internal subnet per row
Partitioning/Raid: use existing ms-be setup

ms-be1051:

  • - receive in system on procurement task T228461
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run
  • - host state in netbox set to staged
  • - handoff for service implementation
  • - service implementer changes from 'staged' status to 'active' status in netbox'

ms-be1052:

  • - receive in system on procurement task T228461
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run
  • - host state in netbox set to staged
  • - handoff for service implementation
  • - service implementer changes from 'staged' status to 'active' status in netbox'

ms-be1053:

  • - receive in system on procurement task T228461
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run
  • - host state in netbox set to staged
  • - handoff for service implementation
  • - service implementer changes from 'staged' status to 'active' status in netbox'

ms-be1054:

  • - receive in system on procurement task T228461
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run
  • - host state in netbox set to staged
  • - handoff for service implementation
  • - service implementer changes from 'staged' status to 'active' status in netbox'

ms-be1055:

  • - receive in system on procurement task T228461
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run
  • - host state in netbox set to staged
  • - handoff for service implementation
  • - service implementer changes from 'staged' status to 'active' status in netbox'

ms-be1056:

  • - receive in system on procurement task T228461
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run
  • - host state in netbox set to staged
  • - handoff for service implementation
  • - service implementer changes from 'staged' status to 'active' status in netbox'

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+2 -2hieradata: use servers _per_port with ms-be105[1-6]
operations/puppetproduction+6 -0hieradata: add ms-be105[1-6]
operations/puppetproduction+10 -0ms-be105[15] dhcp info
operations/puppetproduction+4 -4fixing mac entries for new ms-be systems
operations/puppetproduction+20 -0setting most new ms-be hosts mac addresses
operations/dnsmaster+12 -1ms-be105[1-6] production dns
operations/dnsmaster+37 -0Adding mgmt dns for new ms-be servers
Customize query in gerrit

Related Objects
Search...

Event Timeline

RobH triaged this task as Medium priority.Sep 9 2019, 5:02 PM
RobH created this task.
Restricted Application added a project: SRE. · View Herald TranscriptSep 9 2019, 5:02 PM
RobH renamed this task from rack/setup/install ms-be105[1-6].eqiad.wmnet to (2019-09-15) rack/setup/install ms-be105[1-6].eqiad.wmnet.Sep 9 2019, 5:02 PM
RobH added a parent task: Unknown Object (Task).
RobH edited parent tasks, added: Unknown Object (Task); removed: Unknown Object (Task).

I've set the due date via the require by date on the ordering task:

Need By Date

Mid Q1, to get enough time to be fully in service before EOQ

wiki_willy moved this task from Backlog to Racking Tasks on the ops-eqiad board.Sep 9 2019, 5:05 PM
wiki_willy assigned this task to Jclark-ctr.Sep 13 2019, 10:02 PM
gerritbot added a comment.Sep 16 2019, 6:20 PM

Change 537171 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/dns@master] Adding mgmt dns for new ms-be servers

https://gerrit.wikimedia.org/r/537171

gerritbot added a project: Patch-For-Review.Sep 16 2019, 6:20 PM
gerritbot added a comment.Sep 16 2019, 7:08 PM

Change 537171 merged by Cmjohnson:
[operations/dns@master] Adding mgmt dns for new ms-be servers

https://gerrit.wikimedia.org/r/537171

Maintenance_bot removed a project: Patch-For-Review.Sep 16 2019, 7:10 PM
Cmjohnson updated the task description. (Show Details)Sep 26 2019, 5:39 PM

@fgiunchedi I see you said for raid Partitioning/Raid: "use existing ms-be setup" Unfortunately my memory is not that great anymore can you please remind what the existing raid setup is please,

fgiunchedi added a comment.Sep 27 2019, 7:49 AM
In T232367#5527463, @Cmjohnson wrote:

@fgiunchedi I see you said for raid Partitioning/Raid: "use existing ms-be setup" Unfortunately my memory is not that great anymore can you please remind what the existing raid setup is please,

For sure! All disks in raid0 including ssd, and the ssd must show up as sda/sdb and boot devices, thanks!

fgiunchedi added a comment.Sep 27 2019, 7:54 AM

I see this task is for 6x hosts and parent T228461 is for 9x, wanted to make sure that's expected/wanted ?

Jclark-ctr added a comment.Sep 27 2019, 2:27 PM

@fgiunchedi 3 backend systems to replace ms-be101[6-8]

fgiunchedi added a comment.Sep 30 2019, 1:30 PM
In T232367#5529697, @Jclark-ctr wrote:

@fgiunchedi 3 backend systems to replace ms-be101[6-8]

Thanks @Jclark-ctr, I wanted to make sure the 3x replacement systems not in this task for rack/install/etc are tracked somewhere

fgiunchedi raised the priority of this task from Medium to High.Oct 2 2019, 3:43 PM

Please note that putting these systems in production is becoming urgent, is there a status update and/or ETA?

Cmjohnson updated the task description. (Show Details)Oct 8 2019, 3:50 PM

@fgiunchedi all the on-site work has been completed...they need production DNS

wiki_willy reassigned this task from Jclark-ctr to RobH.Oct 8 2019, 4:04 PM
wiki_willy added a subscriber: wiki_willy.

@RobH - can you take care of DNS for this to get things completed from the dc-ops side for this install? This one's super urgent, so if you can complete in the AM, it would be much appreciated. Thanks, Willy

gerritbot added a comment.Oct 8 2019, 4:35 PM

Change 541567 had a related patch set uploaded (by RobH; owner: RobH):
[operations/dns@master] ms-be105[1-6] production dns

https://gerrit.wikimedia.org/r/541567

gerritbot added a project: Patch-For-Review.Oct 8 2019, 4:35 PM
gerritbot added a comment.Oct 8 2019, 4:49 PM

Change 541567 merged by RobH:
[operations/dns@master] ms-be105[1-6] production dns

https://gerrit.wikimedia.org/r/541567

RobH updated the task description. (Show Details)Oct 8 2019, 5:02 PM
Maintenance_bot removed a project: Patch-For-Review.Oct 8 2019, 5:10 PM
RobH updated the task description. (Show Details)Oct 8 2019, 5:11 PM
gerritbot added a comment.Oct 8 2019, 5:15 PM

Change 541578 had a related patch set uploaded (by RobH; owner: RobH):
[operations/puppet@production] setting most new ms-be hosts mac addresses

https://gerrit.wikimedia.org/r/541578

gerritbot added a project: Patch-For-Review.Oct 8 2019, 5:15 PM

Change 541578 merged by RobH:
[operations/puppet@production] setting most new ms-be hosts mac addresses

https://gerrit.wikimedia.org/r/541578

RobH removed a project: Patch-For-Review.Oct 8 2019, 5:28 PM
RobH updated the task description. (Show Details)
gerritbot added a comment.Oct 8 2019, 5:50 PM

Change 541579 had a related patch set uploaded (by RobH; owner: RobH):
[operations/puppet@production] fixing mac entries for new ms-be systems

https://gerrit.wikimedia.org/r/541579

gerritbot added a project: Patch-For-Review.Oct 8 2019, 5:50 PM
gerritbot added a comment.Oct 8 2019, 5:52 PM

Change 541579 merged by RobH:
[operations/puppet@production] fixing mac entries for new ms-be systems

https://gerrit.wikimedia.org/r/541579

Maintenance_bot removed a project: Patch-For-Review.Oct 8 2019, 6:10 PM
gerritbot added a comment.Oct 8 2019, 10:07 PM

Change 541652 had a related patch set uploaded (by RobH; owner: RobH):
[operations/puppet@production] ms-be105[15] dhcp info

https://gerrit.wikimedia.org/r/541652

gerritbot added a project: Patch-For-Review.Oct 8 2019, 10:07 PM

Change 541652 merged by RobH:
[operations/puppet@production] ms-be105[15] dhcp info

https://gerrit.wikimedia.org/r/541652

Maintenance_bot removed a project: Patch-For-Review.Oct 8 2019, 10:10 PM
RobH updated the task description. (Show Details)Oct 8 2019, 11:54 PM
RobH updated the task description. (Show Details)Oct 8 2019, 11:56 PM
RobH updated the task description. (Show Details)Oct 9 2019, 12:12 AM
RobH updated the task description. (Show Details)Oct 9 2019, 12:28 AM
RobH added a comment.Oct 9 2019, 1:38 AM

@fgiunchedi:

the task description has been updated

  • ms-be105[1236] are all yours
  • i need to finish installation/setup of ms-be105[45]
RobH reassigned this task from RobH to fgiunchedi.Oct 9 2019, 6:30 PM
RobH updated the task description. (Show Details)
RobH removed a project: ops-eqiad.

@fgiunchedi,

ms-be105[1-6].eqiad.wmnet are all online and calling into puppet. You can push them into service as you see fit.

Please note when you push them into service, you still need to change their netbox status from 'staged' to 'active'. You can see the list of the ms-be105* with staged status in netbox here: https://netbox.wikimedia.org/dcim/devices/?q=ms-be105&status=3

Please resolve this task once they are all set to active.

Thanks!

RobH removed a subscriber: RobH.Oct 9 2019, 6:32 PM
gerritbot added a comment.Oct 9 2019, 8:32 PM

Change 541910 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] hieradata: add ms-be105[1-6]

https://gerrit.wikimedia.org/r/541910

gerritbot added a project: Patch-For-Review.Oct 9 2019, 8:32 PM
gerritbot added a comment.Oct 9 2019, 8:36 PM

Change 541910 merged by Filippo Giunchedi:
[operations/puppet@production] hieradata: add ms-be105[1-6]

https://gerrit.wikimedia.org/r/541910

gerritbot added a comment.Oct 9 2019, 8:42 PM

Change 541916 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] hieradata: use servers _per_port with ms-be105[1-6]

https://gerrit.wikimedia.org/r/541916

gerritbot added a comment.Oct 9 2019, 9:02 PM

Change 541916 merged by Filippo Giunchedi:
[operations/puppet@production] hieradata: use servers _per_port with ms-be105[1-6]

https://gerrit.wikimedia.org/r/541916

Maintenance_bot removed a project: Patch-For-Review.Oct 9 2019, 9:10 PM
Stashbot added a comment.Oct 9 2019, 9:27 PM

Mentioned in SAL (#wikimedia-operations) [2019-10-09T21:27:17Z] <godog> swift eqiad-prod: add ms-be105[1-6] - T232367

fgiunchedi added a project: User-fgiunchedi.Oct 9 2019, 11:30 PM
fgiunchedi moved this task from Backlog to Doing on the User-fgiunchedi board.
Stashbot added a comment.Oct 10 2019, 7:33 PM

Mentioned in SAL (#wikimedia-operations) [2019-10-10T19:33:29Z] <godog> swift eqiad-prod: add weight to ms-be105[1-6] - T232367

Dzahn mentioned this in T235234: fix IPMI over LAN on certain HP hosts.Oct 11 2019, 12:35 AM
Stashbot added a comment.Oct 17 2019, 9:39 AM

Mentioned in SAL (#wikimedia-operations) [2019-10-17T09:39:38Z] <godog> swift eqiad-prod: add weight to ms-be105[1-6] - T232367

Stashbot added a comment.Oct 21 2019, 8:08 AM

Mentioned in SAL (#wikimedia-operations) [2019-10-21T08:08:44Z] <godog> swift eqiad-prod: add weight to ms-be105[1-6] - T232367

Stashbot added a comment.Oct 25 2019, 10:01 AM

Mentioned in SAL (#wikimedia-operations) [2019-10-25T10:01:19Z] <godog> swift eqiad-prod: add weight to ms-be105[1-6] - T232367

Stashbot added a comment.Oct 28 2019, 8:15 AM

Mentioned in SAL (#wikimedia-operations) [2019-10-28T08:15:15Z] <godog> swift eqiad-prod: final weight to ms-be105[1-6] - T232367

fgiunchedi mentioned this in Unknown Object (Task).Oct 29 2019, 1:12 PM
fgiunchedi closed this task as Resolved.Oct 30 2019, 9:48 AM
fgiunchedi updated the task description. (Show Details)

Hosts are fully in service, resolving

Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL