Page MenuHomePhabricator

(Need By 8/15/19) rack/setup/install ms-be105[7-9].eqiad.wmnet
Open, HighPublic

Description

This task will track the setup and installation of 3 new ms-be hosts ordered via T228461.

Hostnames: ms-be105[7-9].eqiad.wmnet

Racking Proposal: The other 6 ordered on this batch are in C2, A7, B4, B7, C4, D2, D7. Try to locate these three in 10G racks A4, B2, D4. (selected as they are 10G racks not used by other ms-be hosts ordered in this batch of 9 servers.)

Networking/Subnet/VLAN/IP: Internal subnet per row
Partitioning/Raid: use existing ms-be setup - All disks in their own single disk raid0 arrays including ssd, and the ssds must show up as sda/sdb and boot devices.

These hosts are replacing hosts ms-be101[6-8]. All three of those replaced hosts are located in B5-eqiad. We don't want to put the replacement hosts in that rack as we want rack/row diversity and 10G capability.

ms-be1057:

  • - receive in system on procurement task T228461
  • - rack system in A2
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run
  • - host state in netbox set to staged
  • - handoff for service implementation
  • - service implementer changes from 'staged' status to 'active' status in netbox'

ms-be1058:

  • - receive in system on procurement task T228461
  • - rack system in B2 (even though row B has limited 10G space, this has to be in row B. B2 has no other new ms-be hosts, where B[47] do.
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run
  • - host state in netbox set to staged
  • - handoff for service implementation
  • - service implementer changes from 'staged' status to 'active' status in netbox'

ms-be1059:

  • - receive in system on procurement task T228461
  • - rack system in D2.
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run
  • - host state in netbox set to staged
  • - handoff for service implementation
  • - service implementer changes from 'staged' status to 'active' status in netbox'

Details

Related Gerrit Patches:
operations/puppet : productionhieradata: add ms-be105[7-9]
operations/puppet : productionAdding dhcpd file for ms-be1057
operations/puppet : productionAdding mac address for ms-be1058-59 to dhcp file
operations/dns : masterAdding production dns for ms-be105[7-9]

Event Timeline

RobH triaged this task as High priority.Nov 5 2019, 5:35 PM
RobH created this task.
Restricted Application added a project: Operations. · View Herald TranscriptNov 5 2019, 5:35 PM
RobH added a parent task: Unknown Object (Task).Nov 5 2019, 5:35 PM
RobH updated the task description. (Show Details)Nov 5 2019, 5:43 PM
Jclark-ctr updated the task description. (Show Details)Nov 5 2019, 11:35 PM
Jclark-ctr updated the task description. (Show Details)

host racked and cabled.

RobH claimed this task.EditedNov 6 2019, 12:54 AM
RobH added a subscriber: Jclark-ctr.

Please note the mgmt dns was already input into the dns repo:

ms-be1057 10.65.5.16
ms-be1058 10.65.5.17
ms-be1059 10.65.5.18

Please go ahead and program these and then when this is all ready for remote accessibility, assign back to me. Thanks!

I see ms-be1059 has been changed from row D to row C @Jclark-ctr and I was wondering why? Note that to keep disk space per row balanced we shouldn't put more hosts in row C if we can possibly avoid it.

@fgiunchedi The host was put into rack prior to this ticket being made i can move to row D today

@fgiunchedi The host was put into rack prior to this ticket being made i can move to row D today

Thanks! Please move to row D

RobH added a comment.Nov 6 2019, 4:23 PM

@Jclark-ctr: Once all three hosts are racked, and mgmt is online and accessible, please reassign this task to me for the OS installs.

Thanks!

Jclark-ctr updated the task description. (Show Details)Nov 6 2019, 10:41 PM
Jclark-ctr added a comment.EditedNov 7 2019, 12:26 AM

@RobH Finished configuring bios and mgmt

host                 switch ports
ms-be1057	25
ms-be1058	23
ms-be1059	17
RobH mentioned this in Unknown Object (Task).Nov 8 2019, 12:25 AM
Cmjohnson updated the task description. (Show Details)Mon, Nov 11, 1:41 PM

@Jclark-ctr I am not able to login to mgmt, can you verify that the IP, Gateway and subnet are correct.

Cmjohnson moved this task from Backlog to Cloud Tasks on the ops-eqiad board.Wed, Nov 13, 3:52 PM
Cmjohnson moved this task from Cloud Tasks to Racking Tasks on the ops-eqiad board.
Cmjohnson updated the task description. (Show Details)Wed, Nov 13, 5:01 PM

Change 551170 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/dns@master] Adding production dns for ms-be105[7-9]

https://gerrit.wikimedia.org/r/551170

Change 551170 merged by Cmjohnson:
[operations/dns@master] Adding production dns for ms-be105[7-9]

https://gerrit.wikimedia.org/r/551170

Cmjohnson updated the task description. (Show Details)Fri, Nov 15, 4:08 PM
Cmjohnson reassigned this task from RobH to Jclark-ctr.Mon, Nov 18, 4:16 PM

@Jclark-ctr the mgmt password for ms-be1057 is still not working, can you try again.

Change 551563 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/puppet@production] Adding mac address for ms-be1058-59 to dhcp file

https://gerrit.wikimedia.org/r/551563

Jclark-ctr reassigned this task from Jclark-ctr to RobH.Mon, Nov 18, 9:55 PM

@Cmjohnson entered mgmt password for ms-be1057 again

Change 551563 merged by Cmjohnson:
[operations/puppet@production] Adding mac address for ms-be1058-59 to dhcp file

https://gerrit.wikimedia.org/r/551563

Change 552090 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/puppet@production] Adding dhcpd file for ms-be1057

https://gerrit.wikimedia.org/r/552090

Change 552090 merged by Cmjohnson:
[operations/puppet@production] Adding dhcpd file for ms-be1057

https://gerrit.wikimedia.org/r/552090

wiki_willy renamed this task from rack/setup/install ms-be105[7-9].eqiad.wmnet to (Need By 8/15/19) rack/setup/install ms-be105[7-9].eqiad.wmnet.Fri, Nov 22, 8:59 PM
Cmjohnson updated the task description. (Show Details)

@fgiunchedi These are ready for you for implementation. I removed the ops-eqiad tag. if you have an issue please assign to me and add the ops-eqiad tag back

@fgiunchedi These are ready for you for implementation. I removed the ops-eqiad tag. if you have an issue please assign to me and add the ops-eqiad tag back

Thanks a lot! Will implement service now and let you know if I run into issues

Change 553056 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] hieradata: add ms-be105[7-9]

https://gerrit.wikimedia.org/r/553056

Change 553056 merged by Filippo Giunchedi:
[operations/puppet@production] hieradata: add ms-be105[7-9]

https://gerrit.wikimedia.org/r/553056

Mentioned in SAL (#wikimedia-operations) [2019-11-26T10:30:26Z] <godog> swift eqiad-prod: add ms-be105[7-9] - T237438

@Cmjohnson @Jclark-ctr I'm not blocked on this (thus no reassigning) but ms-be1059 is in row D judging by its ip address and netbox says row C. I believe netbox will need updating

fgiunchedi moved this task from Backlog to Doing on the User-fgiunchedi board.

Mentioned in SAL (#wikimedia-operations) [2019-11-27T08:09:20Z] <godog> swift eqiad-prod: more weight to ms-be105[7-9] - T237438

Mentioned in SAL (#wikimedia-operations) [2019-11-28T09:51:23Z] <godog> swift eqiad-prod: more weight to ms-be105[7-9] - T237438

Mentioned in SAL (#wikimedia-operations) [2019-12-02T09:52:03Z] <godog> swift eqiad-prod: more weight to ms-be105[7-9] - T237438

Mentioned in SAL (#wikimedia-operations) [2019-12-03T14:52:00Z] <godog> swift eqiad-prod: final weight to ms-be105[7-9] - T237438

fgiunchedi reassigned this task from fgiunchedi to Cmjohnson.Thu, Dec 5, 8:37 AM
fgiunchedi added a project: ops-eqiad.

Hosts are fully in service now!

@Cmjohnson @Jclark-ctr I'm not blocked on this (thus no reassigning) but ms-be1059 is in row D judging by its ip address and netbox says row C. I believe netbox will need updating

Passing back to ops-eqiad to correct row position in netbox

fgiunchedi updated the task description. (Show Details)Thu, Dec 5, 8:38 AM
fgiunchedi moved this task from Doing to Radar on the User-fgiunchedi board.Thu, Dec 5, 8:44 AM