Page MenuHomePhabricator
Create Task
Maniphest T230746

(Aug 30th, 2019) rack/setup/install elastic10[53-67].eqiad.wmnet
Closed, ResolvedPublic0 Estimated Story Points
Actions

Description

This task will track the racking of 15 new elastic systems, replacing elastic10[17-31].

Racking Proposal:

This was provided on procurement task T226843.

Current racking configuration (excluding hosts that will be decommissioned):

Row A: 1032(A3), 1033(A3), 1034(A3), 1035(A3), 1044(A6), 1045(A6), 1048(A6) (7 nodes)
Row B: 1036(B3), 1037(B3), 1038(B3), 1039(B3), 1046(B6), 1047(B6), 1049(B4), 1050(B4) (8 nodes)
Row C: 1040(C5), 1041(C5), 1042(C5), 1043(C5), 1051(C7), 1052(C7) (6 nodes)
Row D: none (0 node)

Proposal for new nodes:

Row A: 1053, 1054 (2 nodes, try to avoid A3, then avoid A6)
Row B: 1055 (1 node), Avoid B3, B4, B6 if possible.
Row C: 1056, 1057, 1058 (3 nodes) Avoid C5 first, then avoid C7 if possible
Row D: 1059, 1060, 1061, 1062, 1063, 1064, 1065, 1066, 1067 (9 nodes) evenly space in 10G racks, there are no elastic nodes staying in this row (the older ones in this row are all going away when these come online as they are in the elastic10[17-31] range.)

Minor imbalance (+- 1 node) between rows is not a major issue, feel free to propose another arrangement if it makes more sense to on-sites. Try to evenly space out elastic nodes in the row evenly in 10G racks.

Common specifications:

  • OS: Debian/Stretch
  • IP / Subnet: internal subnet for each row
  • Partitioning : RAID0 software (current systems use elasticsearch-raid0.cfg if we stay with 2 SSD per node, it should be good)

Installation Checklists

elastic1053:

  • - receive in system on procurement task T226843
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run (with role:spare)
  • - host state in netbox set to staged
  • - handoff for service implementation
  • - service implementer changes from 'staged' status to 'active' status in netbox'

elastic1054:

  • - receive in system on procurement task T226843
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run (with role:spare)
  • - host state in netbox set to staged
  • - handoff for service implementation
  • - service implementer changes from 'staged' status to 'active' status in netbox'

elastic1055:

  • - receive in system on procurement task T226843
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run (with role:spare)
  • - host state in netbox set to staged
  • - handoff for service implementation
  • - service implementer changes from 'staged' status to 'active' status in netbox'

elastic1056:

  • - receive in system on procurement task T226843
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run (with role:spare)
  • - host state in netbox set to staged
  • - handoff for service implementation
  • - service implementer changes from 'staged' status to 'active' status in netbox'

elastic1057:

  • - receive in system on procurement task T226843
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run (with role:spare)
  • - host state in netbox set to staged
  • - handoff for service implementation
  • - service implementer changes from 'staged' status to 'active' status in netbox'

elastic1058:

  • - receive in system on procurement task T226843
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run (with role:spare)
  • - host state in netbox set to staged
  • - handoff for service implementation
  • - service implementer changes from 'staged' status to 'active' status in netbox'

elastic1059:

  • - receive in system on procurement task T226843
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run (with role:spare)
  • - host state in netbox set to staged
  • - handoff for service implementation
  • - service implementer changes from 'staged' status to 'active' status in netbox'

elastic1060:

  • - receive in system on procurement task T226843
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run (with role:spare)
  • - host state in netbox set to staged
  • - handoff for service implementation
  • - service implementer changes from 'staged' status to 'active' status in netbox'

Details

SubjectRepoBranchLines +/-
elasticsearch: add node specific configuration for new serversoperations/puppetproduction+89 -4
elasticsearch: initial configuration for new elasticsearch serversoperations/puppetproduction+2 -10
Adding elastic10[53-67].eqiad.wmnet to site.pp role::spareoperations/puppetproduction+8 -0
Adding dhcpd file for elastic10[53-67]operations/puppetproduction+75 -0
Adding dns entries for elastic1053-67operations/dnsmaster+30 -0
Adding mgmt dns entries for new elastic serversoperations/dnsmaster+60 -0
old mgmt entry that was never removed for wmf3151operations/dnsmaster+0 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

RobH triaged this task as Medium priority.Aug 19 2019, 6:58 PM
RobH created this task.
Restricted Application added a project: SRE. · View Herald TranscriptAug 19 2019, 6:58 PM
RobH added a parent task: Unknown Object (Task).Aug 19 2019, 6:58 PM
RobH mentioned this in T221636: Replace elastic1017-1031.
EBernhardson subscribed.Aug 19 2019, 8:25 PM

Try to evenly space out elastic nodes in the row evenly in 1G racks.

All new elastic servers are coming in with 10G cards and should go into 10G racks.

RobH updated the task description. (Show Details)Aug 19 2019, 8:27 PM
In T230746#5423035, @EBernhardson wrote:

Try to evenly space out elastic nodes in the row evenly in 1G racks.

All new elastic servers are coming in with 10G cards and should go into 10G racks.

fixed!

• Cmjohnson moved this task from Backlog to Racking Tasks on the ops-eqiad board.Aug 29 2019, 4:36 PM
RobH renamed this task from rack/setup/install elastic10[53-67].eqiad.wmnet to (Aug 30th, 2019) rack/setup/install elastic10[53-67].eqiad.wmnet.Sep 9 2019, 4:54 PM
• Cmjohnson updated the task description. (Show Details)Sep 10 2019, 1:12 PM
gerritbot added a comment.Sep 19 2019, 5:01 PM

Change 538069 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/dns@master] Adding mgmt dns entries for new elastic servers

https://gerrit.wikimedia.org/r/538069

gerritbot added a project: Patch-For-Review.Sep 19 2019, 5:01 PM
gerritbot added a comment.Sep 19 2019, 5:05 PM

Change 538070 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/dns@master] old mgmt entry that was never removed for wmf3151

https://gerrit.wikimedia.org/r/538070

gerritbot added a comment.Sep 19 2019, 5:07 PM

Change 538070 abandoned by Cmjohnson:
old mgmt entry that was never removed for wmf3151

https://gerrit.wikimedia.org/r/538070

gerritbot added a comment.Sep 19 2019, 5:11 PM

Change 538070 restored by Cmjohnson:
old mgmt entry that was never removed for wmf3151

https://gerrit.wikimedia.org/r/538070

gerritbot added a comment.Sep 19 2019, 5:21 PM

Change 538069 abandoned by Cmjohnson:
Adding mgmt dns entries for new elastic servers

https://gerrit.wikimedia.org/r/538069

gerritbot added a comment.Sep 19 2019, 5:21 PM

Change 538070 merged by Cmjohnson:
[operations/dns@master] old mgmt entry that was never removed for wmf3151

https://gerrit.wikimedia.org/r/538070

gerritbot added a comment.Sep 19 2019, 5:22 PM

Change 538069 restored by Cmjohnson:
Adding mgmt dns entries for new elastic servers

https://gerrit.wikimedia.org/r/538069

gerritbot added a comment.Sep 19 2019, 5:24 PM

Change 538069 merged by Cmjohnson:
[operations/dns@master] Adding mgmt dns entries for new elastic servers

https://gerrit.wikimedia.org/r/538069

Jclark-ctr added a comment.Oct 2 2019, 8:49 PM

Do we want 9 host Row D only has 2 10g racks

Row A: 1053, 1054 (2 nodes, try to avoid A3, then avoid A6)
Row B: 1055 (1 node), Avoid B3, B4, B6 if possible.
Row C: 1056, 1057, 1058 (3 nodes) Avoid C5 first, then avoid C7 if possible
Row D: 1059, 1060, 1061, 1062, 1063, 1064, 1065, 1066, 1067 (9 nodes) evenly space in 10G racks, there are no elastic nodes staying in this row (the older ones in this row are all going away when these come online as they are in the elastic10[17-31] range.)

• Cmjohnson reassigned this task from • Cmjohnson to Jclark-ctr.Oct 3 2019, 5:37 PM
• Cmjohnson subscribed.

I need some clarification on this, please. We do not have 10G space for all of these servers. Especially if you want 9 servers between D2 and D7. There has been a miscommunication in the amount of 10G space we have available. Could someone please clarify whether these need 10G or can they go evenly distributed in 1G racks. Thanks!

EBernhardson added a comment.Oct 3 2019, 5:41 PM

The servers today will not be able to utilize 10G, so they could go in 1G racks for the time being. The cluster can't take advantage of 10G until all the nodes are on 10G.

RobH unsubscribed.Oct 3 2019, 5:51 PM
Gehel added a project: Discovery-Search (Current work).Oct 18 2019, 9:12 AM
Gehel moved this task from Incoming to Blocked/Waiting on the Discovery-Search (Current work) board.
Gehel added subscribers: wiki_willy, Gehel.Oct 28 2019, 4:23 PM

@wiki_willy any idea on a timeline to get those servers racked? At the moment, we have 4 servers down in the eqiad cluster (various issues, 3 of them we won't fix since we have the new servers arrived).

Gehel added a comment.Oct 28 2019, 5:04 PM
In T230746#5544673, @EBernhardson wrote:

The servers today will not be able to utilize 10G, so they could go in 1G racks for the time being. The cluster can't take advantage of 10G until all the nodes are on 10G.

To be clear, this means that we can rack those servers in 1G atm and we can revisit that when we have upgraded the full cluster.

Jclark-ctr added a comment.Oct 29 2019, 12:05 AM

All host racked and powered, console and network finished for all. Will update netbox and finish idrac settings tomorrow

Gehel mentioned this in T233403: Unassigned shards in eqiad.Oct 29 2019, 3:44 PM
Jclark-ctr added a comment.Oct 30 2019, 2:17 AM

updated netbox, Finished Idrac and bios setup

Jclark-ctr updated the task description. (Show Details)Oct 30 2019, 2:18 AM

Host Switchport
elastic1053 33
elastic1054 30
elastic1055 26
elastic1056 23
elastic1057 25
elastic1058 20
elastic1059 21
elastic1060 11
elastic1061 12
elastic1062 28
elastic1063 39
elastic1064 35
elastic1065 4
elastic1066 7
elastic1067 9

Jclark-ctr reassigned this task from Jclark-ctr to Christopher.Oct 30 2019, 12:52 PM
• Cmjohnson updated the task description. (Show Details)Oct 31 2019, 1:39 PM

@Gehel The network switches are setup, all the on-site work has been completed

EBernhardson mentioned this in T130329: Icinga should alert on free disk space < 15% (now < 12%) on Elasticsearch hosts.Nov 1 2019, 4:16 PM
• Mathew.onipe mentioned this in T231446: Reindex commonswiki as shards have grown beyond critical threshold.Nov 5 2019, 4:58 PM
EBernhardson reassigned this task from Christopher to Gehel.Nov 6 2019, 11:20 PM
EBernhardson moved this task from Blocked/Waiting to Incoming on the Discovery-Search (Current work) board.
EBernhardson added a subscriber: Christopher.
Gehel reassigned this task from Gehel to • Cmjohnson.Nov 7 2019, 9:56 AM
Gehel moved this task from Incoming to Blocked/Waiting on the Discovery-Search (Current work) board.

It looks like there is still a few steps before "handoff for service implementation"

gerritbot added a comment.Nov 8 2019, 4:45 PM

Change 549891 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/dns@master] Adding dns entries for elastic1053-67

https://gerrit.wikimedia.org/r/549891

gerritbot added a comment.Nov 8 2019, 4:52 PM

Change 549891 merged by Cmjohnson:
[operations/dns@master] Adding dns entries for elastic1053-67

https://gerrit.wikimedia.org/r/549891

gerritbot added a comment.Nov 8 2019, 5:19 PM

Change 549893 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/puppet@production] Adding dhcpd file for elastic10[53-67]

https://gerrit.wikimedia.org/r/549893

gerritbot added a comment.Nov 8 2019, 5:31 PM

Change 549893 merged by Cmjohnson:
[operations/puppet@production] Adding dhcpd file for elastic10[53-67]

https://gerrit.wikimedia.org/r/549893

• Cmjohnson updated the task description. (Show Details)Nov 8 2019, 5:34 PM
gerritbot added a comment.Nov 11 2019, 1:16 PM

Change 550116 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/puppet@production] Adding elastic10[53-67].eqiad.wmnet to site.pp role::spare

https://gerrit.wikimedia.org/r/550116

gerritbot added a comment.Nov 11 2019, 1:18 PM

Change 550116 merged by Cmjohnson:
[operations/puppet@production] Adding elastic10[53-67].eqiad.wmnet to site.pp role::spare

https://gerrit.wikimedia.org/r/550116

• Cmjohnson reassigned this task from • Cmjohnson to Gehel.Nov 11 2019, 1:21 PM
• Cmjohnson updated the task description. (Show Details)
• Cmjohnson removed a project: ops-eqiad.

@gelhel The new elasticsearch servers are installed and ready for you, I have assigned you the task and removed my project tag. Please reassign and add ops-eqiad tag back if you have any issues.

Gehel moved this task from Blocked/Waiting to Incoming on the Discovery-Search (Current work) board.Nov 11 2019, 1:26 PM

@Cmjohnson thanks! I'll have a look and let you know if there are any issues!

gerritbot added a comment.Nov 13 2019, 10:14 AM

Change 550651 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] elasticsearch: initial configuration for new elasticsearch servers

https://gerrit.wikimedia.org/r/550651

gerritbot added a comment.Nov 13 2019, 10:25 AM

Change 550651 merged by Gehel:
[operations/puppet@production] elasticsearch: initial configuration for new elasticsearch servers

https://gerrit.wikimedia.org/r/550651

Stashbot added a comment.Nov 13 2019, 10:27 AM

Mentioned in SAL (#wikimedia-operations) [2019-11-13T10:27:19Z] <gehel> start configuration of new elasticsearch servers - T230746

gerritbot added a comment.Nov 13 2019, 10:59 AM

Change 550658 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] elasticsearch: add node specific configuration for new servers

https://gerrit.wikimedia.org/r/550658

gerritbot added a comment.Nov 13 2019, 12:55 PM

Change 550658 merged by Gehel:
[operations/puppet@production] elasticsearch: add node specific configuration for new servers

https://gerrit.wikimedia.org/r/550658

Stashbot added a comment.Nov 13 2019, 3:29 PM

Mentioned in SAL (#wikimedia-operations) [2019-11-13T15:29:32Z] <gehel> configuration of new elasticsearch servers completed, all working and pooled - T230746

Stashbot added a comment.Nov 13 2019, 4:21 PM

Mentioned in SAL (#wikimedia-operations) [2019-11-13T16:21:29Z] <gehel> draining elastic1017-1031 to prepare for decommission - T230746

Gehel updated the task description. (Show Details)Nov 14 2019, 10:56 AM
Gehel moved this task from Incoming to Needs Reporting on the Discovery-Search (Current work) board.
TJones closed this task as Resolved.Nov 20 2019, 4:56 PM
bking mentioned this in T314813: Research/fix IPMI errors on production elastic hosts.Aug 8 2022, 8:37 PM
RKemper mentioned this in T317816: Enable 10G networking in cirrus elastic clusters.Sep 14 2022, 9:39 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits