Page MenuHomePhabricator
Create Task
Maniphest T210450

rack/setup/install elastic203[7-9], elastic204[0-9], elastic205[0-4]
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and installation of elastic201[6-9], elastic202[0-9] and elastic203[0-3] .

Racking Proposal:
Existing servers :
elastic2001 : rack A5
elastic2002 : rack A5
elastic2003 : rack A5
elastic2004 : rack A8
elastic2005 : rack A8
elastic2006 : rack A8
elastic2007 : rack B5
elastic2008 : rack B5
elastic2009 : rack B5
elastic2010 : rack B8
elastic2011 : rack B8
elastic2012 : rack B8
elastic2013 : rack C1
elastic2014 : rack C1
elastic2015 : rack C1

New servers:
elastic2037 : rack A2
elastic2038 : rack A2
elastic2039 : rack A7
elastic2040 : rack A7

elastic2041 : rack B2
elastic2042 : rack B2
elastic2043 : rack B7
elastic2044 : rack B7

elastic2045 : rack C2
elastic2046 : rack C2
elastic2047 : rack C2
elastic2048 : rack C7
elastic2049 : rack C7

elastic2050 : rack D2
elastic2051 : rack D2
elastic2052 : rack D2
elastic2053 : rack D7
elastic2054 : rack D7

elastic2037:

  • - receive in system on procurement task T198169
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, private1 vlan for each row)
    • end on-site specific steps
  • - production dns entries added (private1-vlan for each row)
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation (stretch)
  • - puppet accept/initial run
  • - handoff for service implementation

elastic2038:

  • - receive in system on procurement task T198169
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, private1 vlan for each row)
    • end on-site specific steps
  • - production dns entries added (private1-vlan for each row)
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation (stretch)
  • - puppet accept/initial run
  • - handoff for service implementation

elastic2039:

  • - receive in system on procurement task T198169
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, private1 vlan for each row)
    • end on-site specific steps
  • - production dns entries added (private1-vlan for each row)
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation (stretch)
  • - puppet accept/initial run
  • - handoff for service implementation

elastic2040:

  • - receive in system on procurement task T198169
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, private1 vlan for each row)
    • end on-site specific steps
  • - production dns entries added (private1-vlan for each row)
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation (stretch)
  • - puppet accept/initial run
  • - handoff for service implementation

elastic2041:

  • - receive in system on procurement task T198169
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, private1 vlan for each row)
    • end on-site specific steps
  • - production dns entries added (private1-vlan for each row)
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation (stretch)
  • - puppet accept/initial run
  • - handoff for service implementation

elastic2042:

  • - receive in system on procurement task T198169
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, private1 vlan for each row)
    • end on-site specific steps
  • - production dns entries added (private1-vlan for each row)
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation (stretch)
  • - puppet accept/initial run
  • - handoff for service implementation

elastic2043:

  • - receive in system on procurement task T198169
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, private1 vlan for each row)
    • end on-site specific steps
  • - production dns entries added (private1-vlan for each row)
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation (stretch)
  • - puppet accept/initial run
  • - handoff for service implementation

elastic2044:

  • - receive in system on procurement task T198169
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, private1 vlan for each row)
    • end on-site specific steps
  • - production dns entries added (private1-vlan for each row)
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation (stretch)
  • - puppet accept/initial run
  • - handoff for service implementation

elastic2045:

  • - receive in system on procurement task T198169
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, private1 vlan for each row)
    • end on-site specific steps
  • - production dns entries added (private1-vlan for each row)
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation (stretch)
  • - puppet accept/initial run
  • - handoff for service implementation

elastic2046:

  • - receive in system on procurement task T198169
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, private1 vlan for each row)
    • end on-site specific steps
  • - production dns entries added (private1-vlan for each row)
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation (stretch)
  • - puppet accept/initial run
  • - handoff for service implementation

elastic2047:

  • - receive in system on procurement task T198169
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, private1 vlan for each row)
    • end on-site specific steps
  • - production dns entries added (private1-vlan for each row)
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation (stretch)
  • - puppet accept/initial run
  • - handoff for service implementation

elastic2048:

  • - receive in system on procurement task T198169
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, private1 vlan for each row)
    • end on-site specific steps
  • - production dns entries added (private1-vlan for each row)
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation (stretch)
  • - puppet accept/initial run
  • - handoff for service implementation

elastic2049:

  • - receive in system on procurement task T198169
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, private1 vlan for each row)
    • end on-site specific steps
  • - production dns entries added (private1-vlan for each row)
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation (stretch)
  • - puppet accept/initial run
  • - handoff for service implementation

elastic2050:

  • - receive in system on procurement task T198169
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, private1 vlan for each row)
    • end on-site specific steps
  • - production dns entries added (private1-vlan for each row)
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation (stretch)
  • - puppet accept/initial run
  • - handoff for service implementation

elastic2051:

  • - receive in system on procurement task T198169
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, private1 vlan for each row)
    • end on-site specific steps
  • - production dns entries added (private1-vlan for each row)
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation (stretch)
  • - puppet accept/initial run
  • - handoff for service implementation

elastic2052:

  • - receive in system on procurement task T198169
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, private1 vlan for each row)
    • end on-site specific steps
  • - production dns entries added (private1-vlan for each row)
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation (stretch)
  • - puppet accept/initial run
  • - handoff for service implementation

elastic2053:

  • - receive in system on procurement task T198169
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, private1 vlan for each row)
    • end on-site specific steps
  • - production dns entries added (private1-vlan for each row)
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation (stretch)
  • - puppet accept/initial run
  • - handoff for service implementation

elastic2054:

  • - receive in system on procurement task T198169
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, private1 vlan for each row)
    • end on-site specific steps
  • - production dns entries added (private1-vlan for each row)
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation (stretch)
  • - puppet accept/initial run
  • - handoff for service implementation

Details

SubjectRepoBranchLines +/-
DHCP: fix one extra space in MAC of elastic2054operations/puppetproduction+1 -1
DHCP: Add MAC address entries for elastic2045 - elastic2054operations/puppetproduction+50 -0
DNS: Add mgmt and production DNS for elastic2045 - elastic2054operations/dnsmaster+61 -7
elasticsearch: add entries in site.pp for new elasticsearch nodesoperations/puppetproduction+13 -1
Partman: ADD elastic2037-elastic2054operations/puppetproduction+1 -1
DHCP: Add MAC address entries for elastic2037-elastic2044operations/puppetproduction+40 -0
DNS: Add mgmt and production DNS entries for elastic2037-elastic2044operations/dnsmaster+48 -5
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
Unknown Object (Task)
 Resolved• Mathew.onipeT210265 Setup elasticsearch on new codfw servers
 ResolvedGehelT210450 rack/setup/install elastic203[7-9], elastic204[0-9], elastic205[0-4]

Event Timeline

Papaul triaged this task as Medium priority.Nov 26 2018, 9:08 PM
Papaul created this task.
Restricted Application added a project: SRE. · View Herald TranscriptNov 26 2018, 9:08 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Papaul updated the task description. (Show Details)Nov 27 2018, 3:10 AM
Gehel added a subscriber: • Mathew.onipe.Nov 27 2018, 3:44 PM
Papaul renamed this task from rack/setup/install elastic201[6-9], elastic202[0-9] and elastic203[0-3] to rack/setup/install elastic203[7-9], elastic204[0-9], elastic205[0-4].Nov 27 2018, 3:51 PM
Papaul updated the task description. (Show Details)
Papaul added a subscriber: • ayounsi.Nov 28 2018, 1:27 AM

@Gehel My racking proposal was to rack some of those server in rack 4 of row A,B and D. @ayounsi and I will be working on replacing all the 1G switches in those rack soon with 10G switches; or the new servers came with both 1GB and 10GB NIC's. So my question to you is :
Are you going to use the 10G NIC or 1G NIC?

Gehel added a comment.Nov 28 2018, 8:19 AM
In T210450#4779871, @Papaul wrote:

Are you going to use the 10G NIC or 1G NIC?

We're planning on using the 10G NIC.

Papaul added a comment.Nov 28 2018, 4:12 PM

@Gehel In this case the racking proposal will not work since those racks are 1G rack. I will update the task description with the new racking proposal.

Papaul updated the task description. (Show Details)Nov 28 2018, 4:13 PM
Papaul updated the task description. (Show Details)Nov 28 2018, 6:23 PM
Gehel added a comment.Nov 29 2018, 8:59 AM
In T210450#4781852, @Papaul wrote:

@Gehel In this case the racking proposal will not work since those racks are 1G rack. I will update the task description with the new racking proposal.

We want to eventually use those 10G card, as it should speed up recoveries significantly. That being said, we won't be able to fully make use of it until all servers in the cluster have been upgraded. So if that's easier for you, we can start using them with the 1G NIC and switch to 10G at some later time (I suspect this would be more effort than starting with 10G right now, but what do I know).

Gehel added a comment.Nov 29 2018, 9:01 AM

The new racking proposal looks good to me (new servers are still in the same row as the previous proposal, which is all I care about).

Gehel added a parent task: T210265: Setup elasticsearch on new codfw servers.Nov 29 2018, 9:01 AM
Papaul updated the task description. (Show Details)Nov 29 2018, 8:16 PM
Papaul updated the task description. (Show Details)Nov 29 2018, 8:34 PM
Papaul updated the task description. (Show Details)Nov 29 2018, 9:47 PM
Papaul updated the task description. (Show Details)Nov 29 2018, 10:29 PM
gerritbot subscribed.Nov 30 2018, 3:29 AM

Change 476802 had a related patch set uploaded (by Papaul; owner: Papaul):
[operations/dns@master] DNS: Add mgmt and production DNS entries for elastic2037-elastic2044

https://gerrit.wikimedia.org/r/476802

gerritbot added a project: Patch-For-Review.Nov 30 2018, 3:29 AM
Gehel mentioned this in T209779: Degraded RAID on elastic2021.Nov 30 2018, 12:14 PM
gerritbot added a comment.Nov 30 2018, 4:21 PM

Change 476887 had a related patch set uploaded (by Papaul; owner: Papaul):
[operations/puppet@production] DHCP: Add MAC address entries for elastic2037-elastic2044

https://gerrit.wikimedia.org/r/476887

gerritbot added a comment.Nov 30 2018, 4:37 PM

Change 476893 had a related patch set uploaded (by Papaul; owner: Papaul):
[operations/puppet@production] Partman: ADD elastic2037-elastic2054

https://gerrit.wikimedia.org/r/476893

Papaul updated the task description. (Show Details)Nov 30 2018, 4:41 PM
Papaul updated the task description. (Show Details)Nov 30 2018, 4:56 PM
gerritbot added a comment.Nov 30 2018, 5:14 PM

Change 476802 merged by Gehel:
[operations/dns@master] DNS: Add mgmt and production DNS entries for elastic2037-elastic2044

https://gerrit.wikimedia.org/r/476802

gerritbot added a comment.Nov 30 2018, 5:18 PM

Change 476887 merged by Gehel:
[operations/puppet@production] DHCP: Add MAC address entries for elastic2037-elastic2044

https://gerrit.wikimedia.org/r/476887

gerritbot added a comment.Nov 30 2018, 5:19 PM

Change 476893 merged by Gehel:
[operations/puppet@production] Partman: ADD elastic2037-elastic2054

https://gerrit.wikimedia.org/r/476893

Papaul updated the task description. (Show Details)Nov 30 2018, 9:29 PM
Papaul updated the task description. (Show Details)
Papaul updated the task description. (Show Details)Nov 30 2018, 9:44 PM
Papaul updated the task description. (Show Details)Dec 3 2018, 5:10 AM
Papaul added a comment.Dec 3 2018, 1:56 PM

@Gehel The first 8 servers (elastic2037-elastic2044) are ready. The only thing left is the first puppet run. When running puppet agent on those nodes I am getting the error below. I think you can fix that for me. Thank you.

Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Function Call, Could not find data item profile::elasticsearch::rack in any Hiera data file and no default supplied at /etc/puppet/modules/profile/manifests/elasticsearch.pp:21:20 on node elastic2037.codfw.wmnet

Gehel added a comment.Dec 3 2018, 1:58 PM

@Papaul it looks like elastic2037-39 already have entries as role(elasticsearch::cirrus) in site.pp and elastic2040-44 don't have any entry. I'll create them all as spares so that we can apply the elastic role with some level of control.

gerritbot added a comment.Dec 3 2018, 2:05 PM

Change 477272 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] elasticsearch: add entries in site.pp for new elasticsearch nodes

https://gerrit.wikimedia.org/r/477272

gerritbot added a comment.Dec 3 2018, 2:09 PM

Change 477272 merged by Gehel:
[operations/puppet@production] elasticsearch: add entries in site.pp for new elasticsearch nodes

https://gerrit.wikimedia.org/r/477272

Papaul added a comment.Dec 3 2018, 2:40 PM

@Gehel all first 8 servers are all yours

Papaul updated the task description. (Show Details)Dec 3 2018, 2:42 PM
Gehel mentioned this in T211023: Decommission elastic2001-2024.Dec 3 2018, 3:45 PM
Papaul updated the task description. (Show Details)Dec 3 2018, 10:08 PM
Papaul updated the task description. (Show Details)Dec 3 2018, 10:34 PM
gerritbot added a comment.Dec 3 2018, 11:52 PM

Change 477436 had a related patch set uploaded (by Papaul; owner: Papaul):
[operations/dns@master] DNS: Add mgmt and production DNS for elastic2045 - elastic2054

https://gerrit.wikimedia.org/r/477436

RobH mentioned this in T211071: decommission elastic200[01-23] for lease return in December 2018.Dec 3 2018, 11:57 PM
gerritbot added a comment.Dec 4 2018, 1:35 PM

Change 477436 merged by Gehel:
[operations/dns@master] DNS: Add mgmt and production DNS for elastic2045 - elastic2054

https://gerrit.wikimedia.org/r/477436

Papaul updated the task description. (Show Details)Dec 4 2018, 1:43 PM
gerritbot added a comment.Dec 4 2018, 9:31 PM

Change 477675 had a related patch set uploaded (by Papaul; owner: Papaul):
[operations/puppet@production] DHCP: Add MAC address entries for elastic2045 - elastic2054

https://gerrit.wikimedia.org/r/477675

Papaul updated the task description. (Show Details)Dec 4 2018, 9:34 PM
gerritbot added a comment.Dec 4 2018, 9:34 PM

Change 477675 merged by Dzahn:
[operations/puppet@production] DHCP: Add MAC address entries for elastic2045 - elastic2054

https://gerrit.wikimedia.org/r/477675

gerritbot added a comment.Dec 4 2018, 9:37 PM

Change 477676 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] DHCP: fix one extra space in MAC of elastic2054

https://gerrit.wikimedia.org/r/477676

gerritbot added a comment.Dec 4 2018, 9:40 PM

Change 477676 merged by Dzahn:
[operations/puppet@production] DHCP: fix one extra space in MAC of elastic2054

https://gerrit.wikimedia.org/r/477676

Papaul updated the task description. (Show Details)Dec 4 2018, 10:05 PM
Papaul updated the task description. (Show Details)Dec 4 2018, 10:22 PM
Papaul updated the task description. (Show Details)Dec 5 2018, 1:11 AM
Papaul reassigned this task from Papaul to Gehel.Dec 5 2018, 2:25 AM
Papaul updated the task description. (Show Details)

@Gehel all yours

Gehel added a comment.Dec 5 2018, 9:24 AM

@Papaul: thanks! We'll take it from here, and notify you as soon as the old servers are ready for decommission.

Gehel updated the task description. (Show Details)Dec 6 2018, 9:44 AM

All servers configured.

@Papaul I'm not sure if you need to track anything else on this task, but from my side, it can be closed.

• Phabricator_maintenance moved this task from Backlog to Acknowledged on the SRE board.Jan 26 2019, 10:45 PM
Papaul closed this task as Resolved.Apr 12 2019, 12:21 AM

@Gehel We can close this. Thanks

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits