Page MenuHomePhabricator
Create Task
Maniphest T201939

rack/setup/install an-master100[12].eqiad.wmnet
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and installation of two new hosts, ordered on T196079 to replace analytics100[12].

Racking Proposal: The master/standby should be located in different rows for maximum horizontal redundancy. Place them in different rows from one another.

an-master1001.eqiad.wmnet:

  • - receive in system on procurement task T196079
  • - hostname determined and hostname label applied (non-blocker to other onsite steps)
  • - rack system with proposed racking plan (see above) & update racktables (include all system info plus location)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname (if determined)
  • - network port setup (description hostname or asset tag, enable, vlan same as analytics100[12])
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation (stretch)
  • - puppet accept/initial run
  • - handoff for service implementation

an-master1002.eqiad.wmnet

  • - receive in system on procurement task T196079
  • - hostname determined and hostname label applied (non-blocker to other onsite steps)
  • - rack system with proposed racking plan (see above) & update racktables (include all system info plus location)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname (if determined)
  • - network port setup (description hostname or asset tag, enable, vlan same as analytics100[12])
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation (stretch)
  • - puppet accept/initial run
  • - handoff for service implementation

Details

ProjectBranchLines +/-Subject
operations/dnsmaster+2 -0Add IPv6 PTRs for an-master100[1,2]
operations/puppetproduction+7 -1Add IPv6 interface to an-master100[1,2]
operations/dnsmaster+6 -5correcting an-master100[12] production dns
operations/puppetproduction+16 -1setting base install params for an-master100[12]
operations/dnsmaster+40 -39updating an-master100[12] dns entries
operations/dnsmaster+8 -2Adding mgmt dns for analyticsmaster1001-2
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedelukeyT198694 Q1 2018/19 Analytics procurement
Unknown Object (Task)
 ResolvedNoneT201939 rack/setup/install an-master100[12].eqiad.wmnet
 ResolvedCmjohnsonT204999 update label on an-master100[12].eqiad.wmnet

Event Timeline

RobH triaged this task as Medium priority.Aug 14 2018, 4:42 PM
RobH created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 14 2018, 4:42 PM
RobH added a comment.Aug 14 2018, 4:46 PM
This comment was removed by RobH.
RobH reassigned this task from elukey to Ottomata.Aug 14 2018, 4:46 PM
Ottomata added a comment.Aug 14 2018, 4:56 PM

Hm, tough question! I'd be ok with analytics-master1001 and analytics-master1002. Let's do it!

RobH renamed this task from rack/setup/install 2 new hadoop master/standby systems in eqiad to rack/setup/install analytics-master100[12].eqiad.wmnet.Aug 14 2018, 4:58 PM
RobH reassigned this task from Ottomata to Cmjohnson.
RobH updated the task description. (Show Details)
Cmjohnson added a project: ops-eqiad.Aug 14 2018, 5:47 PM
Cmjohnson updated the task description. (Show Details)

@Ottomata the name is entirely too long for labels and tracking. can we shorten it a bit?

RobH added a comment.Aug 14 2018, 6:03 PM
In T201939#4502422, @Cmjohnson wrote:

@Ottomata the name is entirely too long for labels and tracking. can we shorten it a bit?

This was discussed in IRC. So the hostname labels that I've purchased for onsite use in the past is a bit too short. The solution is we'll order newer, longer hostname labels for those hostnames on the longer side. For now, the software will use the longer analytics-master1XXX hostnames, but the visible label field, and the actual hostname label, will use an abbreviated version. Racktables will tie them together, as does the asset tag mgmt entry with the long hostname.

Cmjohnson moved this task from Backlog to Racking Tasks on the ops-eqiad board.Aug 15 2018, 3:24 PM
• fdans moved this task from Incoming to Radar on the Analytics board.Aug 16 2018, 3:43 PM
gerritbot added a subscriber: gerritbot.Aug 16 2018, 4:42 PM

Change 453161 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/dns@master] Adding mgmt dns for analyticsmaster1001-2

https://gerrit.wikimedia.org/r/453161

gerritbot added a project: Patch-For-Review.Aug 16 2018, 4:42 PM

Change 453161 merged by Cmjohnson:
[operations/dns@master] Adding mgmt dns for analyticsmaster1001-2

https://gerrit.wikimedia.org/r/453161

Cmjohnson updated the task description. (Show Details)Aug 16 2018, 5:00 PM
Cmjohnson renamed this task from rack/setup/install analytics-master100[12].eqiad.wmnet to rack/setup/install analyticsmaster100[12].eqiad.wmnet.Aug 21 2018, 4:37 PM
elukey added a subscriber: User-Elukey.Aug 22 2018, 7:51 AM
elukey added a comment.Aug 24 2018, 1:12 PM

@Cmjohnson hi! A couple of (probably stupid) questions:

  • are the final node names analytics-master100[12] or analyticsmaster100[12]? I saw the mgmt entries and I am a bit confused..
  • are the two hosts racked in two different rows?
Cmjohnson added a comment.Aug 29 2018, 4:17 PM

@elukey they are analyticsmaster100x I think during a discussion with ottomata we agreed to remove the hyphen. They are racked in 2 different rows (A5, B8)

Ottomata added a comment.EditedAug 29 2018, 4:22 PM

@Cmjohnson Rob's latest comment is what I thought we agreed to:

The solution is we'll order newer, longer hostname labels for those hostnames on the longer side. For now, the software will use the longer analytics-master1XXX hostnames

Would much prefer the hyphen for consistency.

RobH added a comment.Aug 29 2018, 4:25 PM

I'll handle the dns changes to change the hostname, the label won't change for now since the longer name won't fit.

Chris: Go ahead and finish the onsite steps and task this over to me for installation. I'll handle the hostname stuff then!

elukey awarded a token.Aug 29 2018, 4:25 PM
Cmjohnson updated the task description. (Show Details)Aug 29 2018, 4:48 PM

@RobH network ports are setup and in analytics vlan

A6 ge-6/0/15 up up analytics-master1001
B8 ge-8/0/21 up up analytics-master1002

Cmjohnson renamed this task from rack/setup/install analyticsmaster100[12].eqiad.wmnet to rack/setup/install analytics-master100[12].eqiad.wmnet.Aug 30 2018, 4:45 PM
elukey edited projects, added User-Elukey; removed Patch-For-Review.Sep 3 2018, 5:57 AM
elukey mentioned this in T203635: Replace the Analytics HDFS/Yarn masters (hardware refresh).Sep 6 2018, 7:17 AM
elukey added a comment.Sep 10 2018, 9:09 AM

@RobH we thought to schedule the maintenance window to swap analytics100[1,2] with analytics-master100[1,2] for Sept 22nd, and I'd like to send a heads up to some mailing list today/tomorrow. As far as I can see there are no more blockers in this task except finalizing the hosts' production status, but please let me know if I am mistaken (so I'll amend the maintenance plans). Thanks :)

elukey added a comment.Sep 18 2018, 8:19 PM

ping :)

elukey added a comment.Sep 20 2018, 1:46 PM

@RobH @Cmjohnson any chance that we can get this done this week? I am asking since we have an important maintenance window scheduled for Tuesday, if you guys are busy I'll reschedule for later on.

RobH added a comment.Sep 20 2018, 4:33 PM

Ok, hostname update.

We need to change the port descriptions, rack table entries, and physical labels on these to an-master100X, not analytics-master100X.

RobH renamed this task from rack/setup/install analytics-master100[12].eqiad.wmnet to rack/setup/install an-master100[12].eqiad.wmnet.Sep 20 2018, 5:13 PM
RobH claimed this task.
RobH updated the task description. (Show Details)
gerritbot added a comment.Sep 20 2018, 5:22 PM

Change 461692 had a related patch set uploaded (by RobH; owner: RobH):
[operations/dns@master] updating an-master100[12] dns entries

https://gerrit.wikimedia.org/r/461692

gerritbot added a project: Patch-For-Review.Sep 20 2018, 5:22 PM

Change 461692 merged by RobH:
[operations/dns@master] updating an-master100[12] dns entries

https://gerrit.wikimedia.org/r/461692

gerritbot added a comment.Sep 20 2018, 5:36 PM

Change 461695 had a related patch set uploaded (by RobH; owner: RobH):
[operations/puppet@production] setting base install params for an-master100[12]

https://gerrit.wikimedia.org/r/461695

gerritbot added a comment.Sep 20 2018, 5:38 PM

Change 461695 merged by RobH:
[operations/puppet@production] setting base install params for an-master100[12]

https://gerrit.wikimedia.org/r/461695

RobH removed a project: Patch-For-Review.Sep 20 2018, 5:41 PM
RobH updated the task description. (Show Details)
gerritbot added a comment.Sep 20 2018, 5:59 PM

Change 461698 had a related patch set uploaded (by RobH; owner: RobH):
[operations/dns@master] correcting an-master100[12] production dns

https://gerrit.wikimedia.org/r/461698

gerritbot added a project: Patch-For-Review.Sep 20 2018, 5:59 PM

Change 461698 merged by RobH:
[operations/dns@master] correcting an-master100[12] production dns

https://gerrit.wikimedia.org/r/461698

RobH removed RobH as the assignee of this task.Sep 20 2018, 6:38 PM
RobH removed projects: Patch-For-Review, ops-eqiad.
RobH updated the task description. (Show Details)
gerritbot added a comment.Sep 20 2018, 7:54 PM

Change 461729 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Add IPv6 interface to an-master100[1,2]

https://gerrit.wikimedia.org/r/461729

gerritbot added a project: Patch-For-Review.Sep 20 2018, 7:54 PM
gerritbot added a comment.Sep 21 2018, 2:30 PM

Change 461729 merged by Elukey:
[operations/puppet@production] Add IPv6 interface to an-master100[1,2]

https://gerrit.wikimedia.org/r/461729

gerritbot added a comment.Sep 21 2018, 2:41 PM

Change 461969 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/dns@master] Add IPv6 PTRs for an-master100[1,2]

https://gerrit.wikimedia.org/r/461969

gerritbot added a comment.Sep 21 2018, 2:46 PM

Change 461969 merged by Elukey:
[operations/dns@master] Add IPv6 PTRs for an-master100[1,2]

https://gerrit.wikimedia.org/r/461969

Cmjohnson closed subtask T204999: update label on an-master100[12].eqiad.wmnet as Resolved.Sep 27 2018, 3:14 PM
elukey moved this task from Backlog to Done on the User-Elukey board.Oct 1 2018, 8:17 AM
elukey closed this task as Resolved.Oct 12 2018, 8:11 AM
elukey updated the task description. (Show Details)
Aklapper edited projects, added Analytics-Radar; removed Analytics.Jun 10 2020, 6:44 AM
Aklapper removed a subscriber: User-Elukey.Tue, May 16, 11:25 AM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL