Page MenuHomePhabricator
Create Task
Maniphest T255520

(Due By: 2020-07-17) rack/setup/install <an-test-worker1001-1003>
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of <enter the FQDN/hostname of the hosts being setup here>

Hostname / Racking / Installation Details

Hostnames: What are the hostnames, and have you updated https://wikitech.wikimedia.org/wiki/Infrastructure_naming_conventions ?
Racking Proposal: Where should these systems be racked? Can they share with any existing systems or should they avoid any other systems sharing their rack or row?
Networking/Subnet/VLAN/IP: What are the network details? 1G or 10G? Only one network port connection, or more? Subnet/vlan and IP requirements per connect?
Partitioning/Raid: Is this hardware or software raid and what raid levels should be applied to each disk? What are the partitioning requirements and is there an existing partman recipe?

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

<an-test-worker1001>:

  • - receive in system on procurement task T242148
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run (with role:spare)
  • - host state in netbox set to staged

<an-test-worker1002>:

  • - receive in system on procurement task T242148
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run (with role:spare)
  • - host state in netbox set to staged

<an-test-worker1003>:

  • - receive in system on procurement task T242148
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run (with role:spare)
  • - host state in netbox set to staged

Once the system(s) above have had all checkbox steps completed, this task can be resolved.

Details

SubjectRepoBranchLines +/-
Adding an-test-worker1001-3 to netboot.cfgoperations/puppetproduction+1 -1
install_server: set stretch for an-test-* hostsoperations/puppetproduction+6 -0
Add partman recipe for an-test-worker nodesoperations/puppetproduction+1 -0
Adding an-test-workers to site.pp insetup roleoperations/puppetproduction+4 -0
Add an-test-workers dhcp fileoperations/puppetproduction+15 -0
Updating an-test-worker1003 mgmt ip address - wrong asset tag numberoperations/dnsmaster+1 -1
Adding all dns for an-test-worker hosts in eqiadoperations/dnsmaster+18 -2
Customize query in gerrit

Related Objects
Search...

Event Timeline

RobH created this task.Jun 16 2020, 1:32 AM
Restricted Application added a project: SRE. · View Herald TranscriptJun 16 2020, 1:32 AM
RobH added a parent task: Unknown Object (Task).Jun 16 2020, 1:33 AM
RobH moved this task from Backlog to Racking Tasks on the ops-eqiad board.
RobH added subscribers: Jclark-ctr, elukey.

@elukey,

We need to know the hostname and racking details for these 3 new hadoop testing nodes. Please provide this info and reassign this task from you to @Jclark-ctr, thanks!

Jclark-ctr updated the task description. (Show Details)Jun 17 2020, 5:50 PM
RobH assigned this task to elukey.Jun 17 2020, 7:19 PM
elukey added a comment.Jun 18 2020, 12:09 PM

@RobH we ordered 6 nodes IIRC, 3 with more storage space and 3 called "lightweight", which batch is this?

I have in mind the following naming:

  • an-test-worker100X for the hosts with more storage
  • an-test-master100X for two of the "lightweight" nodes
  • an-test-coord1001 for the remaining "lightweight" node

Already updated https://wikitech.wikimedia.org/wiki/Infrastructure_naming_conventions#Servers

elukey reassigned this task from elukey to Jclark-ctr.Jun 18 2020, 12:10 PM
Jclark-ctr added a comment.Jun 18 2020, 5:41 PM

@elukey 10g or 1g?

elukey added a comment.Jun 22 2020, 1:33 PM
In T255520#6235758, @Jclark-ctr wrote:

@elukey 10g or 1g?

1g thanks!

RobH unsubscribed.Jun 22 2020, 6:50 PM
Jclark-ctr added a comment.Jun 25 2020, 1:15 PM

@elukey any rows that these need to be in?

Jclark-ctr added a subscriber: • Cmjohnson.Jun 25 2020, 2:49 PM

@Cmjohnson will be bulk uploading to netbox after leaving data center
HOST , SWITCHPORT , RACK , UNIT, ASSET TAG
an-test-worker1001 25 A3 25 WMF4833
an-test-worker1002 5 C3 33 WMF4834
an-test-worker1003 43 D3 18 WMF4835

elukey added a comment.Jun 25 2020, 2:50 PM

No preference, if possible one host per row, otherwise any arrangement that fit bests for you!

Jclark-ctr reassigned this task from Jclark-ctr to • Cmjohnson.Jun 25 2020, 8:40 PM

added to netbox

Jclark-ctr updated the task description. (Show Details)Jun 25 2020, 8:42 PM
wiki_willy renamed this task from (Need By: TBD) rack/setup/install <hadoop testing nodes> to (Due By: 2020-07-17) rack/setup/install <hadoop testing nodes>.Jun 26 2020, 7:00 PM
• Cmjohnson added a comment.Jul 2 2020, 4:16 PM
This comment was removed by • Cmjohnson.
• Cmjohnson added a comment.Jul 8 2020, 4:32 PM

+an-test-worker1001 1H IN A 10.65.0.68
+an-test-worker1002 1H IN A 10.65.0.69
+an-test-worker1003 1H IN A 10.65.0.70

• Cmjohnson renamed this task from (Due By: 2020-07-17) rack/setup/install <hadoop testing nodes> to (Due By: 2020-07-17) rack/setup/install <an-test-worker1001-1003>.Jul 8 2020, 4:33 PM
• Cmjohnson updated the task description. (Show Details)
Jclark-ctr added a subscriber: RobH.Jul 13 2020, 5:53 PM

@Cmjohnson

In T255520#6234640, @elukey wrote:

@RobH we ordered 6 nodes IIRC, 3 with more storage space and 3 called "lightweight", which batch is this?

I have in mind the following naming:

  • an-test-worker100X for the hosts with more storage
  • an-test-master100X for two of the "lightweight" nodes
  • an-test-coord1001 for the remaining "lightweight" node

Already updated https://wikitech.wikimedia.org/wiki/Infrastructure_naming_conventions#Servers

RobH unsubscribed.Jul 13 2020, 9:36 PM
Jclark-ctr added a comment.Jul 14 2020, 1:03 PM

@Cmjohnson sorry spaced where flipped in netbox, netbox is correct now.
an-test-worker1001 A5 30 WMF4833
an-test-worker1002 C5 34 WMF4834
an-test-worker1003 D6 32 WMF4835

gerritbot added a comment.Jul 29 2020, 2:21 PM

Change 617148 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/dns@master] Adding all dns for an-test-worker hosts in eqiad

https://gerrit.wikimedia.org/r/617148

gerritbot added a project: Patch-For-Review.Jul 29 2020, 2:21 PM
gerritbot added a comment.Jul 29 2020, 2:25 PM

Change 617148 merged by Cmjohnson:
[operations/dns@master] Adding all dns for an-test-worker hosts in eqiad

https://gerrit.wikimedia.org/r/617148

Maintenance_bot removed a project: Patch-For-Review.Jul 29 2020, 3:10 PM
• Cmjohnson updated the task description. (Show Details)Aug 4 2020, 3:21 PM
gerritbot added a comment.Aug 10 2020, 3:53 PM

Change 619316 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/dns@master] Updating an-test-worker1003 mgmt ip address - wrong asset tag number

https://gerrit.wikimedia.org/r/619316

gerritbot added a project: Patch-For-Review.Aug 10 2020, 3:53 PM

Change 619316 merged by Cmjohnson:
[operations/dns@master] Updating an-test-worker1003 mgmt ip address - wrong asset tag number

https://gerrit.wikimedia.org/r/619316

Maintenance_bot removed a project: Patch-For-Review.Aug 10 2020, 4:12 PM
elukey added a parent task: T255139: Create the new Hadoop test cluster.Aug 14 2020, 9:40 AM
ayounsi subscribed.Aug 18 2020, 7:01 AM

asw2-c-eqiad:ge-5/0/36 has been flapping a lot. I disabled the port feel free to re-enable it when you're working on it.

gerritbot added a comment.Aug 24 2020, 11:50 AM

Change 622125 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/puppet@production] Add an-test-workers dhcp file

https://gerrit.wikimedia.org/r/622125

gerritbot added a project: Patch-For-Review.Aug 24 2020, 11:50 AM
gerritbot added a comment.Aug 24 2020, 11:57 AM

Change 622125 merged by Cmjohnson:
[operations/puppet@production] Add an-test-workers dhcp file

https://gerrit.wikimedia.org/r/622125

• Cmjohnson updated the task description. (Show Details)Aug 24 2020, 11:58 AM

@elukey I do not know what partman recipe you need for these. Need to update that info and enable the ports and the servers are ready for imaging.

gerritbot added a comment.Aug 24 2020, 12:05 PM

Change 622130 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/puppet@production] Adding an-test-workers to site.pp insetup role

https://gerrit.wikimedia.org/r/622130

gerritbot added a comment.Aug 24 2020, 12:06 PM

Change 622130 merged by Cmjohnson:
[operations/puppet@production] Adding an-test-workers to site.pp insetup role

https://gerrit.wikimedia.org/r/622130

Maintenance_bot removed a project: Patch-For-Review.Aug 24 2020, 12:11 PM
ayounsi unsubscribed.Aug 25 2020, 6:12 AM
elukey added a comment.Aug 26 2020, 7:23 AM

@Cmjohnson we can use the standard recipe for misc nodes, these should have 4 disks so I'd say partman/standard.cfg partman/raid10-4dev.cfg ?

gerritbot added a comment.Aug 26 2020, 7:27 AM

Change 622476 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Add partman recipe for an-test-worker nodes

https://gerrit.wikimedia.org/r/622476

gerritbot added a project: Patch-For-Review.Aug 26 2020, 7:27 AM
gerritbot added a comment.Aug 26 2020, 7:29 AM

Change 622476 merged by Elukey:
[operations/puppet@production] Add partman recipe for an-test-worker nodes

https://gerrit.wikimedia.org/r/622476

ops-monitoring-bot added a comment.Aug 26 2020, 7:57 AM

Script wmf-auto-reimage was launched by elukey on cumin1001.eqiad.wmnet for hosts:

['an-test-worker1002.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202008260757_elukey_23276.log.

gerritbot added a comment.Aug 26 2020, 8:38 AM

Change 622529 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] install_server: set stretch for an-test-* hosts

https://gerrit.wikimedia.org/r/622529

ops-monitoring-bot added a comment.Aug 26 2020, 8:40 AM

Completed auto-reimage of hosts:

['an-test-worker1002.eqiad.wmnet']

and were ALL successful.

gerritbot added a comment.Aug 26 2020, 8:53 AM

Change 622529 merged by Elukey:
[operations/puppet@production] install_server: set stretch for an-test-* hosts

https://gerrit.wikimedia.org/r/622529

ops-monitoring-bot added a comment.Aug 26 2020, 8:58 AM

Script wmf-auto-reimage was launched by elukey on cumin1001.eqiad.wmnet for hosts:

['an-test-worker1002.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202008260858_elukey_24809.log.

ops-monitoring-bot added a comment.Aug 26 2020, 9:39 AM

Completed auto-reimage of hosts:

['an-test-worker1002.eqiad.wmnet']

and were ALL successful.

elukey added a comment.Aug 26 2020, 9:57 AM

The an-test-worker1002 host is now running Stretch (not ready for Buster yet) with the following lvs volumes:

elukey@an-test-worker1002:~$ sudo lvs
  LV   VG  Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  root vg0 -wi-ao----  74.50g
  srv  vg0 -wi-ao----  14.48t
  swap vg0 -wi-ao---- 976.00m

I think that it is acceptable, we'll get 3x15T total space (in the current Hadoop test cluster we use only half a terabyte). We could think about other partition schemes but it feels that it would complicate maintenance, so I'd say let's reimage 1001/1003 and see how it goes.

ops-monitoring-bot added a comment.Aug 26 2020, 9:58 AM

Script wmf-auto-reimage was launched by elukey on cumin1001.eqiad.wmnet for hosts:

['an-test-worker1001.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202008260958_elukey_16720.log.

ops-monitoring-bot added a comment.Aug 26 2020, 10:12 AM

Script wmf-auto-reimage was launched by elukey on cumin1001.eqiad.wmnet for hosts:

['an-test-worker1003.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202008261012_elukey_30327.log.

ops-monitoring-bot added a comment.Aug 26 2020, 10:40 AM

Completed auto-reimage of hosts:

['an-test-worker1001.eqiad.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Aug 26 2020, 10:53 AM

Completed auto-reimage of hosts:

['an-test-worker1003.eqiad.wmnet']

and were ALL successful.

elukey added a comment.Aug 26 2020, 12:51 PM

Hosts reimaged, and the status on netbox is "Staged". @Cmjohnson please check if there is anything left to do, if not let's close :)

gerritbot added a comment.Aug 26 2020, 5:22 PM

Change 622619 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/puppet@production] Adding an-test-worker1001-3 to netboot.cfg

https://gerrit.wikimedia.org/r/622619

gerritbot added a comment.Aug 26 2020, 5:35 PM

Change 622619 abandoned by Cmjohnson:
[operations/puppet@production] Adding an-test-worker1001-3 to netboot.cfg

Reason:

https://gerrit.wikimedia.org/r/622619

• Cmjohnson closed this task as Resolved.Aug 27 2020, 1:37 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL