Page MenuHomePhabricator
Create Task
Maniphest T254892

(Due By: 2020-07-11) rack/setup/install an-worker[1096-1101]
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of an-worker1096+

Hostname / Racking / Installation Details

Hostnames: an-worker1XXX
Racking Proposal: For now we are going to install these into the regular Hadoop cluster and experiment using them with labels for direct job scheduling. As such they can be considered regular analytics worker nodes and named accordinly: an-worker1XXX
They should be racked spread as evenly as possible along with the other hadoop worker nodes.

Networking/Subnet/VLAN/IP: 10G, single port, analytics vlan

Partitioning/Raid: Same as other analytics workers: partman/custom/analytics-flex.cfg

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

an-workers1096:

  • - receive in system on procurement task T242147
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run (with role:spare)
  • - host state in netbox set to staged

an-workers1097:

  • - receive in system on procurement task T242147
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname

[]x - network port setup (description, enable, vlan)

    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run (with role:spare)
  • - host state in netbox set to staged

an-workers1098:

  • - receive in system on procurement task T242147
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run (with role:spare)
  • - host state in netbox set to staged

an-workers1099:

  • - receive in system on procurement task T242147
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run (with role:spare)
  • - host state in netbox set to staged

an-workers1100:

  • - receive in system on procurement task T242147
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)

[x]x - bios/drac/serial setup/testing

  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run (with role:spare)
  • - host state in netbox set to staged

an-workers1101:

  • - receive in system on procurement task T242147
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run (with role:spare)
  • - host state in netbox set to staged

Once the system(s) above have had all checkbox steps completed, this task can be resolved.

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+2 -1install_server: change partition scheme for Hadoop workers with GPUs
operations/dnsmaster+4 -4updating an-worker1101 dns to reflect correct rack location
operations/puppetproduction+13 -0Adding nodes an-worker1096-1117 to site.pp
operations/dnsmaster+25 -0Adding production dns for an-worker1096-1101 ipv4/ipv6
Customize query in gerrit

Related Objects
Search...

Event Timeline

RobH created this task.Jun 9 2020, 2:40 PM
Restricted Application added a project: SRE. · View Herald TranscriptJun 9 2020, 2:40 PM
RobH added a parent task: Unknown Object (Task).Jun 9 2020, 2:40 PM
RobH moved this task from Backlog to Racking Tasks on the ops-eqiad board.
RobH removed a subscriber: RobH.
wiki_willy assigned this task to Jclark-ctr.Jun 22 2020, 9:40 PM
wiki_willy renamed this task from (Need By: TBD) rack/setup/install an-worker[1096-1101] to (Due By: 2020-07-11) rack/setup/install an-worker[1096-1101].Jun 26 2020, 7:00 PM
elukey added a subscriber: elukey.Jul 20 2020, 3:46 PM
Jclark-ctr reassigned this task from Jclark-ctr to Cmjohnson.Jul 20 2020, 10:08 PM
Jclark-ctr updated the task description. (Show Details)
Jclark-ctr added a subscriber: Jclark-ctr.

name rack position asset_tag switchport
an-worker1096 A2 39 WMF4839 27
an-worker1097 B4 36 WMF4840 47
an-worker1098 B7 35 WMF4841 39
an-worker1099 C2 32 WMF4842 31
an-worker1100 C4 28 WMF4843 39
an-worker1101 C7 37 WMF4844 37

RobH mentioned this in T259071: (Need By: TBD) rack/setup/install an-worker11[02-17].Jul 28 2020, 8:10 PM
elukey added a comment.Jul 29 2020, 6:34 AM

Cross-posting: we had a chat a while ago with dc-ops about 10g-enabled racks and availability, ending up in T243521#6005828. The config may be outdated, but we probably need to sync before adding all the nodes just to be sure?

Cmjohnson added a subscriber: wiki_willy.Aug 4 2020, 4:11 PM

@elukey @wiki_willy I am getting ready to do all of these and T259071 this week. What do you need? Do they no longer need 10G? Please let me know before any more work is done with them.

elukey added a comment.Aug 4 2020, 4:13 PM

Yep all good, you can proceed with 10G :)

gerritbot added a comment.Aug 27 2020, 3:24 PM

Change 622828 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/dns@master] Adding production dns for an-worker1096-1101 ipv4/ipv6

https://gerrit.wikimedia.org/r/622828

gerritbot added a project: Patch-For-Review.Aug 27 2020, 3:24 PM
gerritbot added a comment.Aug 27 2020, 4:17 PM

Change 622828 merged by Cmjohnson:
[operations/dns@master] Adding production dns for an-worker1096-1101 ipv4/ipv6

https://gerrit.wikimedia.org/r/622828

Maintenance_bot removed a project: Patch-For-Review.Aug 27 2020, 5:10 PM
gerritbot added a comment.Sep 2 2020, 6:19 PM

Change 623847 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/puppet@production] Adding nodes an-worker1096-1117 to site.pp

https://gerrit.wikimedia.org/r/623847

gerritbot added a project: Patch-For-Review.Sep 2 2020, 6:19 PM
gerritbot added a comment.Sep 2 2020, 6:26 PM

Change 623847 merged by Cmjohnson:
[operations/puppet@production] Adding nodes an-worker1096-1117 to site.pp

https://gerrit.wikimedia.org/r/623847

Maintenance_bot removed a project: Patch-For-Review.Sep 2 2020, 7:10 PM
ops-monitoring-bot added a comment.Sep 2 2020, 7:46 PM

Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts:

an-worker1096.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202009021946_cmjohnson_26117_an-worker1096_eqiad_wmnet.log.

ops-monitoring-bot added a comment.Sep 2 2020, 8:52 PM

Completed auto-reimage of hosts:

['an-worker1096.eqiad.wmnet']

Of which those FAILED:

['an-worker1096.eqiad.wmnet']
ops-monitoring-bot added a comment.Sep 3 2020, 4:46 PM

Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts:

an-worker1100.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202009031646_cmjohnson_21870_an-worker1100_eqiad_wmnet.log.

ops-monitoring-bot added a comment.Sep 3 2020, 5:00 PM

Completed auto-reimage of hosts:

['an-worker1100.eqiad.wmnet']

Of which those FAILED:

['an-worker1100.eqiad.wmnet']
ops-monitoring-bot added a comment.Sep 3 2020, 5:03 PM

Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts:

an-worker1100.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202009031703_cmjohnson_4494_an-worker1100_eqiad_wmnet.log.

ops-monitoring-bot added a comment.Sep 3 2020, 5:15 PM

Completed auto-reimage of hosts:

['an-worker1100.eqiad.wmnet']

Of which those FAILED:

['an-worker1100.eqiad.wmnet']
ops-monitoring-bot added a comment.Sep 3 2020, 5:28 PM

Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts:

an-worker1096.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202009031728_cmjohnson_27929_an-worker1096_eqiad_wmnet.log.

ops-monitoring-bot added a comment.Sep 3 2020, 5:29 PM

Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts:

an-worker1097.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202009031729_cmjohnson_29059_an-worker1097_eqiad_wmnet.log.

ops-monitoring-bot added a comment.Sep 3 2020, 5:31 PM

Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts:

an-worker1098.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202009031730_cmjohnson_30569_an-worker1098_eqiad_wmnet.log.

ops-monitoring-bot added a comment.Sep 3 2020, 5:32 PM

Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts:

an-worker1099.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202009031731_cmjohnson_31602_an-worker1099_eqiad_wmnet.log.

ops-monitoring-bot added a comment.Sep 3 2020, 5:33 PM

Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts:

an-worker1100.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202009031733_cmjohnson_608_an-worker1100_eqiad_wmnet.log.

ops-monitoring-bot added a comment.Sep 3 2020, 5:37 PM

Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts:

an-worker1101.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202009031737_cmjohnson_7040_an-worker1101_eqiad_wmnet.log.

ops-monitoring-bot added a comment.Sep 3 2020, 6:05 PM

Completed auto-reimage of hosts:

['an-worker1096.eqiad.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Sep 3 2020, 6:07 PM

Completed auto-reimage of hosts:

['an-worker1098.eqiad.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Sep 3 2020, 6:35 PM

Completed auto-reimage of hosts:

['an-worker1097.eqiad.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Sep 3 2020, 6:35 PM

Completed auto-reimage of hosts:

['an-worker1100.eqiad.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Sep 3 2020, 6:35 PM

Completed auto-reimage of hosts:

['an-worker1099.eqiad.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Sep 3 2020, 6:38 PM

Completed auto-reimage of hosts:

['an-worker1101.eqiad.wmnet']

Of which those FAILED:

['an-worker1101.eqiad.wmnet']
ops-monitoring-bot added a comment.Sep 3 2020, 7:59 PM

Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts:

an-worker1101.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202009031959_cmjohnson_10210_an-worker1101_eqiad_wmnet.log.

gerritbot added a comment.Sep 3 2020, 8:18 PM

Change 624258 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/dns@master] updating an-worker1101 dns to reflect correct rack location

https://gerrit.wikimedia.org/r/624258

gerritbot added a project: Patch-For-Review.Sep 3 2020, 8:18 PM
gerritbot added a comment.Sep 3 2020, 8:20 PM

Change 624258 merged by Cmjohnson:
[operations/dns@master] updating an-worker1101 dns to reflect correct rack location

https://gerrit.wikimedia.org/r/624258

ops-monitoring-bot added a comment.Sep 3 2020, 8:22 PM

Completed auto-reimage of hosts:

['an-worker1101.eqiad.wmnet']

Of which those FAILED:

['an-worker1101.eqiad.wmnet']
ops-monitoring-bot added a comment.Sep 3 2020, 8:22 PM

Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts:

an-worker1101.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202009032022_cmjohnson_31526_an-worker1101_eqiad_wmnet.log.

ops-monitoring-bot added a comment.Sep 3 2020, 9:38 PM

Completed auto-reimage of hosts:

['an-worker1101.eqiad.wmnet']

and were ALL successful.

Cmjohnson updated the task description. (Show Details)Sep 4 2020, 10:29 AM
Cmjohnson closed this task as Resolved.Sep 4 2020, 10:31 AM

resolving

elukey reopened this task as Open.Sep 7 2020, 7:04 AM
elukey added a comment.Sep 7 2020, 7:06 AM

Re-opening since some things need attention (mostly from the Analytics team):

  • these hosts don't have the flex-bay 2 disk hw raid IIRC, but 24x1.8TB disks
  • these hosts need Stretch and not Buster (we are not ready to migrate yet)
ops-monitoring-bot added a comment.Sep 7 2020, 8:53 AM

Script wmf-auto-reimage was launched by elukey on cumin1001.eqiad.wmnet for hosts:

['an-worker1096.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202009070853_elukey_28208.log.

gerritbot added a comment.Sep 7 2020, 9:22 AM

Change 625608 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] install_server: change partition scheme for Hadoop workers with GPUs

https://gerrit.wikimedia.org/r/625608

gerritbot added a comment.Sep 7 2020, 9:23 AM

Change 625608 merged by Elukey:
[operations/puppet@production] install_server: change partition scheme for Hadoop workers with GPUs

https://gerrit.wikimedia.org/r/625608

ops-monitoring-bot added a comment.Sep 7 2020, 9:35 AM

Completed auto-reimage of hosts:

['an-worker1096.eqiad.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Sep 7 2020, 9:38 AM

Script wmf-auto-reimage was launched by elukey on cumin1001.eqiad.wmnet for hosts:

['an-worker1096.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202009070938_elukey_6717.log.

ops-monitoring-bot added a comment.Sep 7 2020, 10:04 AM

Completed auto-reimage of hosts:

['an-worker1096.eqiad.wmnet']

Of which those FAILED:

['an-worker1096.eqiad.wmnet']
ops-monitoring-bot added a comment.Sep 7 2020, 10:14 AM

Script wmf-auto-reimage was launched by elukey on cumin1001.eqiad.wmnet for hosts:

['an-worker1096.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202009071014_elukey_16133.log.

ops-monitoring-bot added a comment.Sep 7 2020, 11:05 AM

Completed auto-reimage of hosts:

['an-worker1096.eqiad.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Sep 7 2020, 1:00 PM

Script wmf-auto-reimage was launched by elukey on cumin1001.eqiad.wmnet for hosts:

['an-worker1097.eqiad.wmnet', 'an-worker1098.eqiad.wmnet', 'an-worker1099.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202009071300_elukey_2391.log.

ops-monitoring-bot added a comment.Sep 7 2020, 2:06 PM

Completed auto-reimage of hosts:

['an-worker1097.eqiad.wmnet']

Of which those FAILED:

['an-worker1099.eqiad.wmnet', 'an-worker1098.eqiad.wmnet']
ops-monitoring-bot added a comment.Sep 7 2020, 3:57 PM

Script wmf-auto-reimage was launched by elukey on cumin1001.eqiad.wmnet for hosts:

['an-worker1110.eqiad.wmnet', 'an-worker1101.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202009071557_elukey_6414.log.

ops-monitoring-bot added a comment.Sep 7 2020, 5:05 PM

Completed auto-reimage of hosts:

['an-worker1110.eqiad.wmnet']

Of which those FAILED:

['an-worker1101.eqiad.wmnet']
ops-monitoring-bot added a comment.Sep 22 2020, 3:41 PM

Script wmf-auto-reimage was launched by elukey on cumin1001.eqiad.wmnet for hosts:

['an-worker1101.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202009221541_elukey_30906.log.

ops-monitoring-bot added a comment.Sep 22 2020, 4:45 PM

Completed auto-reimage of hosts:

['an-worker1101.eqiad.wmnet']

Of which those FAILED:

['an-worker1101.eqiad.wmnet']
ops-monitoring-bot added a comment.Sep 22 2020, 5:02 PM

Script wmf-auto-reimage was launched by elukey on cumin1001.eqiad.wmnet for hosts:

['an-worker1101.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202009221702_elukey_30132.log.

ops-monitoring-bot added a comment.Sep 22 2020, 6:05 PM

Completed auto-reimage of hosts:

['an-worker1101.eqiad.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Sep 28 2020, 2:34 PM

Script wmf-auto-reimage was launched by elukey on cumin1001.eqiad.wmnet for hosts:

['an-worker1099.eqiad.wmnet', 'an-worker1100.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202009281434_elukey_14594.log.

ops-monitoring-bot added a comment.Sep 28 2020, 3:35 PM

Completed auto-reimage of hosts:

['an-worker1099.eqiad.wmnet', 'an-worker1100.eqiad.wmnet']

and were ALL successful.

Cmjohnson closed this task as Resolved.Oct 1 2020, 3:14 PM
ops-monitoring-bot added a comment.Oct 2 2020, 9:15 AM

Script wmf-auto-reimage was launched by elukey on cumin1001.eqiad.wmnet for hosts:

['an-worker1117.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202010020915_elukey_713.log.

ops-monitoring-bot added a comment.Oct 5 2020, 5:14 PM

Script wmf-auto-reimage was launched by elukey on cumin1001.eqiad.wmnet for hosts:

['an-worker1111.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202010051714_elukey_31106.log.

ops-monitoring-bot added a comment.Oct 5 2020, 5:38 PM

Completed auto-reimage of hosts:

['an-worker1111.eqiad.wmnet']

Of which those FAILED:

['an-worker1111.eqiad.wmnet']
ops-monitoring-bot added a comment.Oct 5 2020, 5:39 PM

Script wmf-auto-reimage was launched by elukey on cumin1001.eqiad.wmnet for hosts:

['an-worker1113.eqiad.wmnet', 'an-worker1114.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202010051739_elukey_4928.log.

ops-monitoring-bot added a comment.Oct 5 2020, 6:05 PM

Completed auto-reimage of hosts:

['an-worker1113.eqiad.wmnet']

Of which those FAILED:

['an-worker1114.eqiad.wmnet']
RobH mentioned this in Unknown Object (Task).Oct 26 2020, 9:10 PM
Content licensed under Creative Commons Attribution-ShareAlike 4.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL