Page MenuHomePhabricator
Create Task
Maniphest T249062

(Need by: TDB) rack/setup/install cloudelastic100[56]
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of cloudelastic100[56]

Hostname / Racking / Installation Details

Hostnames: cloudelastic100[56]
Racking Proposal: Those servers should be evenly spread across rows with cloudelastic100[1-4]
Current racking:
cloudelastic1001: A2
cloudelastic1002: B2
cloudelastic1003: C2
cloudelastic1004: D2

New servers (proposal, we're fine as long as those 2 servers are not in the same row):
cloudelastic1005: A
cloudelastic1006: B

Networking/Subnet/VLAN/IP: 10G, single network port connection, same vlan as existing cloudelastic servers
Partitioning/Raid: RAID10 software: raid10-gpt-srv-lvm-ext4-6disks.cfg

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

cloudelastic1005:

  • - receive in system on procurement task T233720
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run (with role:spare)
  • - host state in netbox set to staged

cloudelastic1006:

  • - receive in system on procurement task T233720
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run (with role:spare)
  • - host state in netbox set to staged

Once the system(s) above have had all checkbox steps completed, this task can be resolved.

Details

SubjectRepoBranchLines +/-
Assign role insetup to cloudelastic100[5,6]operations/puppetproduction+5 -0
Adding netboot.cfg and dhcpd for cloudelastic100[56]operations/puppetproduction+13 -1
Adding production dns for cloudelastic1005-6operations/dnsmaster+9 -1
Adding mgmt dns for cloudelastic100[56]operations/dnsmaster+8 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

RobH created this task.Mar 31 2020, 9:43 PM
Restricted Application added a project: SRE. · View Herald TranscriptMar 31 2020, 9:43 PM
RobH moved this task from Backlog to Racking Tasks on the ops-eqiad board.Mar 31 2020, 9:43 PM
RobH added a parent task: Unknown Object (Task).
RobH mentioned this in Unknown Object (Task).
elukey subscribed.Apr 1 2020, 6:08 AM
EBernhardson subscribed.Apr 13 2020, 4:43 PM
Jclark-ctr updated the task description. (Show Details)Apr 16 2020, 3:07 PM
RobH unsubscribed.Apr 16 2020, 5:30 PM
Jclark-ctr reassigned this task from Jclark-ctr to • Cmjohnson.Apr 21 2020, 2:34 PM
Jclark-ctr subscribed.

cloudelastic1005. rack A4 U34 WMF5337 switchport 29

cloudelastic1006 rack b4 u23. WMF5338 switchport 41

Jclark-ctr updated the task description. (Show Details)Apr 21 2020, 2:35 PM
gerritbot added a comment.Apr 21 2020, 4:50 PM

Change 591377 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/dns@master] Adding mgmt dns for cloudelastic100[56]

https://gerrit.wikimedia.org/r/591377

gerritbot added a project: Patch-For-Review.Apr 21 2020, 4:50 PM
gerritbot added a comment.Apr 21 2020, 4:52 PM

Change 591377 merged by Cmjohnson:
[operations/dns@master] Adding mgmt dns for cloudelastic100[56]

https://gerrit.wikimedia.org/r/591377

• Cmjohnson updated the task description. (Show Details)Apr 21 2020, 4:53 PM

updated bios/mgmt/idrac

Maintenance_bot removed a project: Patch-For-Review.Apr 21 2020, 5:11 PM
gerritbot added a comment.Apr 27 2020, 7:04 PM

Change 592743 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/dns@master] Adding production dns for cloudelastic1005-6

https://gerrit.wikimedia.org/r/592743

gerritbot added a project: Patch-For-Review.Apr 27 2020, 7:04 PM
gerritbot added a comment.Apr 27 2020, 7:08 PM

Change 592743 merged by Cmjohnson:
[operations/dns@master] Adding production dns for cloudelastic1005-6

https://gerrit.wikimedia.org/r/592743

• Cmjohnson updated the task description. (Show Details)Apr 27 2020, 7:09 PM
Maintenance_bot removed a project: Patch-For-Review.Apr 27 2020, 7:12 PM
elukey added a comment.Apr 28 2020, 5:57 AM

@Cmjohnson thanks a lot for the work, if you have time to prioritize these two nodes during the next days it would be super helpful (the elastic cluster is a little bit under pressure during reindexations, and two more nodes will help a lot). I can take over from os install onward if you are busy!

elukey mentioned this in T231517: Investigate and fix GC issues on cloudelastic machines.Apr 29 2020, 5:57 AM
• Cmjohnson updated the task description. (Show Details)Apr 30 2020, 8:19 PM
gerritbot added a comment.Apr 30 2020, 8:57 PM

Change 593613 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/puppet@production] Adding netboot.cfg and dhcpd for cloudelastic100[56]

https://gerrit.wikimedia.org/r/593613

gerritbot added a project: Patch-For-Review.Apr 30 2020, 8:57 PM
gerritbot added a comment.May 4 2020, 7:13 AM

Change 593613 merged by Elukey:
[operations/puppet@production] Adding netboot.cfg and dhcpd for cloudelastic100[56]

https://gerrit.wikimedia.org/r/593613

gerritbot added a comment.May 4 2020, 7:19 AM

Change 594093 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Assign role insetup to cloudelastic100[5,6]

https://gerrit.wikimedia.org/r/594093

gerritbot added a comment.May 4 2020, 7:28 AM

Change 594093 merged by Elukey:
[operations/puppet@production] Assign role insetup to cloudelastic100[5,6]

https://gerrit.wikimedia.org/r/594093

ops-monitoring-bot added a comment.May 4 2020, 7:35 AM

Script wmf-auto-reimage was launched by elukey on cumin1001.eqiad.wmnet for hosts:

cloudelastic1005.wikimedia.org

The log can be found in /var/log/wmf-auto-reimage/202005040735_elukey_97184_cloudelastic1005_wikimedia_org.log.

ops-monitoring-bot added a comment.May 4 2020, 7:35 AM

Completed auto-reimage of hosts:

['cloudelastic1005.wikimedia.org']

Of which those FAILED:

['cloudelastic1005.wikimedia.org']
ops-monitoring-bot added a comment.May 4 2020, 7:36 AM

Script wmf-auto-reimage was launched by elukey on cumin1001.eqiad.wmnet for hosts:

cloudelastic1005.wikimedia.org

The log can be found in /var/log/wmf-auto-reimage/202005040736_elukey_97196_cloudelastic1005_wikimedia_org.log.

elukey updated the task description. (Show Details)May 4 2020, 7:40 AM
Maintenance_bot removed a project: Patch-For-Review.May 4 2020, 8:10 AM
ops-monitoring-bot added a comment.May 4 2020, 8:37 AM

Completed auto-reimage of hosts:

['cloudelastic1005.wikimedia.org']

Of which those FAILED:

['cloudelastic1005.wikimedia.org']
ops-monitoring-bot added a comment.May 4 2020, 8:54 AM

Script wmf-auto-reimage was launched by elukey on cumin1001.eqiad.wmnet for hosts:

cloudelastic1006.wikimedia.org

The log can be found in /var/log/wmf-auto-reimage/202005040854_elukey_105244_cloudelastic1006_wikimedia_org.log.

ops-monitoring-bot added a comment.May 4 2020, 9:10 AM

Completed auto-reimage of hosts:

['cloudelastic1006.wikimedia.org']

Of which those FAILED:

['cloudelastic1006.wikimedia.org']
ops-monitoring-bot added a comment.May 4 2020, 9:11 AM

Script wmf-auto-reimage was launched by elukey on cumin1001.eqiad.wmnet for hosts:

cloudelastic1005.wikimedia.org

The log can be found in /var/log/wmf-auto-reimage/202005040911_elukey_107851_cloudelastic1005_wikimedia_org.log.

elukey added a comment.May 4 2020, 9:32 AM

@Cmjohnson, @Jclark-ctr I am unable to DHCP 1005/1006, I don't see any DHCP REQUEST landing to either install1003/2003. I checked 1005's system config and I noticed that the NICs config is the following:

  • NIC in Slot 2 Port 1 - B0:26:28:CD:07:C0 - Boot option PXE - No link speed registered
  • NIC in Slot 2 Port 2 - B0:26:28:CD:07:C1 - Boot option None - 10Gbps link speed registered

Boot is attempted only on the NIC in slot 1, but I think that the other one is connected to the switch. I am not sure what to do in these cases, is it ok to set Boot Option PXE to the second NIC or do we need to swap cables?

ops-monitoring-bot added a comment.May 4 2020, 10:12 AM

Completed auto-reimage of hosts:

['cloudelastic1005.wikimedia.org']

Of which those FAILED:

['cloudelastic1005.wikimedia.org']
• Cmjohnson added a comment.May 4 2020, 3:19 PM

@elukey Thanks, looks like the dac cable is in the wrong nic port. This will require an on-site visit.

Jclark-ctr added a comment.May 5 2020, 2:42 PM

dac cable is in the wrong nic port switched. Confirmed with @elukey is working now

ops-monitoring-bot added a comment.May 5 2020, 2:45 PM

Script wmf-auto-reimage was launched by elukey on cumin1001.eqiad.wmnet for hosts:

cloudelastic1005.wikimedia.org

The log can be found in /var/log/wmf-auto-reimage/202005051445_elukey_67550_cloudelastic1005_wikimedia_org.log.

elukey added a comment.May 5 2020, 3:04 PM
15:03:13 | cloudelastic1005.wikimedia.org | WARNING: unable to verify that BIOS boot parameters are back to normal, got:
Boot parameter version: 1
Boot parameter 5 is valid/unlocked
Boot parameter data: 8000020000
 Boot Flags :
   - Boot Flag Valid
   - Options apply to only next boot
   - BIOS PC Compatible (legacy) boot
   - Boot Device Selector : No override
   - Console Redirection control : System Default
   - Lock Out Sleep Button
   - BIOS verbosity : Request console redirection be enabled
   - BIOS Mux Control Override : BIOS uses recommended setting of the mux at the end of POST
ops-monitoring-bot added a comment.May 5 2020, 3:07 PM

Completed auto-reimage of hosts:

['cloudelastic1005.wikimedia.org']

and were ALL successful.

ops-monitoring-bot added a comment.May 5 2020, 3:08 PM

Script wmf-auto-reimage was launched by elukey on cumin1001.eqiad.wmnet for hosts:

cloudelastic1006.wikimedia.org

The log can be found in /var/log/wmf-auto-reimage/202005051508_elukey_71925_cloudelastic1006_wikimedia_org.log.

ops-monitoring-bot added a comment.May 5 2020, 3:32 PM

Completed auto-reimage of hosts:

['cloudelastic1006.wikimedia.org']

and were ALL successful.

elukey updated the task description. (Show Details)May 5 2020, 3:56 PM
elukey closed this task as Resolved.May 6 2020, 12:47 PM
elukey updated the task description. (Show Details)
elukey added a comment.Jun 3 2020, 9:19 AM
09:09:53 | druid1002.eqiad.wmnet | WARNING: unable to verify that BIOS boot parameters are back to normal, got:
Boot parameter version: 1
Boot parameter 5 is valid/unlocked
Boot parameter data: 0004000000
 Boot Flags :
   - Boot Flag Invalid
   - Options apply to only next boot
   - BIOS PC Compatible (legacy) boot
   - Boot Device Selector : Force PXE
   - Console Redirection control : System Default
   - BIOS verbosity : Console redirection occurs per BIOS configuration setting (default)
   - BIOS Mux Control Override : BIOS uses recommended setting of the mux at the end of POST
gerritbot added a comment.Jun 4 2020, 7:15 PM

Change 602469 had a related patch set uploaded (by Ryan Kemper; owner: Ryan Kemper):
[operations/puppet@production] cloudelastic: bring new nodes into service

https://gerrit.wikimedia.org/r/602469

gerritbot added a project: Patch-For-Review.Jun 4 2020, 7:15 PM
RKemper closed subtask T254353: Service implementation for cloudelastic100[56] as Resolved.Aug 27 2020, 4:07 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL