Page MenuHomePhabricator
Create Task
Maniphest T260370

(Need By: 2020-08-31) rack/setup/install es10[26-34].eqiad.wmnet
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of es10[26-34].eqiad.wmnet

Hostname / Racking / Installation Details

Hostnames: es1026, es1027, es1028, es1029, es1030, es1031, es1032, es1033, es1034
Racking Proposal:
2 hosts at A1
2 hosts at B1
2 hosts at C3
2 hosts at D8
1 host at A3

Networking/Subnet/VLAN/IP: 1G. Same VLAN internal as the rest of existing es hosts (es1019 for instance)
Partitioning/Raid: RAID10 strip size 256k (@Marostegui will take of adding them to the correct recipe on puppet)
OS Distro: Buster

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

es1026:

  • - receive in system on procurement task T257785 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible) https://gerrit.wikimedia.org/r/c/operations/puppet/+/620881
  • - OS installation
  • - puppet accept/initial run (with role:spare)
  • - host state in netbox set to staged

es1027:

  • - receive in system on procurement task T257785 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible) https://gerrit.wikimedia.org/r/c/operations/puppet/+/620881
  • - OS installation
  • - puppet accept/initial run (with role:spare)
  • - host state in netbox set to staged

es1028:

  • - receive in system on procurement task T257785 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible) https://gerrit.wikimedia.org/r/c/operations/puppet/+/620881
  • - OS installation
  • - puppet accept/initial run (with role:spare)
  • - host state in netbox set to staged

es1029:

  • - receive in system on procurement task T257785 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible) https://gerrit.wikimedia.org/r/c/operations/puppet/+/620881
  • - OS installation
  • - puppet accept/initial run (with role:spare)
  • - host state in netbox set to staged

es1030:

  • - receive in system on procurement task T257785 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible) https://gerrit.wikimedia.org/r/c/operations/puppet/+/620881
  • - OS installation
  • - puppet accept/initial run (with role:spare)
  • - host state in netbox set to staged

es1031:

  • - receive in system on procurement task T257785 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible) https://gerrit.wikimedia.org/r/c/operations/puppet/+/620881
  • - OS installation
  • - puppet accept/initial run (with role:spare)
  • - host state in netbox set to staged

es1032:

  • - receive in system on procurement task T257785 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible) https://gerrit.wikimedia.org/r/c/operations/puppet/+/620881
  • - OS installation
  • - puppet accept/initial run (with role:spare)
  • - host state in netbox set to staged

es1033:

  • - receive in system on procurement task T257785 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible) https://gerrit.wikimedia.org/r/c/operations/puppet/+/620881
  • - OS installation
  • - puppet accept/initial run (with role:spare)
  • - host state in netbox set to staged

es1034:

  • - receive in system on procurement task T257785 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible) https://gerrit.wikimedia.org/r/c/operations/puppet/+/620881
  • - OS installation
  • - puppet accept/initial run (with role:spare)
  • - host state in netbox set to staged

Once the system(s) above have had all checkbox steps completed, this task can be resolved.

Details

SubjectRepoBranchLines +/-
site.pp: Remove duplicate external store entries.operations/puppetproduction+1 -10
Add new es servers to site.pp setup role and mac addresses to dhcpoperations/puppetproduction+54 -0
updating site.pp entry for new ES servers in eqiadoperations/puppetproduction+9 -9
mariadb: Allow install new es hosts in eqiad/codfwoperations/puppetproduction+48 -1
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
 ResolvedMarosteguiT261717 Productionize es20[26-34] and es10[26-34]
 ResolvedRobHT260370 (Need By: 2020-08-31) rack/setup/install es10[26-34].eqiad.wmnet

Event Timeline

RobH created this task.Aug 13 2020, 4:24 PM
Restricted Application added a project: SRE. · View Herald TranscriptAug 13 2020, 4:24 PM
RobH added a parent task: Unknown Object (Task).Aug 13 2020, 4:24 PM
RobH moved this task from Backlog to Racking Tasks on the ops-eqiad board.
RobH moved this task from Backlog to Acknowledged on the SRE board.
RobH unsubscribed.
Marostegui moved this task from Triage to Blocked external/Not db team on the DBA board.Aug 17 2020, 5:27 AM
gerritbot added a comment.Aug 18 2020, 8:17 AM

Change 620881 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Allow install new es hosts in eqiad/codfw

https://gerrit.wikimedia.org/r/620881

gerritbot added a project: Patch-For-Review.Aug 18 2020, 8:17 AM
gerritbot added a comment.Aug 18 2020, 9:37 AM

Change 620881 merged by Marostegui:
[operations/puppet@production] mariadb: Allow install new es hosts in eqiad/codfw

https://gerrit.wikimedia.org/r/620881

Marostegui added a comment.Aug 18 2020, 9:39 AM

I have merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/620881 so the hosts will get installed with RAID10, notifications disabled and spare role.
Pending from DC-Ops are the usual DNS and DHCP commits.

Marostegui updated the task description. (Show Details)Aug 18 2020, 9:41 AM
Maintenance_bot removed a project: Patch-For-Review.Aug 18 2020, 10:11 AM
wiki_willy reassigned this task from • Cmjohnson to Jclark-ctr.Aug 19 2020, 5:58 PM
wiki_willy added a subscriber: • Cmjohnson.
Marostegui added a parent task: T261717: Productionize es20[26-34] and es10[26-34].Sep 1 2020, 5:22 AM
Marostegui added a subscriber: wiki_willy.Sep 23 2020, 11:47 AM

@wiki_willy do you think it is feasible to have these hosts racked&installed by 30th Oct?

wiki_willy added a subscriber: RobH.Sep 23 2020, 2:58 PM

Hi @Marostegui - I think that should be doable. During my sync up with @Cmjohnson and @RobH tomorrow, we'll discuss and see if we can get a solid ETA for you. Thanks, Willy

Marostegui added a comment.Sep 23 2020, 2:59 PM

Thank you!

wiki_willy reassigned this task from Jclark-ctr to • Cmjohnson.Sep 25 2020, 2:00 AM
wiki_willy added a subscriber: Jclark-ctr.

Confirmed with @Cmjohnson and @RobH today, that these es1026-1034 hosts will be ready for you by end of October. Thanks, Willy

• Cmjohnson added a comment.Oct 1 2020, 2:49 PM

@Jclark-ctr I see that the task has been resolved in coupa but I don't see the servers anywhere and they're not in netbox. Where are they?

wiki_willy added a comment.Oct 1 2020, 7:25 PM

@Cmjohnson - I just asked John and he says these are in shipping. Thanks Willy

• Cmjohnson updated the task description. (Show Details)Oct 26 2020, 3:59 PM
RobH unsubscribed.Oct 26 2020, 4:15 PM
gerritbot added a comment.Oct 26 2020, 5:37 PM

Change 636467 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/puppet@production] Add new es servers to site.pp setup role and mac addresses to dhcp

https://gerrit.wikimedia.org/r/636467

gerritbot added a project: Patch-For-Review.Oct 26 2020, 5:37 PM
gerritbot added a comment.Oct 26 2020, 5:39 PM

Change 636467 merged by Cmjohnson:
[operations/puppet@production] Add new es servers to site.pp setup role and mac addresses to dhcp

https://gerrit.wikimedia.org/r/636467

Maintenance_bot removed a project: Patch-For-Review.Oct 26 2020, 6:11 PM
Marostegui updated the task description. (Show Details)Oct 27 2020, 8:08 AM

Per my chat with Chris, updating the rack location from A2 to A1 and from C2 to C3

• Cmjohnson reassigned this task from • Cmjohnson to RobH.Oct 28 2020, 3:19 PM
• Cmjohnson updated the task description. (Show Details)
• Cmjohnson added a subscriber: RobH.

@RobH These still need the raid setup, you mentioned you could do that. If not please let me know and I will take care of it. Other than that they are ready for install.

gerritbot added a comment.Oct 28 2020, 4:09 PM

Change 636972 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/puppet@production] updating site.pp entry for new ES servers in eqiad

https://gerrit.wikimedia.org/r/636972

gerritbot added a project: Patch-For-Review.Oct 28 2020, 4:09 PM

Change 636972 merged by Cmjohnson:
[operations/puppet@production] updating site.pp entry for new ES servers in eqiad

https://gerrit.wikimedia.org/r/636972

gerritbot added a comment.Oct 28 2020, 4:20 PM

Change 636974 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] site.pp: Remove duplicate external store entries.

https://gerrit.wikimedia.org/r/636974

gerritbot added a comment.Oct 28 2020, 4:21 PM

Change 636974 merged by Marostegui:
[operations/puppet@production] site.pp: Remove duplicate external store entries.

https://gerrit.wikimedia.org/r/636974

Maintenance_bot removed a project: Patch-For-Review.Oct 28 2020, 5:10 PM
ops-monitoring-bot added a comment.Oct 30 2020, 7:01 PM

Script wmf-auto-reimage was launched by robh on cumin1001.eqiad.wmnet for hosts:

es1026.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202010301901_robh_31694_es1026_eqiad_wmnet.log.

ops-monitoring-bot added a comment.Oct 30 2020, 7:14 PM

Completed auto-reimage of hosts:

['es1026.eqiad.wmnet']

Of which those FAILED:

['es1026.eqiad.wmnet']
ops-monitoring-bot added a comment.Oct 30 2020, 7:50 PM

Script wmf-auto-reimage was launched by robh on cumin1001.eqiad.wmnet for hosts:

es1026.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202010301950_robh_13320_es1026_eqiad_wmnet.log.

ops-monitoring-bot added a comment.Oct 30 2020, 8:15 PM

Completed auto-reimage of hosts:

['es1026.eqiad.wmnet']

and were ALL successful.

RobH updated the task description. (Show Details)Oct 30 2020, 8:16 PM
ops-monitoring-bot added a comment.Oct 30 2020, 8:17 PM

Script wmf-auto-reimage was launched by robh on cumin1001.eqiad.wmnet for hosts:

['es1027.eqiad.wmnet', 'es1028.eqiad.wmnet', 'es1029.eqiad.wmnet', 'es1030.eqiad.wmnet', 'es1031.eqiad.wmnet', 'es1032.eqiad.wmnet', 'es1033.eqiad.wmnet', 'es1034.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202010302017_robh_8109.log.

ops-monitoring-bot added a comment.Oct 30 2020, 8:42 PM

Completed auto-reimage of hosts:

['es1027.eqiad.wmnet', 'es1029.eqiad.wmnet', 'es1028.eqiad.wmnet', 'es1033.eqiad.wmnet', 'es1031.eqiad.wmnet', 'es1030.eqiad.wmnet', 'es1032.eqiad.wmnet', 'es1034.eqiad.wmnet']

and were ALL successful.

RobH closed this task as Resolved.Oct 30 2020, 8:49 PM
RobH updated the task description. (Show Details)

All installations complete and hosts are calling into puppet. They've all been set to staged in netbox, and the DBA team can set them to active when they deploy them to service.

RobH removed RobH as the assignee of this task.Oct 30 2020, 8:51 PM
wiki_willy assigned this task to • Cmjohnson.Oct 30 2020, 8:56 PM

Thanks @Cmjohnson and @RobH for prioritizing this one. Nice work getting it turned over in time. Thanks, Willy

Marostegui reopened this task as Open.Oct 31 2020, 7:41 AM

es1032 has RAID0 instead of RAID10.
Can we get that one re-done with RAID10 and strip size 256?

Thanks!

root@es1032:~# megacli -LDPDInfo -aAll | grep "RAID Level"
RAID Level          : Primary-0, Secondary-0, RAID Level Qualifier-0
root@es1032:~# pvs
  PV         VG   Fmt  Attr PSize  PFree
  /dev/sda3  tank lvm2 a--  21.78t <12.69t

All the other hosts look good. from disk, memory, strip size, cpu....point of view.

RobH claimed this task.Nov 2 2020, 3:46 PM
In T260370#6593013, @Marostegui wrote:

es1032 has RAID0 instead of RAID10.
Can we get that one re-done with RAID10 and strip size 256?

Thanks!

root@es1032:~# megacli -LDPDInfo -aAll | grep "RAID Level"
RAID Level          : Primary-0, Secondary-0, RAID Level Qualifier-0
root@es1032:~# pvs
  PV         VG   Fmt  Attr PSize  PFree
  /dev/sda3  tank lvm2 a--  21.78t <12.69t

All the other hosts look good. from disk, memory, strip size, cpu....point of view.

Apologies! I set them ALL wrong and I thought I had fixed.

ops-monitoring-bot added a comment.Nov 3 2020, 12:07 AM

Script wmf-auto-reimage was launched by robh on cumin1001.eqiad.wmnet for hosts:

es1032.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202011030007_robh_23724_es1032_eqiad_wmnet.log.

ops-monitoring-bot added a comment.Nov 3 2020, 12:24 AM

Completed auto-reimage of hosts:

['es1032.eqiad.wmnet']

Of which those FAILED:

['es1032.eqiad.wmnet']
RobH added a comment.Nov 3 2020, 12:26 AM

This gets to the debian loader, and halts on 'Probing EDD' which had no issues on the other hosts. I'm still investigating on what is different about this than the other es hosts in this batch.

ops-monitoring-bot added a comment.Nov 3 2020, 12:44 AM

Script wmf-auto-reimage was launched by robh on cumin1001.eqiad.wmnet for hosts:

es1032.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202011030044_robh_30404_es1032_eqiad_wmnet.log.

ops-monitoring-bot added a comment.Nov 3 2020, 12:44 AM

Completed auto-reimage of hosts:

['es1032.eqiad.wmnet']

Of which those FAILED:

['es1032.eqiad.wmnet']
ops-monitoring-bot added a comment.Nov 3 2020, 12:45 AM

Script wmf-auto-reimage was launched by robh on cumin1001.eqiad.wmnet for hosts:

es1032.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202011030045_robh_31476_es1032_eqiad_wmnet.log.

RobH updated the task description. (Show Details)Nov 3 2020, 12:52 AM
ops-monitoring-bot added a comment.Nov 3 2020, 1:11 AM

Completed auto-reimage of hosts:

['es1032.eqiad.wmnet']

and were ALL successful.

RobH closed this task as Resolved.Nov 3 2020, 1:25 AM
In T260370#6598690, @ops-monitoring-bot wrote:

Completed auto-reimage of hosts:

['es1032.eqiad.wmnet']

and were ALL successful.

Done!

Marostegui added a comment.Nov 3 2020, 5:35 AM

Thanks Rob, es1032 looks good now:

Name                :Virtual Disk 0
RAID Level          : Primary-1, Secondary-0, RAID Level Qualifier-0
Size                : 10.913 TB
Sector Size         : 512
Is VD emulated      : No
Mirror Data         : 10.913 TB

root@es1032:~# pvs
  PV         VG   Fmt  Attr PSize   PFree
  /dev/sda3  tank lvm2 a--  <10.87t 1.77t
root@es1032:~# df -hT /srv
Filesystem            Type  Size  Used Avail Use% Mounted on
/dev/mapper/tank-data xfs   9.1T   11G  9.1T   1% /srv
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL