Task has been created, @Marostegui will take care of creating the racking strategy on January, but if for some reason you are in a hurry (we are not), it will be distributed among rows (as much as reasonable) for redundancy, not having 2 of these on the same rack, and as much as possible and reasonable, also avoiding other es10* hosts on the same rack.

I believe I already defined the racking proposal at T235659

Partitioning recipe was also already defined in puppet some days ago so we are good on that front too.

Thanks guys

These hosts have been assigned to the partman recipe and set as initial spare for the installation: https://github.com/wikimedia/puppet/blob/production/manifests/site.pp#L331 https://github.com/wikimedia/puppet/blob/production/modules/install_server/files/autoinstall/netboot.cfg#L75

You guys think this will be ready by 31st Jan?
Thanks.

When adding the MAC addresses to the DHCP file, make sure to add the following line:

option pxelinux.pathprefix "http://apt.wikimedia.org/tftpboot/stretch-installer-bootif/";

Example of how it is done in codfw:

host es2023 {
    hardware ethernet B0:26:28:F5:16:C8;
    fixed-address es2023.codfw.wmnet;
    option pxelinux.pathprefix "http://apt.wikimedia.org/tftpboot/stretch-installer-bootif/";
}

In T241359#5809112, @Marostegui wrote:

You guys think this will be ready by 31st Jan?
Thanks.

Any ETA on when we can expect these hosts to be ready?

@Marostegui I do not know yet, there are several racking/setup tasks that I am trying to get through. I need to check with @Jclark-ctr and see if they're even in racks yet.

In T241359#5827691, @Cmjohnson wrote:

@Marostegui I do not know yet, there are several racking/setup tasks that I am trying to get through. I need to check with @Jclark-ctr and see if they're even in racks yet.

If it helps, I can do the OS installation myself if DC-Ops do the switches, RAIDs, DNS and send me the MAC addresses.
Thanks!

These host are not in racks yet. I can rack these today but do not have ip`s yet so can not setup yet. @Cmjohnson if you can add ip`s to this ticket i can configure host. once i get those turn around should be quick

I see that the racking recomendation is both in 10g and 1g racks @Marostegui and A1 is a network rack

would these racks work for you?
es1020.eqiad.wmnet A3
es1021.eqiad.wmnet B3
es1022.eqiad.wmnet C5
es1023.eqiad.wmnet D6
es1024.eqiad.wmnet A5
es1025.eqiad.wmnet B5

In T241359#5830750, @Jclark-ctr wrote:

I see that the racking recomendation is both in 10g and 1g racks @Marostegui and A1 is a network rack

would these racks work for you?
es1020.eqiad.wmnet A3
es1021.eqiad.wmnet B3
es1022.eqiad.wmnet C5
es1023.eqiad.wmnet D6
es1024.eqiad.wmnet A5
es1025.eqiad.wmnet B5

Those racks work for me.

These hosts have 1G and 10G interfaces, but we will only be using 1G for the time being.

Change 568577 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/dns@master] adding mgmt dns for es1019-1025

https://gerrit.wikimedia.org/r/568577

Change 568577 merged by Cmjohnson:
[operations/dns@master] adding mgmt dns for es1019-1025

https://gerrit.wikimedia.org/r/568577

updated mgmt dns

+es1020 1H IN A 10.65.4.144
+es1021 1H IN A 10.65.4.145
+es1022 1H IN A 10.65.4.146
+es1023 1H IN A 10.65.4.147
+es1024 1H IN A 10.65.4.148
+es1025 1H IN A 10.65.4.149

@Jclark-ctr @wiki_willy @Cmjohnson any rough estimation on when we can expect these hosts to be online?
As I said, I am happy to do the OS installation myself once you've done the racking/BIOS/RAID + idrac + switches steps. Just send me the MACs addresses and I will take care of the puppet stuff and OS installs
Reminder, RAID10 (256k stripsize)

Thanks!

Change 572327 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/dns@master] Adding production dns for new es servers

https://gerrit.wikimedia.org/r/572327

dns has been updated, I need to verify the network ports. I have a conflict with es1025...that port shows utilized (@Jclark-ctr)

Change 572327 merged by Cmjohnson:
[operations/dns@master] Adding production dns for new es servers

https://gerrit.wikimedia.org/r/572327

Change 572953 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/puppet@production] Adding macs for es102[0-5] to dhcpd file

https://gerrit.wikimedia.org/r/572953

Change 572953 merged by Cmjohnson:
[operations/puppet@production] Adding macs for es102[0-5] to dhcpd file

https://gerrit.wikimedia.org/r/572953

Change 572970 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/puppet@production] updating dhpd file for es102[0-5]. used the wrong eth port

https://gerrit.wikimedia.org/r/572970

Change 572970 merged by Cmjohnson:
[operations/puppet@production] updating dhpd file for es102[0-5]. used the wrong eth port

https://gerrit.wikimedia.org/r/572970

Change 573069 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] install_server: Pass bootif installer to new ES hosts

https://gerrit.wikimedia.org/r/573069

Change 573069 merged by Marostegui:
[operations/puppet@production] install_server: Pass bootif installer to new ES hosts

https://gerrit.wikimedia.org/r/573069

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['es1020.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202002190823_marostegui_56932.log.

Completed auto-reimage of hosts:

['es1020.eqiad.wmnet']

Of which those FAILED:

['es1020.eqiad.wmnet']

In T241359#5896178, @ops-monitoring-bot wrote:

Completed auto-reimage of hosts:

['es1020.eqiad.wmnet']

Of which those FAILED:

['es1020.eqiad.wmnet']

This is due to the new Debian 9.12 point release, Moritz will rebuild the image.

Mentioned in SAL (#wikimedia-operations) [2020-02-19T10:09:57Z] <moritzm> updated tftpboot environment for stretch-bootif for the 9.12 point release T241359

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['es1020.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202002191011_marostegui_76403.log.

Completed auto-reimage of hosts:

['es1020.eqiad.wmnet']

and were ALL successful.

es1020 installed correctly: RAID10, 256k strip size, BBU and Cache policy right disk space, memory and cpus

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['es1021.eqiad.wmnet', 'es1022.eqiad.wmnet', 'es1023.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202002191045_marostegui_82897.log.

Completed auto-reimage of hosts:

['es1022.eqiad.wmnet', 'es1023.eqiad.wmnet', 'es1021.eqiad.wmnet']

and were ALL successful.

In T241359#5896474, @ops-monitoring-bot wrote:

Completed auto-reimage of hosts:

['es1022.eqiad.wmnet', 'es1023.eqiad.wmnet', 'es1021.eqiad.wmnet']

and were ALL successful.

RAID10, 256k strip size, BBU and Cache policy right disk space, memory and cpus looking good.

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['es1024.eqiad.wmnet', 'es1025.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202002191128_marostegui_94337.log.

Completed auto-reimage of hosts:

['es1024.eqiad.wmnet']

Of which those FAILED:

['es1024.eqiad.wmnet']

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['es1024.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202002191402_marostegui_125718.log.

es1025: RAID10, 256k strip size, BBU and Cache policy right disk space, memory and cpus looking good.

@Cmjohnson can you double check es1024's link?
It cannot PXE boot:

Booting from PXE Device 1: Integrated NIC 1 Port 1 Partition 1
PXE: No media detected.
Boot Failed: PXE Device 1: Integrated NIC 1 Port 1 Partition 1

Booting from PXE Device 1: Integrated NIC 1 Port 1 Partition 1
PXE: No media detected.
Boot Failed: PXE Device 1: Integrated NIC 1 Port 1 Partition 1

No boot device available or Operating System detected.
Please ensure a compatible bootable media is available.

Available Actions:
F1 to Continue and Retry Boot Order
F2 for System Setup (BIOS)
F10 for Lifecycle Controller
- Enable/Configure iDRAC
- Update or Backup/Restore Server Firmware
- Help Install an Operating System
F11 for Boot Manager

Completed auto-reimage of hosts:

['es1024.eqiad.wmnet']

and were ALL successful.

es1024: RAID10, 256k strip size, BBU and Cache policy right disk space, memory and cpus looking good.