Page MenuHomePhabricator

(Need By: TBD) rack/setup/install mw14[14-56]
Closed, ResolvedPublic

Description

This task will track the racking, setup, and OS installation of mw14[14-56]

Hostname / Racking / Installation Details

Hostnames: mw14[14-56] (43 hosts)
Racking Proposal: Even distribution across all four rows, these are replacing mw[1261-1290,1293-1306] and ideally match those hosts fairly closely.
Networking/Subnet/VLAN/IP: 1g, internal vlan, single production network + mgmt
Partitioning/Raid: existing mw recipe
OS Distro: existing mw systems are currently stretch

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

mw1414:

  • - receive in system on procurement task T271155 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - firmware update (bios 2.11.2, idrac 5.00.00.00 , network 21.80.9)
  • - operations/puppet update - https://gerrit.wikimedia.org/r/c/operations/puppet/+/701181
  • - OS installation & initital puppet run via wmf-auto-reimage or wmf-auto-reimage-host
  • - host state in netbox set to staged

mw1415:

  • - receive in system on procurement task T271155 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - firmware update (bios 2.11.2, idrac 5.00.00.00 , network 21.80.9)
  • - operations/puppet update - https://gerrit.wikimedia.org/r/c/operations/puppet/+/701181
  • - OS installation & initital puppet run via wmf-auto-reimage or wmf-auto-reimage-host
  • - host state in netbox set to staged

mw1416:

  • - receive in system on procurement task T271155 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - firmware update (bios 2.11.2, idrac 5.00.00.00 , network 21.80.9)
  • - operations/puppet update - https://gerrit.wikimedia.org/r/c/operations/puppet/+/701181
  • - OS installation & initital puppet run via wmf-auto-reimage or wmf-auto-reimage-host
  • - host state in netbox set to staged

mw1417:

  • - receive in system on procurement task T271155 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - firmware update (bios 2.11.2, idrac 5.00.00.00 , network 21.80.9)
  • - operations/puppet update - https://gerrit.wikimedia.org/r/c/operations/puppet/+/701181
  • - OS installation & initital puppet run via wmf-auto-reimage or wmf-auto-reimage-host
  • - host state in netbox set to staged

mw1418:

  • - receive in system on procurement task T271155 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - firmware update (bios 2.11.2, idrac 5.00.00.00 , network 21.80.9)
  • - operations/puppet update - https://gerrit.wikimedia.org/r/c/operations/puppet/+/701181
  • - OS installation & initital puppet run via wmf-auto-reimage or wmf-auto-reimage-host
  • - host state in netbox set to staged

mw1419:

  • - receive in system on procurement task T271155 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - firmware update (bios 2.11.2, idrac 5.00.00.00 , network 21.80.9)
  • - operations/puppet update - https://gerrit.wikimedia.org/r/c/operations/puppet/+/701181
  • - OS installation & initital puppet run via wmf-auto-reimage or wmf-auto-reimage-host
  • - host state in netbox set to staged

mw1420:

  • - receive in system on procurement task T271155 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - firmware update (bios 2.11.2, idrac 5.00.00.00 , network 21.80.9)
  • - operations/puppet update - https://gerrit.wikimedia.org/r/c/operations/puppet/+/701181
  • - OS installation & initital puppet run via wmf-auto-reimage or wmf-auto-reimage-host
  • - host state in netbox set to staged

mw1421:

  • - receive in system on procurement task T271155 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - firmware update (bios 2.11.2, idrac 5.00.00.00 , network 21.80.9)
  • - operations/puppet update - https://gerrit.wikimedia.org/r/c/operations/puppet/+/701181
  • - OS installation & initital puppet run via wmf-auto-reimage or wmf-auto-reimage-host
  • - host state in netbox set to staged

mw1422:

  • - receive in system on procurement task T271155 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - firmware update - idrac done - bios and network pending
  • - firmware update (bios 2.11.2, idrac 5.00.00.00 , network 21.80.9)
  • - operations/puppet update - https://gerrit.wikimedia.org/r/c/operations/puppet/+/701181
  • - host state in netbox set to staged

mw1423:

  • - receive in system on procurement task T271155 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - firmware update (bios 2.11.2, idrac 5.00.00.00 , network 21.80.9)
  • - operations/puppet update - https://gerrit.wikimedia.org/r/c/operations/puppet/+/701181
  • - OS installation & initital puppet run via wmf-auto-reimage or wmf-auto-reimage-host
  • - host state in netbox set to staged

mw1424:

  • - receive in system on procurement task T271155 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - firmware update (bios 2.11.2, idrac 5.00.00.00 , network 21.80.9)
  • - operations/puppet update - https://gerrit.wikimedia.org/r/c/operations/puppet/+/701181
  • - OS installation & initital puppet run via wmf-auto-reimage or wmf-auto-reimage-host
  • - host state in netbox set to staged

mw1425:

  • - receive in system on procurement task T271155 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - firmware update (bios 2.11.2, idrac 5.00.00.00 , network 21.80.9)
  • - operations/puppet update - this should include updates to install_server dhcp and netboot, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - operations/puppet update - https://gerrit.wikimedia.org/r/c/operations/puppet/+/701181
  • - host state in netbox set to staged

mw1426:

  • - receive in system on procurement task T271155 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - firmware update (bios 2.11.2, idrac 5.00.00.00 , network 21.80.9)
  • - operations/puppet update - this should include updates to install_server dhcp and netboot, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - operations/puppet update - https://gerrit.wikimedia.org/r/c/operations/puppet/+/701181
  • - host state in netbox set to staged

mw1427:

  • - receive in system on procurement task T271155 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - firmware update (bios 2.11.2, idrac 5.00.00.00 , network 21.80.9)
  • - operations/puppet update - https://gerrit.wikimedia.org/r/c/operations/puppet/+/701181
  • - OS installation & initital puppet run via wmf-auto-reimage or wmf-auto-reimage-host
  • - host state in netbox set to staged

mw1428:

  • - receive in system on procurement task T271155 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - firmware update (bios 2.11.2, idrac 5.00.00.00 , network 21.80.9)
  • - operations/puppet update - https://gerrit.wikimedia.org/r/c/operations/puppet/+/701181
  • - OS installation & initital puppet run via wmf-auto-reimage or wmf-auto-reimage-host
  • - host state in netbox set to staged

mw1429:

  • - receive in system on procurement task T271155 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - firmware update (bios 2.11.2, idrac 5.00.00.00 , network 21.80.9)
  • - operations/puppet update - https://gerrit.wikimedia.org/r/c/operations/puppet/+/701181
  • - OS installation & initital puppet run via wmf-auto-reimage or wmf-auto-reimage-host
  • - host state in netbox set to staged

mw1430:

  • - receive in system on procurement task T271155 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - firmware update (bios 2.11.2, idrac 5.00.00.00 , network 21.80.9)
  • - operations/puppet update - https://gerrit.wikimedia.org/r/c/operations/puppet/+/701181
  • - OS installation & initital puppet run via wmf-auto-reimage or wmf-auto-reimage-host
  • - host state in netbox set to staged

mw1431:

  • - receive in system on procurement task T271155 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - firmware update (bios 2.11.2, idrac 5.00.00.00 , network 21.80.9)
  • - operations/puppet update - https://gerrit.wikimedia.org/r/c/operations/puppet/+/701181
  • - OS installation & initital puppet run via wmf-auto-reimage or wmf-auto-reimage-host
  • - host state in netbox set to staged

mw1432:

  • - receive in system on procurement task T271155 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - firmware update (bios 2.11.2, idrac 5.00.00.00 , network 21.80.9)
  • - operations/puppet update - https://gerrit.wikimedia.org/r/c/operations/puppet/+/701181
  • - OS installation & initital puppet run via wmf-auto-reimage or wmf-auto-reimage-host
  • - host state in netbox set to staged

mw1433:

  • - receive in system on procurement task T271155 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - firmware update (bios 2.11.2, idrac 5.00.00.00 , network 21.80.9)
  • - operations/puppet update - https://gerrit.wikimedia.org/r/c/operations/puppet/+/701181
  • - OS installation & initital puppet run via wmf-auto-reimage or wmf-auto-reimage-host
  • - host state in netbox set to staged

mw1434:

  • - receive in system on procurement task T271155 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - firmware update (bios 2.11.2, idrac 5.00.00.00 , network 21.80.9)
  • - operations/puppet update - https://gerrit.wikimedia.org/r/c/operations/puppet/+/701181
  • - OS installation & initital puppet run via wmf-auto-reimage or wmf-auto-reimage-host
  • - host state in netbox set to staged

mw1435:

  • - receive in system on procurement task T271155 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - firmware update (bios 2.11.2, idrac 5.00.00.00 , network 21.80.9)
  • - operations/puppet update - https://gerrit.wikimedia.org/r/c/operations/puppet/+/701181
  • - OS installation & initital puppet run via wmf-auto-reimage or wmf-auto-reimage-host
  • - host state in netbox set to staged

mw1436:

  • - receive in system on procurement task T271155 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - firmware update (bios 2.11.2, idrac 5.00.00.00 , network 21.80.9)
  • - operations/puppet update - https://gerrit.wikimedia.org/r/c/operations/puppet/+/701181
  • - OS installation & initital puppet run via wmf-auto-reimage or wmf-auto-reimage-host
  • - host state in netbox set to staged

mw1437:

  • - receive in system on procurement task T271155 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - firmware update (bios 2.11.2, idrac 5.00.00.00 , network 21.80.9)
  • - operations/puppet update - https://gerrit.wikimedia.org/r/c/operations/puppet/+/701181
  • - OS installation & initital puppet run via wmf-auto-reimage or wmf-auto-reimage-host
  • - host state in netbox set to staged

mw1438:

  • - receive in system on procurement task T271155 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - firmware update (bios 2.11.2, idrac 5.00.00.00 , network 21.80.9)
  • - operations/puppet update - https://gerrit.wikimedia.org/r/c/operations/puppet/+/701181
  • - OS installation & initital puppet run via wmf-auto-reimage or wmf-auto-reimage-host
  • - host state in netbox set to staged

mw1439:

  • - receive in system on procurement task T271155 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - firmware update (bios 2.11.2, idrac 5.00.00.00 , network 21.80.9)
  • - operations/puppet update - https://gerrit.wikimedia.org/r/c/operations/puppet/+/701181
  • - OS installation & initital puppet run via wmf-auto-reimage or wmf-auto-reimage-host
  • - host state in netbox set to staged

mw1440:

  • - receive in system on procurement task T271155 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - firmware update (bios 2.11.2, idrac 5.00.00.00 , network 21.80.9)
  • - operations/puppet update - https://gerrit.wikimedia.org/r/c/operations/puppet/+/701181
  • - OS installation & initital puppet run via wmf-auto-reimage or wmf-auto-reimage-host
  • - host state in netbox set to staged

mw1441:

  • - receive in system on procurement task T271155 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - firmware update (bios 2.11.2, idrac 5.00.00.00 , network 21.80.9)
  • - operations/puppet update - https://gerrit.wikimedia.org/r/c/operations/puppet/+/701181
  • - OS installation & initital puppet run via wmf-auto-reimage or wmf-auto-reimage-host
  • - host state in netbox set to staged

mw1442:

  • - receive in system on procurement task T271155 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - firmware update (bios 2.11.2, idrac 5.00.00.00 , network 21.80.9)
  • - operations/puppet update - https://gerrit.wikimedia.org/r/c/operations/puppet/+/701181
  • - OS installation & initital puppet run via wmf-auto-reimage or wmf-auto-reimage-host
  • - host state in netbox set to staged

mw1443:

  • - receive in system on procurement task T271155 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - firmware update (bios 2.11.2, idrac 5.00.00.00 , network 21.80.9)
  • - operations/puppet update - https://gerrit.wikimedia.org/r/c/operations/puppet/+/701181
  • - OS installation & initital puppet run via wmf-auto-reimage or wmf-auto-reimage-host
  • - host state in netbox set to staged

mw1444:

  • - receive in system on procurement task T271155 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - firmware update (bios 2.11.2, idrac 5.00.00.00 , network 21.80.9)
  • - operations/puppet update - https://gerrit.wikimedia.org/r/c/operations/puppet/+/701181
  • - OS installation & initital puppet run via wmf-auto-reimage or wmf-auto-reimage-host
  • - host state in netbox set to staged / active

mw1445:

  • - receive in system on procurement task T271155 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - firmware update (bios 2.11.2, idrac 5.00.00.00 , network 21.80.9)
  • - operations/puppet update - this should include updates to install_server dhcp and netboot, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - operations/puppet update - https://gerrit.wikimedia.org/r/c/operations/puppet/+/701181
  • - host state in netbox set to staged

mw1446:

  • - receive in system on procurement task T271155 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - firmware update (bios 2.11.2, idrac 5.00.00.00 , network 21.80.9)
  • - operations/puppet update - https://gerrit.wikimedia.org/r/c/operations/puppet/+/701181
  • - OS installation & initital puppet run via wmf-auto-reimage or wmf-auto-reimage-host
  • - host state in netbox set to staged

mw1447:

  • - receive in system on procurement task T271155 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - firmware update (bios 2.11.2, idrac 5.00.00.00 , network 21.80.9)
  • - operations/puppet update - https://gerrit.wikimedia.org/r/c/operations/puppet/+/701181
  • - OS installation & initital puppet run via wmf-auto-reimage or wmf-auto-reimage-host
  • - host state in netbox set to staged

mw1448:

  • - receive in system on procurement task T271155 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - firmware update (bios, idrac, network)
  • - operations/puppet update - this should include updates to install_server dhcp and netboot, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via wmf-auto-reimage or wmf-auto-reimage-host
  • - host state in netbox set to staged

mw1449:

  • - receive in system on procurement task T271155 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - firmware update (bios, idrac, network)
  • - operations/puppet update - this should include updates to install_server dhcp and netboot, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via wmf-auto-reimage or wmf-auto-reimage-host
  • - host state in netbox set to staged

mw1450:

  • - receive in system on procurement task T271155 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - firmware update (bios, idrac, network)
  • - operations/puppet update - this should include updates to install_server dhcp and netboot, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via wmf-auto-reimage or wmf-auto-reimage-host
  • - host state in netbox set to staged

mw1451:

  • - receive in system on procurement task T271155 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - firmware update (bios, idrac, network)
  • - operations/puppet update - this should include updates to install_server dhcp and netboot, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via wmf-auto-reimage or wmf-auto-reimage-host
  • - host state in netbox set to staged

mw1452:

  • - receive in system on procurement task T271155 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - firmware update (bios, idrac, network)
  • - operations/puppet update - this should include updates to install_server dhcp and netboot, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via wmf-auto-reimage or wmf-auto-reimage-host
  • - host state in netbox set to staged

mw1453:

  • - receive in system on procurement task T271155 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - firmware update (bios, idrac, network)
  • - operations/puppet update - this should include updates to install_server dhcp and netboot, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via wmf-auto-reimage or wmf-auto-reimage-host
  • - host state in netbox set to staged

mw1454:

  • - receive in system on procurement task T271155 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - firmware update (bios, idrac, network)
  • - operations/puppet update - this should include updates to install_server dhcp and netboot, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via wmf-auto-reimage or wmf-auto-reimage-host
  • - host state in netbox set to staged

mw1455:

  • - receive in system on procurement task T271155 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - firmware update (bios, idrac, network)
  • - operations/puppet update - this should include updates to install_server dhcp and netboot, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via wmf-auto-reimage or wmf-auto-reimage-host
  • - host state in netbox set to staged

mw1456:

  • - receive in system on procurement task T271155 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - firmware update (bios, idrac, network)
  • - operations/puppet update - this should include updates to install_server dhcp and netboot, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via wmf-auto-reimage or wmf-auto-reimage-host
  • - host state in netbox set to staged

Once the system(s) above have had all checkbox steps completed, this task can be resolved.

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

mw1414-,mw1422
racked, dns updated and homer ran. BIOS/Idrac is not setup yet

@RobH mw1414-mw1422 in rack A3 are install ready, need password changed.

@RobH mw1423-1447 are ready for you now as well

RobH updated the task description. (Show Details)

@Cmjohnson,

Please review and test the following servers, as their mgmt is offline. This can be caused by the cable not being properly seated, idrac misconfiguration, or hardware failure (listed in order of most to least probable.)

mw1414
mw1415
mw1416
mw1417
mw1418
mw1419
mw1420
mw1421
mw1422

I've updated the task description with full checklists, please check the mgmt and ensure it works for the above servers (the are unchecked for bios/drac testing in checklist). I'm assigning this back to you, but I'll keep working on the other systems in the group in the meantime.

@Cmjohnson,

Please review and test the following servers, as their mgmt is offline. This can be caused by the cable not being properly seated, idrac misconfiguration, or hardware failure (listed in order of most to least probable.)

mw1414
mw1415
mw1416
mw1417
mw1418
mw1419
mw1420
mw1421
mw1422

I've updated the task description with full checklists, please check the mgmt and ensure it works for the above servers (the are unchecked for bios/drac testing in checklist). I'm assigning this back to you, but I'll keep working on the other systems in the group in the meantime.

So the remainder of the hosts above mw1447 are also pending racking it seems, so those need to be racked and updated before installation as well. I'm not sure if this should be assigned to Chris or to John for these corrections, but these aren't ready for installation. I've spent the day troubleshooting remote command line loading of firmware, and have gotten it to work (which much assistance) for idrac and updated the per server checklist to reflect which hosts have had idrac firmware updated.

@RobH mw1414-1422 were missing the mgmt cables. Fixed and they're good to go. it appears John racked the others, I will add that to my list.

Change 701181 had a related patch set uploaded (by RobH; author: RobH):

[operations/puppet@production] install params for mw14[14-47]

https://gerrit.wikimedia.org/r/701181

Change 701181 merged by RobH:

[operations/puppet@production] install params for mw14[14-47]

https://gerrit.wikimedia.org/r/701181

Script wmf-auto-reimage was launched by robh on cumin1001.eqiad.wmnet for hosts:

mw1414.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202106232211_robh_373_mw1414_eqiad_wmnet.log.

Completed auto-reimage of hosts:

['mw1414.eqiad.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by robh on cumin1001.eqiad.wmnet for hosts:

['mw1415.eqiad.wmnet', 'mw1416.eqiad.wmnet', 'mw1417.eqiad.wmnet', 'mw1418.eqiad.wmnet', 'mw1419.eqiad.wmnet', 'mw1420.eqiad.wmnet', 'mw1421.eqiad.wmnet', 'mw1422.eqiad.wmnet', 'mw1423.eqiad.wmnet', 'mw1424.eqiad.wmnet', 'mw1425.eqiad.wmnet', 'mw1426.eqiad.wmnet', 'mw1427.eqiad.wmnet', 'mw1428.eqiad.wmnet', 'mw1429.eqiad.wmnet', 'mw1430.eqiad.wmnet', 'mw1431.eqiad.wmnet', 'mw1432.eqiad.wmnet', 'mw1433.eqiad.wmnet', 'mw1434.eqiad.wmnet', 'mw1435.eqiad.wmnet', 'mw1436.eqiad.wmnet', 'mw1437.eqiad.wmnet', 'mw1438.eqiad.wmnet', 'mw1439.eqiad.wmnet', 'mw1440.eqiad.wmnet', 'mw1441.eqiad.wmnet', 'mw1442.eqiad.wmnet', 'mw1443.eqiad.wmnet', 'mw1444.eqiad.wmnet', 'mw1445.eqiad.wmnet', 'mw1446.eqiad.wmnet', 'mw1447.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202106232255_robh_7267.log.

Completed auto-reimage of hosts:

['mw1444.eqiad.wmnet']

Of which those FAILED:

['mw1444.eqiad.wmnet']

Script wmf-auto-reimage was launched by robh on cumin1001.eqiad.wmnet for hosts:

mw1444.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202106241502_robh_29880_mw1444_eqiad_wmnet.log.

Completed auto-reimage of hosts:

['mw1444.eqiad.wmnet']

Of which those FAILED:

['mw1444.eqiad.wmnet']
This comment was removed by Cmjohnson.

@Jclark-ctr please cable mw1448-mw1458, and update netbox and task with switch port info.

In case it helps here, today we shut down 6 servers in A5 (T280203#7190053), you can replace those with new servers.

we are now about to get to the range mw1448 through mw1450 which are already in rack but not in DNS yet. Could you do these next?

mw1444 is in DNS but not reachable via SSH (1443, 1445, 1446 are), could you take a look what's special with 1444 please

all decoms in T280203 are done except 4 canary API server and those are now blocked by this ticket

Dzahn raised the priority of this task from Medium to High.Aug 2 2021, 12:06 PM

mw1448 D8 U Port31 Cableid#23000004
mw1449 D8 U Port32 Cableid#23000005
mw1450 D8 U Port33 Cableid#23000013
mw1451 A1 U8 Port20 Cableid#23000030
mw1452 A1 U22 Port21 Cableid#23000022
mw1453 A8 U4 Port17 Cableid#23000037
mw1454 A8 U5 Port18 Cableid#23000041
mw1455 A8 U6 Port19 Cableid#23000008
mw1456 A8 U7 Port20 Cableid#23000031

@Jclark-ctr We already have quite a few more servers in row D than in A, B or C, with A having the smallest number. Would it be possible to put all of these in A (any rack) for balancing? Thanks!

@Dzahn we have no more spaces in row A for any host

@Jclark-ctr ACK! though in T280203 we have decom'ed about 20 servers in A that are completely out of production yet still have to be removed from the rack. Could we replace old mw servers with new mw servers there?

@Dzahn that can be done but it will delay the last 3 mw servers until we can get time to remove the decom servers and then unracking and de-cabling the 3 servers and re-racking them in row A and re-cabling. I purposely did not do any dns/port changes on mw1448-mw1450 until this can happen.

Thank you @Cmjohnson we would also be happy with getting thew servers in production now and later move a few of them in a separate action, if that isn't making it harder for you guys.

@Dzahn the on-site work is complete for all of the servers, I moved mw1448-1450 to rack A5. I swapped the network cable for mw1444.

Thanks a lot @Cmjohnson ! I will continue getting them into production now.

Hi @Cmjohnson regarding mw1444, I still could not ssh to the server but I could ssh to mgmt and saw it is currently in an endless loop trying to PXE boot but PXE boot fails with "PXE-E61: Media test failure,0check cable. I guess the replaced cable might also be bad (or the NIC or switch port)? Could you check on that one again? Thanks

Hi again @Cmjohnson regarding mw1448 through mw1457: I see they are now in DNS but I could not ssh to them and it appears the mgmt password is not set yet to the standard one so I could not login. In netbox they are still in state "planned". Could you please check the mgmt password is set? And move them from planned to staging in netbox? Thanks!

@Dzahn no worries, the on-stie work is done but needs firmware updates and the passwords reset. I'll have these for you NLT tomorrow. Regarding mw1444, the issue is fixed. The cable was not connected to the correct port.

All the firmware has been updated and the mgmt password set

Mentioned in SAL (#wikimedia-operations) [2021-08-13T09:42:23Z] <mutante> mw1448, mw1449, mw1450 - powering on via mgmt - OS install, initial setup (T279309, T273915)

Change 712928 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] DHCP: add MAC addresses for mw1448 through mw1456

https://gerrit.wikimedia.org/r/712928

Change 712928 merged by Dzahn:

[operations/puppet@production] DHCP: add MAC addresses for mw1448 through mw1456

https://gerrit.wikimedia.org/r/712928

Mentioned in SAL (#wikimedia-operations) [2021-08-13T11:11:35Z] <jelto> mw1455 - powering on via mgmt - OS install, initial setup (T279309, T273915)

Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts:

['mw1450.eqiad.wmnet', 'mw1451.eqiad.wmnet', 'mw1452.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202108131213_dzahn_18061.log.

Completed auto-reimage of hosts:

['mw1450.eqiad.wmnet', 'mw1451.eqiad.wmnet', 'mw1452.eqiad.wmnet']

and were ALL successful.

@Dzahn no worries, the on-stie work is done but needs firmware updates and the passwords reset. I'll have these for you NLT tomorrow. Regarding mw1444, the issue is fixed. The cable was not connected to the correct port.

Thanks @Cmjohnson . mw1444 is now in production as API server (set to active in netbox).

For mw1448 through mw1456 I have pulled the MAC addresses and added them to DHCP, then we installed the OS.

mw1453 seems to be a special case. Unlike the other hosts it would not reboot when the cookbook tries to reboot it and manually restarting also seemed to be stuck. Maybe a problem with IPMI or hardware.

Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts:

mw1453.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202108132022_cmjohnson_24039_mw1453_eqiad_wmnet.log.

Completed auto-reimage of hosts:

['mw1453.eqiad.wmnet']

Of which those FAILED:

['mw1453.eqiad.wmnet']

Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts:

['mw1455.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202108161351_dzahn_5691.log.

Mentioned in SAL (#wikimedia-operations) [2021-08-16T13:53:30Z] <mutante> mw1455 - mysteriously showing a bunch of issues in icinga, broken packages, envoy, memcached etc, after recent fresh install, trying another reimage (T273915)

Completed auto-reimage of hosts:

['mw1455.eqiad.wmnet']

and were ALL successful.

Change 713293 had a related patch set uploaded (by Cmjohnson; author: Cmjohnson):

[operations/puppet@production] Fixing dhcpd entry for mw1453

https://gerrit.wikimedia.org/r/713293

Change 713293 merged by Cmjohnson:

[operations/puppet@production] Fixing dhcpd entry for mw1453

https://gerrit.wikimedia.org/r/713293

mw1455 had issues but is now fine after simply repeating the cookbook one more time. (don't know why)

mw1453 had the wrong MAC address (my bad, thanks Chris for fixing it)

Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts:

mw1453.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202108161638_cmjohnson_24603_mw1453_eqiad_wmnet.log.

Completed auto-reimage of hosts:

['mw1453.eqiad.wmnet']

Of which those FAILED:

['mw1453.eqiad.wmnet']

Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts:

mw1453.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202108161735_cmjohnson_28333_mw1453_eqiad_wmnet.log.

Completed auto-reimage of hosts:

['mw1453.eqiad.wmnet']

and were ALL successful.

Cmjohnson updated the task description. (Show Details)

@Dzahn mw1453 is installed and ready for you, the mac address was off in the dhcpd file.

Thanks @Cmjohnson . just had to reimage mw1456 and now everything is done and in production (active in netbox)

Change 767787 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/puppet@production] scap: Switch mw1306 to mw1318 for scap proxy role

https://gerrit.wikimedia.org/r/767787

Change 767788 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/puppet@production] mw130[2-6]: Remove and decomission

https://gerrit.wikimedia.org/r/767788

Change 767787 merged by Alexandros Kosiaris:

[operations/puppet@production] scap: Switch mw1306 to mw1318 for scap proxy role

https://gerrit.wikimedia.org/r/767787

Change 767788 merged by Alexandros Kosiaris:

[operations/puppet@production] mw130[2-6]: Remove and decomission

https://gerrit.wikimedia.org/r/767788