Page MenuHomePhabricator
Create Task
Maniphest T290192

Q1:(Need By: TBD) rack/setup/install mw241[2-9].codfw.wmnet
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of mw241[2-9].codfw.wmnet

Hostname / Racking / Installation Details

Hostnames: mw241[2-9].codfw.wmnet
Racking Proposal: C3-codfw
Networking/Subnet/VLAN/IP: 1G, internal vlan, single connection
Partitioning/Raid: sw raid, standard, raid1-2dev
OS Distro: Buster (default unless otherwise specified)

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

mw2412:

  • - receive in system on procurement task T286516 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to install_server dhcp and netboot, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via wmf-auto-reimage or wmf-auto-reimage-host
  • - host state in netbox set to staged

mw2413:

  • - receive in system on procurement task T286516 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to install_server dhcp and netboot, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via wmf-auto-reimage or wmf-auto-reimage-host
  • - host state in netbox set to staged

mw2414:

  • - receive in system on procurement task T286516 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to install_server dhcp and netboot, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via wmf-auto-reimage or wmf-auto-reimage-host
  • - host state in netbox set to staged

mw2415:

  • - receive in system on procurement task T286516 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to install_server dhcp and netboot, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via wmf-auto-reimage or wmf-auto-reimage-host
  • - host state in netbox set to staged

mw2416:

  • - receive in system on procurement task T286516 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to install_server dhcp and netboot, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via wmf-auto-reimage or wmf-auto-reimage-host
  • - host state in netbox set to staged

mw2417:

  • - receive in system on procurement task T286516 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to install_server dhcp and netboot, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via wmf-auto-reimage or wmf-auto-reimage-host
  • - host state in netbox set to staged

mw2418:

  • - receive in system on procurement task T286516 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to install_server dhcp and netboot, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via wmf-auto-reimage or wmf-auto-reimage-host
  • - host state in netbox set to staged

mw2419:

  • - receive in system on procurement task T286516 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to install_server dhcp and netboot, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via wmf-auto-reimage or wmf-auto-reimage-host
  • - host state in netbox set to staged

Once the system(s) above have had all checkbox steps completed, this task can be resolved.

Details

SubjectRepoBranchLines +/-
conftool-date: add mw2412 through mw2419 as new appserversoperations/puppetproduction+8 -0
site: use appserver in codfw C3, cleanup duplicate insetup definitionoperations/puppetproduction+10 -8
Add mw24[12-19] MAC address and to site.ppoperations/puppetproduction+46 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

RobH created this task.Sep 1 2021, 6:40 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 1 2021, 6:40 PM
RobH renamed this task from (Need By: TBD) rack/setup/install mw241[2-9].codfw.wmnet to Q1:(Need By: TBD) rack/setup/install mw241[2-9].codfw.wmnet.Sep 1 2021, 6:40 PM
RobH assigned this task to Papaul.
RobH added a parent task: Unknown Object (Task).
RobH moved this task from Backlog to Racking Tasks on the ops-codfw board.
RobH unsubscribed.
RobH mentioned this in Unknown Object (Task).Sep 1 2021, 6:44 PM
Maintenance_bot added a project: SRE.Sep 1 2021, 6:45 PM
Papaul updated the task description. (Show Details)Sep 27 2021, 11:22 PM
gerritbot added a comment.Sep 28 2021, 12:59 PM

Change 724415 had a related patch set uploaded (by Papaul; author: Papaul):

[operations/puppet@production] Add mw24[12-19] MAC address and to site.pp

https://gerrit.wikimedia.org/r/724415

gerritbot added a project: Patch-For-Review.Sep 28 2021, 12:59 PM
gerritbot added a comment.Sep 28 2021, 1:16 PM

Change 724415 merged by Papaul:

[operations/puppet@production] Add mw24[12-19] MAC address and to site.pp

https://gerrit.wikimedia.org/r/724415

Maintenance_bot removed a project: Patch-For-Review.Sep 28 2021, 2:10 PM
ops-monitoring-bot added a comment.Sep 28 2021, 3:53 PM

Cookbook cookbooks.sre.experimental.reimage was started by pt1979@cumin2002 for host mw2412.codfw.wmnet

ops-monitoring-bot added a comment.Sep 28 2021, 4:10 PM

Cookbook cookbooks.sre.experimental.reimage executed with errors:

  • mw2412 (FAIL)
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Sep 28 2021, 4:46 PM

Cookbook cookbooks.sre.experimental.reimage was started by pt1979@cumin2002 for host mw2412.codfw.wmnet

ops-monitoring-bot added a comment.Sep 28 2021, 5:00 PM

Cookbook cookbooks.sre.experimental.reimage completed:

  • mw2412 (WARN)
    • Downtimed on Icinga
    • Unable to disable Puppet, the host may have been unreachable
    • Removed from Puppet and PuppetDB
    • Removed from Debmonitor
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/wmf-auto-reimage/202109281645_pt1979_236935_mw2412.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
ops-monitoring-bot added a comment.Sep 28 2021, 5:04 PM

Cookbook cookbooks.sre.experimental.reimage was started by pt1979@cumin2002 for host mw2413.codfw.wmnet

Papaul updated the task description. (Show Details)Sep 28 2021, 5:06 PM
ops-monitoring-bot added a comment.Sep 28 2021, 5:23 PM

Cookbook cookbooks.sre.experimental.reimage executed with errors:

  • mw2413 (FAIL)
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • The reimage failed, see the cookbook logs for the details
Papaul added a subscriber: Volans.Sep 28 2021, 5:30 PM

@Volans mw2413 failed with the same error

ops-monitoring-bot added a comment.Sep 28 2021, 5:35 PM

Cookbook cookbooks.sre.experimental.reimage was started by pt1979@cumin2002 for host mw2413.codfw.wmnet

ops-monitoring-bot added a comment.Sep 28 2021, 5:50 PM

Cookbook cookbooks.sre.experimental.reimage completed:

  • mw2413 (WARN)
    • Downtimed on Icinga
    • Unable to disable Puppet, the host may have been unreachable
    • Removed from Puppet and PuppetDB
    • Removed from Debmonitor
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/wmf-auto-reimage/202109281735_pt1979_244529_mw2413.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
Papaul updated the task description. (Show Details)Sep 28 2021, 6:04 PM
ops-monitoring-bot added a comment.Sep 28 2021, 9:34 PM

Script wmf-auto-reimage was launched by pt1979 on cumin2002.codfw.wmnet for hosts:

mw2414.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202109282134_pt1979_274485_mw2414_codfw_wmnet.log.

ops-monitoring-bot added a comment.Sep 28 2021, 9:40 PM

Script wmf-auto-reimage was launched by pt1979 on cumin2002.codfw.wmnet for hosts:

mw2415.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202109282140_pt1979_276513_mw2415_codfw_wmnet.log.

ops-monitoring-bot added a comment.Sep 28 2021, 9:59 PM

Completed auto-reimage of hosts:

['mw2414.codfw.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Sep 28 2021, 9:59 PM

Script wmf-auto-reimage was launched by pt1979 on cumin2002.codfw.wmnet for hosts:

mw2416.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202109282159_pt1979_279172_mw2416_codfw_wmnet.log.

Papaul updated the task description. (Show Details)Sep 28 2021, 10:02 PM
ops-monitoring-bot added a comment.Sep 28 2021, 10:06 PM

Completed auto-reimage of hosts:

['mw2415.codfw.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Sep 28 2021, 10:07 PM

Script wmf-auto-reimage was launched by pt1979 on cumin2002.codfw.wmnet for hosts:

mw2417.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202109282207_pt1979_282626_mw2417_codfw_wmnet.log.

ops-monitoring-bot added a comment.Sep 28 2021, 10:27 PM

Completed auto-reimage of hosts:

['mw2416.codfw.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Sep 28 2021, 10:29 PM

Script wmf-auto-reimage was launched by pt1979 on cumin2002.codfw.wmnet for hosts:

mw2418.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202109282228_pt1979_284985_mw2418_codfw_wmnet.log.

ops-monitoring-bot added a comment.Sep 28 2021, 10:34 PM

Completed auto-reimage of hosts:

['mw2417.codfw.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Sep 28 2021, 10:55 PM

Completed auto-reimage of hosts:

['mw2418.codfw.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Sep 28 2021, 10:55 PM

Script wmf-auto-reimage was launched by pt1979 on cumin2002.codfw.wmnet for hosts:

mw2419.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202109282255_pt1979_290876_mw2419_codfw_wmnet.log.

ops-monitoring-bot added a comment.Sep 28 2021, 11:21 PM

Completed auto-reimage of hosts:

['mw2419.codfw.wmnet']

and were ALL successful.

Papaul updated the task description. (Show Details)Sep 28 2021, 11:23 PM
Papaul added subscribers: jijiki, Dzahn.Sep 29 2021, 12:56 AM

@jijiki @Dzahn this is all ready for service

Thank you.

Papaul closed this task as Resolved.Sep 30 2021, 4:03 PM
Papaul updated the task description. (Show Details)
Papaul updated the task description. (Show Details)

complete

gerritbot added a comment.Apr 21 2022, 12:01 PM

Change 785147 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] site: use appserver in codfw C3, cleanup duplicate insetup definition

https://gerrit.wikimedia.org/r/785147

gerritbot added a project: Patch-For-Review.Apr 21 2022, 12:01 PM
gerritbot added a comment.Apr 25 2022, 6:40 PM

Change 785147 merged by Dzahn:

[operations/puppet@production] site: use appserver in codfw C3, cleanup duplicate insetup definition

https://gerrit.wikimedia.org/r/785147

Maintenance_bot removed a project: Patch-For-Review.Apr 25 2022, 7:30 PM
gerritbot added a comment.Apr 25 2022, 8:20 PM

Change 785918 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] conftool-date: add mw2412 through mw2419 as new appservers

https://gerrit.wikimedia.org/r/785918

gerritbot added a project: Patch-For-Review.Apr 25 2022, 8:20 PM
gerritbot added a comment.Apr 26 2022, 4:23 PM

Change 785918 merged by Dzahn:

[operations/puppet@production] conftool-date: add mw2412 through mw2419 as new appservers

https://gerrit.wikimedia.org/r/785918

Maintenance_bot removed a project: Patch-For-Review.Apr 26 2022, 4:35 PM
Stashbot added a comment.Apr 26 2022, 7:48 PM

Mentioned in SAL (#wikimedia-operations) [2022-04-26T19:48:18Z] <mutante> mw2419 - set weight to 25 in conftool, scap pull, first time in production, jobrunner/videoscaler T290192

Dzahn reopened this task as Open.Apr 27 2022, 9:27 PM
Dzahn claimed this task.
Dzahn removed projects: ops-codfw, DC-Ops.
Dzahn added a subscriber: Papaul.
Dzahn edited subscribers, added: Jelto; removed: Papaul.

after https://gerrit.wikimedia.org/r/c/operations/puppet/+/785918 the conftool-data change does not appear on https://config-master.wikimedia.org/pybal/codfw/ ?

Papaul added a comment.Apr 27 2022, 9:30 PM

@Dzahn i think it is best to create another task for this issue and not reopen the rack/setup task. Thanks

Dzahn added a subscriber: Papaul.Apr 28 2022, 10:01 PM

@Papaul I removed you from the ticket and any tags related to dcops though. Still an issue?

Dzahn mentioned this in T307255: move mw241[2-9].codfw.wmnet into production.Apr 29 2022, 8:31 PM
Dzahn added a parent task: T307255: move mw241[2-9].codfw.wmnet into production.
Dzahn closed this task as Resolved.Apr 29 2022, 8:34 PM
In T290192#7886070, @Papaul wrote:

@Dzahn i think it is best to create another task for this issue and not reopen the rack/setup task. Thanks

replaced with T307255

Dzahn reassigned this task from Dzahn to Papaul.Apr 29 2022, 8:34 PM
akosiaris mentioned this in T313730: decommission mw2251-mw2255, mw2257-mw2258.Jul 25 2022, 2:21 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL