Page MenuHomePhabricator

Q1:rack/setup/install new codfw memcached hosts
Closed, ResolvedPublic

Description

This task will track the racking, setup, and OS installation of two new memcached hosts in codfw.

Hostname / Racking / Installation Details

Hostnames: What are the hostnames, and have you updated https://wikitech.wikimedia.org/wiki/Infrastructure_naming_conventions ? mc-wf200{1,2}, and yes.
Racking Proposal: The two servers need to be racked in different rows from one another. New service cluster, so other memcached hosts/service groups placement are not relevant.
Networking Setup: # of Connections:1, Speed:1G. Vlan:Private AAAA records:Y, Additional IP records (Cassandra)? No
Partitioning/Raid: HW Raid: Y. Partman recipe and/or desired Raid Level: Raid 1
OS Distro: Bullseye (default unless otherwise specified):
Sub-team Technical Contact: @Joe

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

mc-wf2001:
  • - receive in system on procurement task T311854 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
mc-wf2002:
  • - receive in system on procurement task T311854 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.

Related Objects

Event Timeline

RobH added a subscriber: Papaul.

@Joe,

Reassigning this to you per our IRC discussion. Pending needs from you/serviceops:

  • Please populate racking details section with hostname, OS and all other fields before these hosts arrive.
    • The order task is being escalated for placement this week (before July 30th) so expect to see these arrive before mid-August.
  • Once details are added into this task, please reassign it to @Papaul

You'll notice there is a 'implemenetation task' also linked in, which was a serviceops ask for any servers being handed off to the team. I don't have hostname info, so it is generic like this task.

RobH mentioned this in Unknown Object (Task).Jul 27 2022, 6:34 PM
RobH added a parent task: Unknown Object (Task).Jul 27 2022, 6:36 PM
RobH unsubscribed.
Joe removed Joe as the assignee of this task.Aug 8 2022, 6:43 AM
Joe assigned this task to Papaul.
Joe updated the task description. (Show Details)
Joe added a subscriber: RobH.

@RobH the task should be complete with all the info, reassigning to Papaul

@Joe can you please provide me with the partman recipe to use for those servers.The description says only Raid1 .

thanks

Change 825398 had a related patch set uploaded (by Papaul; author: Papaul):

[operations/puppet@production] Add mc-wf200[12] to site.pp

https://gerrit.wikimedia.org/r/825398

Change 825398 merged by Papaul:

[operations/puppet@production] Add mc-wf200[12] to site.pp

https://gerrit.wikimedia.org/r/825398

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mc-wf2001.codfw.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mc-wf2001.codfw.wmnet with OS bullseye completed:

  • mc-wf2001 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202208221950_pt1979_1534342_mc-wf2001.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mc-wf2002.codfw.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mc-wf2002.codfw.wmnet with OS bullseye completed:

  • mc-wf2002 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202208222033_pt1979_1543616_mc-wf2002.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
Papaul updated the task description. (Show Details)

complete