Page MenuHomePhabricator
Create Task
Maniphest T301399

Q3:(Need By: TBD) rack/setup/install restbase2027
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of restbase2027

Hostname / Racking / Installation Details

Hostnames: restbase2027
Racking Proposal: row d
Networking/Subnet/VLAN/IP: match other restbase, (1) 1g private1 vlan - use restbase networking config in netbox - 3 Cassandra instances
Partitioning/Raid: match other restbase (cassandrahosts-3ssd-jbod)
OS Distro: buster

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

restbase2027: D5/U10 ge-5/0/9
  • - receive in system on procurement task T300287 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.

Details

SubjectRepoBranchLines +/-
install_server: use non-reuse partition for new hostoperations/puppetproduction+3 -3
Add restbase2027 to site.pp and netboot.cfgoperations/puppetproduction+6 -1
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
 ResolvedPapaulT301399 Q3:(Need By: TBD) rack/setup/install restbase2027

Event Timeline

RobH created this task.Feb 9 2022, 7:30 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 9 2022, 7:30 PM
RobH moved this task from Backlog to Racking Tasks on the ops-codfw board.Feb 9 2022, 7:30 PM
RobH mentioned this in Unknown Object (Task).
RobH added a parent task: Unknown Object (Task).
RobH unsubscribed.
hnowlan updated the task description. (Show Details)Feb 18 2022, 11:47 AM
Papaul updated the task description. (Show Details)Mar 24 2022, 3:23 PM
gerritbot added a comment.Mar 24 2022, 10:40 PM

Change 773651 had a related patch set uploaded (by Papaul; author: Papaul):

[operations/puppet@production] Add restbase2027 to site.pp and netboot.cfg

https://gerrit.wikimedia.org/r/773651

gerritbot added a project: Patch-For-Review.Mar 24 2022, 10:40 PM
Papaul updated the task description. (Show Details)Mar 24 2022, 10:40 PM
gerritbot added a comment.Mar 24 2022, 11:05 PM

Change 773651 merged by Papaul:

[operations/puppet@production] Add restbase2027 to site.pp and netboot.cfg

https://gerrit.wikimedia.org/r/773651

Maintenance_bot removed a project: Patch-For-Review.Mar 24 2022, 11:10 PM
ops-monitoring-bot added a comment.Mar 24 2022, 11:57 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host restbase2027.codfw.wmnet with OS buster

Papaul added a comment.Mar 25 2022, 12:38 AM

getting the message below during install

reuse-parts: Recipe device matching failed │
                 │ ERROR: =dev=md0 matches zero devices       │
                 │                                            │
                 │ All devices:                               │
                 │ =dev=sda                                   │
                 │ =dev=sdb                                   │
                 │ =dev=sdc                                   │
                 │                                            │
                 │     <Go Back>               <Continue>
ops-monitoring-bot added a comment.Mar 25 2022, 12:39 AM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host restbase2027.codfw.wmnet with OS buster executed with errors:

  • restbase2027 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Mar 28 2022, 3:02 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host restbase2027.codfw.wmnet with OS buster

Papaul reassigned this task from Papaul to hnowlan.Mar 28 2022, 3:12 PM
Papaul subscribed.

@hnowlan can you please check the partman recipe when done assign the task back to me.

Thanks

Papaul updated the task description. (Show Details)Mar 28 2022, 3:12 PM
ops-monitoring-bot added a comment.Mar 28 2022, 3:34 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host restbase2027.codfw.wmnet with OS buster executed with errors:

  • restbase2027 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • The reimage failed, see the cookbook logs for the details
Papaul updated the task description. (Show Details)Apr 6 2022, 1:41 PM

@hnowlan any update on this?

hnowlan added a comment.Apr 12 2022, 3:42 PM

For new hosts, it seems the reuse profile won't work as it expects an existing array. The non-reuse cassandrahosts-3ssd-jbod config is required.

gerritbot added a comment.Apr 12 2022, 3:45 PM

Change 779494 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] install_server: use non-reuse partition for new host

https://gerrit.wikimedia.org/r/779494

gerritbot added a project: Patch-For-Review.Apr 12 2022, 3:45 PM
gerritbot added a comment.Apr 12 2022, 4:04 PM

Change 779494 merged by Hnowlan:

[operations/puppet@production] install_server: use non-reuse partition for new host

https://gerrit.wikimedia.org/r/779494

hnowlan reassigned this task from hnowlan to Papaul.Apr 12 2022, 4:05 PM
Papaul added a comment.Apr 12 2022, 4:06 PM

@hnowlan
" use non-reuse partition for new host"
so if you want to re-image this host later down the road you will have to change it again in netboot.cfg to use reuse-parts.cfg ?

hnowlan added a comment.Apr 12 2022, 4:09 PM
In T301399#7848614, @Papaul wrote:

@hnowlan
" use non-reuse partition for new host"
so if you want to re-image this host later down the road you will have to change it again in netboot.cfg to use reuse-parts.cfg ?

Yep - once the imaging is complete I'll change it to the reuse profile like the other existing hosts and it'll be ready to reimage for future upgrades.

ops-monitoring-bot added a comment.Apr 12 2022, 4:30 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host restbase2027.codfw.wmnet with OS buster

Maintenance_bot removed a project: Patch-For-Review.Apr 12 2022, 4:31 PM
ops-monitoring-bot added a comment.Apr 12 2022, 5:03 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host restbase2027.codfw.wmnet with OS buster completed:

  • restbase2027 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202204121630_pt1979_2141565_restbase2027.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
Papaul closed this task as Resolved.Apr 12 2022, 5:43 PM
Papaul updated the task description. (Show Details)

@hnowlan complete

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits