Page MenuHomePhabricator
Create Task
Maniphest T356216

Q#:rack/setup/install (2) cloudbackup hosts
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of two new hosts to replace cloudbackup200[1,2]

Hostname / Racking / Installation Details

Hostnames: cloudbackup2003.codfw.wmnet and cloudbackup2004.codfw.wmnet
Racking Proposal: These are not part of the codfw1dev cluster, so they should not be placed in one of the existing codfw1dev racks (which have switches set up specifically for that.) They're not an HA pair, so it's fine if they're in the same rack elsewhere.
Networking Setup: 1 10G connection each. Private vlan (.codfw.wmnet), yes to AAAA records.
Partitioning/Raid: Hardware raid 1 for the two smaller OS drives, Hardware raid6 collecting all the remaining drives into one volume.
OS Distro: Bookworm
Sub-team Technical Contact: @Andrew

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

cloudbackup2003:
  • Receive in system on procurement task T356088 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
cloudbackup2004:
  • Receive in system on procurement task T356088 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
Prepare cloudbackup200[12] for decomAndrew Bogottoperations/puppetproduction+2 -5
cinder backups: move real backup workloads to 200[34]Andrew Bogottoperations/puppetproduction+4 -7
backy2: apply fix-backy2-crypto-imports on BookwormAndrew Bogottoperations/puppetproduction+2 -2
cloudbackup: put new cloudbackup hosts into service with minimal test loadAndrew Bogottoperations/puppetproduction+6 -1
site.pp: add insetup entries for new cloudbackup200[34] hostsAndrew Bogottoperations/puppetproduction+5 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

RobH created this task.Jan 30 2024, 8:33 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 30 2024, 8:33 PM
RobH renamed this task from Q#:rack/setup/install X to Q#:rack/setup/install (2) cloudbackup hosts.Jan 30 2024, 8:33 PM
RobH assigned this task to Andrew.
RobH moved this task from Backlog to Racking Tasks on the ops-codfw board.

@Andrew: I've assigned this task to you for you to populate the racking details, additionally please add the servers to the site.pp file with the insetup role and their partition info.

Once done, you can unassign this from yourself, it is already in the proper queue and column for when the hosts arrive.

RobH mentioned this in Unknown Object (Task).Jan 30 2024, 8:34 PM
RobH unsubscribed.
Jhancock.wm subscribed.Mar 18 2024, 3:05 PM

@Andrew we've received these servers. Could you update this ticket with racking requirements and names of the servers?

Jhancock.wm updated the task description. (Show Details)Mar 18 2024, 3:05 PM
Maintenance_bot added a project: SRE.Mar 18 2024, 3:29 PM
Andrew reassigned this task from Andrew to Jhancock.wm.Mar 20 2024, 3:59 AM
Andrew updated the task description. (Show Details)

Sorry for the slow response! I hope I've now included all that you need.

Jhancock.wm updated the task description. (Show Details)Mar 21 2024, 1:43 PM
Jhancock.wm updated the task description. (Show Details)Mar 21 2024, 3:41 PM
Jhancock.wm added a comment.Mar 22 2024, 2:43 PM

@Andrew thanks for the update. Can I bug you to update the site.pp as well? Thanks!

gerritbot added a comment.Mar 22 2024, 2:48 PM

Change #1013559 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] site.pp: add insetup entries for new cloudbackup200[34] hosts

https://gerrit.wikimedia.org/r/1013559

gerritbot added a project: Patch-For-Review.Mar 22 2024, 2:48 PM
gerritbot added a comment.Mar 22 2024, 2:52 PM

Change #1013559 merged by Andrew Bogott:

[operations/puppet@production] site.pp: add insetup entries for new cloudbackup200[34] hosts

https://gerrit.wikimedia.org/r/1013559

Jhancock.wm updated the task description. (Show Details)Mar 22 2024, 5:38 PM
ops-monitoring-bot added a comment.Mar 22 2024, 5:51 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host cloudbackup2003.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Mar 22 2024, 5:51 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host cloudbackup2004.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Mar 22 2024, 6:35 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host cloudbackup2004.codfw.wmnet with OS bookworm completed:

  • cloudbackup2004 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202403221814_jhancock_3473792_cloudbackup2004.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jhancock.wm updated the task description. (Show Details)Mar 22 2024, 6:36 PM
Jhancock.wm added a comment.Mar 22 2024, 7:14 PM

cloudbackup2003 has os after a few attempts. had to delete and redo the virtual disks twice before it took. but now it is having issues with the puppet server. will try again later.

ops-monitoring-bot added a comment.Mar 22 2024, 11:15 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host cloudbackup2003.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Mar 22 2024, 11:58 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host cloudbackup2003.codfw.wmnet with OS bookworm executed with errors:

  • cloudbackup2003 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Generated Puppet certificate
    • The reimage failed, see the cookbook logs for the details,You can also try typing "install-console" cloudbackup2003.codfw.wmnet to get a root shellbut depending on the failure this may not work.
ops-monitoring-bot added a comment.Mar 25 2024, 2:10 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host cloudbackup2003.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Mar 25 2024, 2:10 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host cloudbackup2003.codfw.wmnet with OS bookworm executed with errors:

  • cloudbackup2003 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • The reimage failed, see the cookbook logs for the details,You can also try typing "install-console" cloudbackup2003.codfw.wmnet to get a root shellbut depending on the failure this may not work.
ops-monitoring-bot added a comment.Mar 25 2024, 2:22 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host cloudbackup2003.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Mar 25 2024, 5:01 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host cloudbackup2003.codfw.wmnet with OS bookworm

Papaul subscribed.Mar 27 2024, 4:48 PM

@Jhancock.wm this is what 2003 is showing on console

┌───────────────────────┤ [!!] Partition disks ├────────────────────────┐
   │                                                                       │
   │                 Failed to partition the selected disk                 │
   │ This probably happened because the selected disk or free space is too │
   │ small to be automatically partitioned.                                │
   │                                                                       │
   │     <Go Back>
Andrew added a comment.EditedMar 28 2024, 1:40 PM

I suspect that on re-run partman isn't properly zeroing out the partition table before it starts so we're accumulating cruft. I'd try visiting the CLI and deleting everything manually before rerunning.

ops-monitoring-bot added a comment.Mar 28 2024, 3:53 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host cloudbackup2003.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Mar 28 2024, 4:47 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host cloudbackup2003.codfw.wmnet with OS bookworm completed:

  • cloudbackup2003 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Unable to downtime the new host on Icinga/Alertmanager, the sre.hosts.downtime cookbook returned 99
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202403281626_jhancock_1120412_cloudbackup2003.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (EVPN Switch)
Jhancock.wm closed this task as Resolved.Mar 28 2024, 4:47 PM
Jhancock.wm updated the task description. (Show Details)

issue fixed and ready to go @Andrew

Andrew reopened this task as Open.Mar 29 2024, 1:01 PM
Andrew claimed this task.
gerritbot added a comment.Mar 29 2024, 1:05 PM

Change #1015526 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] cloudbackup: put new cloudbackup hosts into service with minimal test load

https://gerrit.wikimedia.org/r/1015526

gerritbot added a comment.Mar 29 2024, 1:52 PM

Change #1015526 merged by Andrew Bogott:

[operations/puppet@production] cloudbackup: put new cloudbackup hosts into service with minimal test load

https://gerrit.wikimedia.org/r/1015526

gerritbot added a comment.Mar 29 2024, 2:35 PM

Change #1015535 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] backy2: apply fix-backy2-crypto-imports on Bookworm

https://gerrit.wikimedia.org/r/1015535

gerritbot added a comment.Mar 29 2024, 2:36 PM

Change #1015535 merged by Andrew Bogott:

[operations/puppet@production] backy2: apply fix-backy2-crypto-imports on Bookworm

https://gerrit.wikimedia.org/r/1015535

Andrew added a comment.Mar 29 2024, 2:40 PM

These are now set up and should start running a few backup jobs over the weekend. I need to check back and make sure things are actually working.

taavi closed subtask T361537: connectivity from cloudbackup200[34] and eqiad ceph as Resolved.Apr 2 2024, 7:38 AM
gerritbot added a comment.Apr 8 2024, 6:13 PM

Change #1017915 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] cinder backups: move real backup workloads to 200[34]

https://gerrit.wikimedia.org/r/1017915

gerritbot added a comment.Apr 8 2024, 6:13 PM

Change #1017916 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] Prepare cloudbackup200[12] for decom

https://gerrit.wikimedia.org/r/1017916

gerritbot added a comment.Apr 8 2024, 6:14 PM

Change #1017915 merged by Andrew Bogott:

[operations/puppet@production] cinder backups: move real backup workloads to 200[34]

https://gerrit.wikimedia.org/r/1017915

gerritbot added a comment.Apr 12 2024, 3:09 AM

Change #1017916 merged by Andrew Bogott:

[operations/puppet@production] Prepare cloudbackup200[12] for decom

https://gerrit.wikimedia.org/r/1017916

Jhancock.wm moved this task from Racking Tasks to Blocked on the ops-codfw board.Apr 16 2024, 1:24 PM
Papaul added a comment.Apr 17 2024, 12:23 AM

@Jhancock.wm anything else left to be done on this task?

Andrew closed this task as Resolved.Apr 17 2024, 1:27 PM

These are now in service and working fine.

Maintenance_bot removed a project: Patch-For-Review.Apr 26 2024, 6:59 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits