Page MenuHomePhabricator
Create Task
Maniphest T370678

Q1:rack/setup/install cloudlb2004-dev
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of cloudlb2004-dev

Hostname / Racking / Installation Details

Hostnames: cloudlb2004-dev.codfw.wmnet
Racking Proposal: codfw B1
Networking Setup: # of Connections: 1 - Speed: 10G. - VLAN: cloudsw1-b1-codfw - AAAA records: Y, Additional IP records (Cassandra)? No
Partitioning/Raid: HW Raid: N, Partman recipe and/or desired Raid Level: raid 10, mirror the two drives
OS Distro: Bookworm
Sub-team Technical Contact: Arturo Borrero

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

cloudlb2004-dev
  • Receive in system on procurement task T368965 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
cloudlb2004-dev: use insetup role and add partman recipeoperations/puppetproduction+6 -2
cloudlb2004-dev: give it a puppet roleoperations/puppetproduction+1 -1
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
 ResolvedJhancock.wmT370678 Q1:rack/setup/install cloudlb2004-dev
 Resolved• aborreroT377126 replace cloudlb2001-dev with cloudlb2004-dev
 ResolvedRequestJhancock.wmT392686 decommission cloudlb2001-dev.codfw.wmnet

Event Timeline

RobH created this task.Jul 22 2024, 5:06 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 22 2024, 5:06 PM
RobH moved this task from Backlog to Racking Tasks on the ops-codfw board.Jul 22 2024, 5:06 PM
RobH mentioned this in Unknown Object (Task).
RobH added a parent task: Unknown Object (Task).
RobH unsubscribed.
Maintenance_bot added a project: SRE.Jul 22 2024, 5:29 PM
Jhancock.wm claimed this task.Aug 13 2024, 1:51 PM
Jhancock.wm updated the task description. (Show Details)
Jhancock.wm updated the task description. (Show Details)Aug 15 2024, 7:37 PM
Jhancock.wm added a subscriber: • aborrero.Aug 15 2024, 8:23 PM

@aborrero when you have a moment, can you do this step for me please? thanks!
Update the operations/puppet repo

ops-monitoring-bot added a comment.Aug 15 2024, 10:10 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host cloudlb2004-dev.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Aug 15 2024, 11:30 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host cloudlb2004-dev.codfw.wmnet with OS bookworm executed with errors:

  • cloudlb2004-dev (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details,You can also try typing "install-console" cloudlb2004-dev.codfw.wmnet to get a root shellbut depending on the failure this may not work.
Jhancock.wm updated the task description. (Show Details)Aug 16 2024, 2:23 PM
ops-monitoring-bot added a comment.Aug 19 2024, 6:31 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host cloudlb2004-dev.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Aug 19 2024, 7:51 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host cloudlb2004-dev.codfw.wmnet with OS bookworm executed with errors:

  • cloudlb2004-dev (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details,You can also try typing "install-console" cloudlb2004-dev.codfw.wmnet to get a root shellbut depending on the failure this may not work.
Jhancock.wm added a comment.Sep 30 2024, 4:19 PM

@aborrero I accidentally ran a few imaging attempts while just going through lists. Could you update the site.pp file for us? Thanks!

• aborrero added a comment.Oct 1 2024, 12:55 PM

I'm sorry I completely missed the pings here. I will get this done today.

• aborrero added a project: User-aborrero.Oct 1 2024, 3:20 PM
• aborrero moved this task from Backlog to Doing on the User-aborrero board.
gerritbot added a comment.Oct 1 2024, 3:24 PM

Change #1077049 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloudlb2004-dev: give it a puppet role

https://gerrit.wikimedia.org/r/1077049

gerritbot added a project: Patch-For-Review.Oct 1 2024, 3:24 PM

Change #1077049 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloudlb2004-dev: give it a puppet role

https://gerrit.wikimedia.org/r/1077049

• aborrero added a comment.Oct 1 2024, 3:27 PM

please @Jhancock.wm try now.

Maintenance_bot removed a project: Patch-For-Review.Oct 1 2024, 3:31 PM
ops-monitoring-bot added a comment.Oct 1 2024, 4:30 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host cloudlb2004-dev.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Oct 1 2024, 5:50 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host cloudlb2004-dev.codfw.wmnet with OS bookworm executed with errors:

  • cloudlb2004-dev (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details,You can also try typing "sudo install-console cloudlb2004-dev.codfw.wmnet" to get a root shellbut depending on the failure this may not work.
ops-monitoring-bot added a comment.Oct 2 2024, 1:20 AM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host cloudlb2004-dev.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Oct 2 2024, 2:40 AM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host cloudlb2004-dev.codfw.wmnet with OS bookworm executed with errors:

  • cloudlb2004-dev (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details,You can also try typing "sudo install-console cloudlb2004-dev.codfw.wmnet" to get a root shellbut depending on the failure this may not work.
gerritbot added a comment.Oct 2 2024, 10:11 AM

Change #1077337 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloudlb2004-dev: use insetup role and add partman recipe

https://gerrit.wikimedia.org/r/1077337

gerritbot added a project: Patch-For-Review.Oct 2 2024, 10:11 AM
gerritbot added a comment.Oct 2 2024, 11:08 AM

Change #1077337 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloudlb2004-dev: use insetup role and add partman recipe

https://gerrit.wikimedia.org/r/1077337

• aborrero moved this task from Doing to Radar/observer on the User-aborrero board.Oct 2 2024, 11:22 AM
Maintenance_bot removed a project: Patch-For-Review.Oct 2 2024, 11:30 AM
• aborrero added a comment.Oct 3 2024, 12:42 PM

please @Jhancock.wm try again with this one after the patch I merged yesterday.

Jhancock.wm updated the task description. (Show Details)Oct 7 2024, 4:32 PM
ops-monitoring-bot added a comment.Oct 7 2024, 8:12 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host cloudlb2004-dev.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Oct 7 2024, 9:32 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host cloudlb2004-dev.codfw.wmnet with OS bookworm executed with errors:

  • cloudlb2004-dev (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details,You can also try typing "sudo install-console cloudlb2004-dev.codfw.wmnet" to get a root shellbut depending on the failure this may not work.
ops-monitoring-bot added a comment.Oct 8 2024, 2:30 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host cloudlb2004-dev.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Oct 8 2024, 3:33 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host cloudlb2004-dev.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Oct 8 2024, 4:48 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host cloudlb2004-dev.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Oct 8 2024, 4:48 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host cloudlb2004-dev.codfw.wmnet with OS bookworm executed with errors:

  • cloudlb2004-dev (FAIL)
    • The reimage failed, see the cookbook logs for the details,You can also try typing "sudo install-console cloudlb2004-dev.codfw.wmnet" to get a root shellbut depending on the failure this may not work.
ops-monitoring-bot added a comment.Oct 8 2024, 7:54 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host cloudlb2004-dev.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Oct 9 2024, 12:59 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host cloudlb2004-dev.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Oct 9 2024, 2:20 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host cloudlb2004-dev.codfw.wmnet with OS bookworm executed with errors:

  • cloudlb2004-dev (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details,You can also try typing "sudo install-console cloudlb2004-dev.codfw.wmnet" to get a root shellbut depending on the failure this may not work.
ops-monitoring-bot added a comment.Oct 11 2024, 4:16 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host cloudlb2004-dev.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Oct 11 2024, 4:57 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host cloudlb2004-dev.codfw.wmnet with OS bookworm completed:

  • cloudlb2004-dev (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202410111637_jhancock_1551871_cloudlb2004-dev.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jhancock.wm closed this task as Resolved.Oct 11 2024, 5:11 PM

@aborrero this is finally ready. turned into a learning opprotunity

Jhancock.wm updated the task description. (Show Details)Oct 11 2024, 5:12 PM
taavi closed subtask T377126: replace cloudlb2001-dev with cloudlb2004-dev as Resolved.May 27 2025, 12:38 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits