Page MenuHomePhabricator
Create Task
Maniphest T342161

Q1:rack/setup/install cloudservices1006.eqiad.wmnet
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of cloudservices1006.eqiad.wmnet

Hostname / Racking / Installation Details

Hostnames: cloudservices1006.eqiad.wmnet
Racking Proposal: eqiad C8
Networking Setup: cloudsw switch, cloud-host vlan
Partitioning/Raid: HW Raid: Y/N, Partman recipe and/or desired Raid Level: standard
OS Distro: Bullseye (default unless otherwise specified)
Sub-team Technical Contact: WMCS @aborrero, @Andrew or SRE/IF @cmooney

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

cloudservices1006:
  • - receive in system on procurement task T341245 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.

Details

SubjectRepoBranchLines +/-
Add cloudservices1006 to site.ppoperations/puppetproduction+4 -0
Add cloudservices1006 to netboot.cfgoperations/puppetproduction+1 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

RobH created this task.Jul 18 2023, 7:21 PM
RobH added a parent task: Unknown Object (Task).
RobH mentioned this in Unknown Object (Task).
Maintenance_bot added a project: SRE.Jul 18 2023, 7:29 PM
aborrero added a project: User-aborrero.Jul 19 2023, 11:38 AM
aborrero moved this task from Backlog to Radar on the User-aborrero board.
RobH unsubscribed.Jul 19 2023, 12:27 PM
Jclark-ctr moved this task from Backlog to Racking Tasks on the ops-eqiad board.Jul 19 2023, 7:03 PM
aborrero mentioned this in T341494: cloud @ eqiad: hardware re-racking plan.Jul 25 2023, 11:06 AM
aborrero mentioned this in T341060: openstack eqiad1: introduce cloud-private and cloudlb.
aborrero moved this task from Radar to Doing on the User-aborrero board.Jul 25 2023, 11:10 AM
gerritbot added a comment.Jul 25 2023, 11:21 AM

Change 941383 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloudservices1006: prepare service

https://gerrit.wikimedia.org/r/941383

gerritbot added a project: Patch-For-Review.Jul 25 2023, 11:21 AM
Jclark-ctr subscribed.Jul 26 2023, 2:46 PM

Server has not arrived yet they should arrive any day now.

Jul 24, 2023 9:16 PM
Departed Terminal Location ASHBURN, 20147, US

aborrero moved this task from Doing to Next on the User-aborrero board.Jul 27 2023, 10:01 AM
aborrero added a parent task: T341494: cloud @ eqiad: hardware re-racking plan.Jul 28 2023, 8:21 AM
Jclark-ctr added a comment.Aug 2 2023, 1:39 PM

@aborrero @Andrew Will this server be using single or dual network connection?

Jclark-ctr added a comment.Aug 2 2023, 1:43 PM

cloudservices1006 C8 u20 port 19 (cableid - 5321) port 43(cableid 5329)

Jclark-ctr updated the task description. (Show Details)Aug 2 2023, 1:43 PM
Jclark-ctr updated the task description. (Show Details)Aug 2 2023, 1:53 PM
Jclark-ctr added a comment.Aug 2 2023, 2:00 PM

@aborrero @Andrew i have connected two network ports to prevent any blockers from remote work but only have entered one into netbox while we wait for answer on number of network connections

Jclark-ctr updated the task description. (Show Details)Aug 2 2023, 4:03 PM
Jclark-ctr added a comment.Aug 2 2023, 4:10 PM

Raid has not been configured yet on server. What raid is needed for this server @aborrero @Andrew

Andrew added a comment.Aug 2 2023, 9:59 PM

I'm sure that this only needs a single nic connected unless @aborrero has something truly ambitious in mind. Seems easy enough to leave the second cable attached until he's back in September though. Thank you!

Andrew added a comment.Aug 2 2023, 10:01 PM

Oh, raid-wise: the existing cloudservices use

echo partman/standard.cfg partman/raid1-2dev.cfg ;; \

That, or similar, is fine for this host as well. Nothing too strenuous will be happening on disk.

Jclark-ctr moved this task from Racking Tasks to Remote Work on the ops-eqiad board.Aug 4 2023, 12:43 PM
aborrero added a comment.Aug 29 2023, 9:17 AM
In T342161#9064747, @Andrew wrote:

I'm sure that this only needs a single nic connected unless @aborrero has something truly ambitious in mind. Seems easy enough to leave the second cable attached until he's back in September though. Thank you!

Single NIC is fine.

aborrero added a comment.Aug 29 2023, 9:53 AM

Is there any blocker for this server to get it ready to go?

aborrero moved this task from Next to Doing on the User-aborrero board.Aug 29 2023, 11:24 AM
aborrero moved this task from Doing to Blocked on the User-aborrero board.Aug 29 2023, 11:27 AM
aborrero mentioned this in T342621: eqiad1: cloudlb: transition DNS clients (VMs) to the new BGP-based recursor VIP.Aug 29 2023, 12:39 PM
Papaul claimed this task.Aug 29 2023, 3:24 PM
ops-monitoring-bot added a comment.Aug 29 2023, 3:27 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host cloudservices1006.eqiad.wmnet with OS bullseye

Papaul added a subscriber: VRiley-WMF.Aug 29 2023, 3:29 PM

@Jclark-ctr @VRiley-WMF i am taking over this task to finish it for the cloud team. thanks

aborrero added a comment.Aug 29 2023, 3:46 PM

Please use partman/raid10-4dev.cfg as it seems the most standard for 4 devices.

gerritbot added a comment.Aug 29 2023, 3:47 PM

Change 953299 had a related patch set uploaded (by Papaul; author: Papaul):

[operations/puppet@production] Add cloudservices1006 to netboot.cfg

https://gerrit.wikimedia.org/r/953299

gerritbot added a comment.Aug 29 2023, 3:49 PM

Change 953299 merged by Papaul:

[operations/puppet@production] Add cloudservices1006 to netboot.cfg

https://gerrit.wikimedia.org/r/953299

ops-monitoring-bot added a comment.Aug 29 2023, 5:55 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host cloudservices1006.eqiad.wmnet with OS bullseye executed with errors:

  • cloudservices1006 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • The reimage failed, see the cookbook logs for the details
gerritbot added a comment.Aug 29 2023, 6:49 PM

Change 953327 had a related patch set uploaded (by Papaul; author: Papaul):

[operations/puppet@production] Add cloudservices1006 to site.pp

https://gerrit.wikimedia.org/r/953327

gerritbot added a comment.Aug 29 2023, 6:53 PM

Change 953327 merged by Papaul:

[operations/puppet@production] Add cloudservices1006 to site.pp

https://gerrit.wikimedia.org/r/953327

ops-monitoring-bot added a comment.Aug 29 2023, 6:54 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host cloudservices1006.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Aug 29 2023, 6:55 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host cloudservices1006.eqiad.wmnet with OS bullseye executed with errors:

  • cloudservices1006 (FAIL)
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Aug 29 2023, 6:55 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host cloudservices1006.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Aug 29 2023, 7:24 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host cloudservices1006.eqiad.wmnet with OS bullseye completed:

  • cloudservices1006 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202308291901_pt1979_2816826_cloudservices1006.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Papaul closed this task as Resolved.Aug 29 2023, 7:26 PM
Papaul updated the task description. (Show Details)

@aborrero all your's

aborrero changed the status of subtask T345240: cloudservices1006: put into service from Open to In Progress.Aug 30 2023, 10:46 AM
ops-monitoring-bot added a comment.Sep 1 2023, 6:22 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host cloudservices1006.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 1 2023, 6:22 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host cloudservices1006.eqiad.wmnet with OS bullseye executed with errors:

  • cloudservices1006 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • The reimage failed, see the cookbook logs for the details
aborrero added a comment.Sep 4 2023, 8:42 AM

We are already working at service level with this box. We should coordinate reimage/reboots etc.

aborrero closed subtask T345240: cloudservices1006: put into service as Resolved.Sep 11 2023, 2:33 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits