Page MenuHomePhabricator
Create Task
Maniphest T314335

Q1:rack/setup/install druid10[09-11]
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of druid10[09-11]

Hostname / Racking / Installation Details

Hostnames: druid10[09-11]
Racking Proposal: Please distribute anywhere between rows A-F
Networking Setup: 1 x 1Gb connection each please - private vlan (analytics vlan not required)
Partitioning/Raid: HW Raid: N, Partman recipe partman/raid10-8dev.cfg
OS Distro: Bullseye
Sub-team Technical Contact: Data-Engineering team - @BTullis SRE contact

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

druid1009:
  • - receive in system on procurement task T311755 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
druid1010:
  • - receive in system on procurement task T311755 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
druid1011:
  • - receive in system on procurement task T311755 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.

Details

SubjectRepoBranchLines +/-
new servers druid10[09-11] adding basic install infooperations/puppetproduction+2 -1
new servers druid10[09-11] adding basic install infooperations/puppetproduction+4 -0
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
 Resolved• CmjohnsonT314335 Q1:rack/setup/install druid10[09-11]

Event Timeline

RobH created this task.Aug 1 2022, 8:10 PM
RobH moved this task from Backlog to Racking Tasks on the ops-eqiad board.
RobH mentioned this in Unknown Object (Task).
RobH updated the task description. (Show Details)
RobH added a parent task: Unknown Object (Task).
JArguello-WMF moved this task from Incoming (new tickets) to Event Platform Backlog on the Data-Engineering board.Aug 2 2022, 5:34 PM
RobH unsubscribed.Aug 2 2022, 6:02 PM
• EChetty edited projects, added Data-Engineering-Planning; removed Data-Engineering.Aug 3 2022, 12:19 PM
• EChetty moved this task from Backlog to Shared Data Infra on the Data-Engineering-Planning board.Aug 15 2022, 8:44 AM
• EChetty added a project: Shared-Data-Infrastructure.Aug 15 2022, 8:50 AM
• EChetty moved this task from Backlog to To be discussed on the Shared-Data-Infrastructure board.Aug 15 2022, 9:06 AM
Jclark-ctr updated the task description. (Show Details)Aug 31 2022, 2:14 PM
Jclark-ctr added a comment.Aug 31 2022, 2:44 PM
This comment was removed by Jclark-ctr.
Jclark-ctr added a comment.Sep 2 2022, 1:53 PM
This comment was removed by Jclark-ctr.
Jclark-ctr updated the task description. (Show Details)Sep 2 2022, 1:55 PM
Jclark-ctr added a comment.EditedSep 2 2022, 3:00 PM

druid1009 E2 U38 Port43 Cableid 23000023
druid1010 E3 U38 Port43 Cableid 23000054
druid1011 F2 U38 Port43 Cableid 23000028

Jclark-ctr reassigned this task from Jclark-ctr to • Cmjohnson.Sep 2 2022, 3:00 PM
Jclark-ctr subscribed.
• EChetty moved this task from To be discussed to Estimated/Discussed on the Shared-Data-Infrastructure board.Sep 6 2022, 1:23 PM
• EChetty moved this task from Estimated/Discussed to To be discussed on the Shared-Data-Infrastructure board.
• EChetty moved this task from To be discussed to Discussed (Tracking) on the Shared-Data-Infrastructure board.Sep 27 2022, 1:51 PM
• Cmjohnson added a comment.Oct 21 2022, 2:11 PM

The dns has been updated but I am not getting any mgmt connection, I need to check to make sure the mgmt cables are connected.

Jclark-ctr added a comment.Oct 24 2022, 1:50 PM

Verified mgmt cables they are connected and have link

• Cmjohnson added a comment.Oct 27 2022, 1:30 PM

The mgmt links are still not working, The DNS is correct but I am unable to ping the servers.

Jclark-ctr added a comment.Oct 28 2022, 3:23 PM

@Cmjohnson looked at druid10[09-11] bios has not been configured yet. no ip address in set for idrac have you ran the bios script yet?

wiki_willy moved this task from Racking Tasks to Remote Work on the ops-eqiad board.Nov 22 2022, 7:53 PM
Papaul updated the task description. (Show Details)Jan 13 2023, 2:08 AM
Papaul updated the task description. (Show Details)Jan 13 2023, 2:33 AM
Papaul subscribed.Jan 13 2023, 8:41 PM

@BTullis can you please specify the exact partman recipe to use?
Thanks

Papaul updated the task description. (Show Details)Jan 13 2023, 9:02 PM
Papaul added a comment.Jan 17 2023, 5:24 PM

@BTullis any update on this?

BTullis added a comment.Jan 18 2023, 4:51 PM

Hi @Papaul - apologies for the delay in getting back to you.

Please could we use the partman/raid10-8dev.cfg recipe?

I think that this will require setting all of the disks to be 'non-RAID' mode, from the RAID controller configuration in the BIOS, as described here.

We could configure RAID10 in hardware if we have any issues with it this software RAID approach, but I would rather try this MD RAID approach first. Thanks.

BTullis updated the task description. (Show Details)Jan 18 2023, 4:52 PM
gerritbot added a comment.Jan 20 2023, 8:40 PM

Change 881924 had a related patch set uploaded (by Jclark-ctr; author: jclark):

[operations/puppet@production] new servers druid10[09-11] adding basic install info

https://gerrit.wikimedia.org/r/881924

gerritbot added a project: Patch-For-Review.Jan 20 2023, 8:40 PM
gerritbot added a comment.Jan 20 2023, 9:07 PM

Change 881924 merged by Papaul:

[operations/puppet@production] new servers druid10[09-11] adding basic install info

https://gerrit.wikimedia.org/r/881924

gerritbot added a comment.Jan 20 2023, 9:24 PM

Change 881929 had a related patch set uploaded (by Jclark-ctr; author: jclark):

[operations/puppet@production] new servers druid10[09-11] adding basic install info

https://gerrit.wikimedia.org/r/881929

gerritbot added a comment.Jan 20 2023, 11:04 PM

Change 881929 merged by Papaul:

[operations/puppet@production] new servers druid10[09-11] adding basic install info

https://gerrit.wikimedia.org/r/881929

Jclark-ctr updated the task description. (Show Details)Jan 20 2023, 11:12 PM
Maintenance_bot removed a project: Patch-For-Review.Jan 20 2023, 11:30 PM
ops-monitoring-bot added a comment.Jan 24 2023, 2:20 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1001 for host druid1010.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jan 24 2023, 7:05 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1001 for host druid1010.eqiad.wmnet with OS bullseye completed:

  • druid1010 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202301241420_jclark_161590_druid1010.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (row E/F)
ops-monitoring-bot added a comment.Jan 24 2023, 7:06 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1001 for host druid1011.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jan 24 2023, 9:07 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1001 for host druid1011.eqiad.wmnet with OS bullseye completed:

  • druid1011 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202301241906_jclark_304421_druid1011.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (row E/F)
ops-monitoring-bot added a comment.Jan 24 2023, 9:08 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1001 for host druid1009.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jan 24 2023, 9:44 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1001 for host druid1009.eqiad.wmnet with OS bullseye completed:

  • druid1009 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202301242107_jclark_365952_druid1009.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (row E/F)
Jclark-ctr updated the task description. (Show Details)Jan 25 2023, 11:26 AM
Jclark-ctr closed this task as Resolved.Jan 25 2023, 11:32 AM

This is complete @BTullis

BTullis added a comment.Jan 25 2023, 11:33 AM

Great. Many thanks.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL