Page MenuHomePhabricator
Create Task
Maniphest T294302

Q2:(Need By: TBD) rack/setup/install prometheus200[56]
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of <enter the FQDN/hostname of the hosts being setup here>

Hostname / Racking / Installation Details

Hostnames: prometheus200[56]
Racking Proposal: Different rows than one another
Networking/Subnet/VLAN/IP: 1g connection to private1 vlan
Partitioning/Raid: partman/standard.cfg partman/raid10-6dev.cfg partman/custom/prometheus.cfg (note this is experimental, I (Filippo) can take of tweaking the recipe)
OS Distro: Bullseye

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

prometheus2005:

  • - receive in system on procurement task T291979 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to install_server dhcp and netboot, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via wmf-auto-reimage or wmf-auto-reimage-host
  • - host state in netbox set to staged

prometheus2006:

  • - receive in system on procurement task T291979 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to install_server dhcp and netboot, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via wmf-auto-reimage or wmf-auto-reimage-host
  • - host state in netbox set to staged

Once the system(s) above have had all checkbox steps completed, this task can be resolved.

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
install_server: tweak reserved space for prometheusFilippo Giunchedioperations/puppetproduction+2 -2
install_server: tweak reserved space for prometheusFilippo Giunchedioperations/puppetproduction+3 -2
install_server: fix partman config for new prometheus hostsFilippo Giunchedioperations/puppetproduction+2 -3
Fix partman for prometheus200[56]Papauloperations/puppetproduction+1 -0
Add prometheus200[56] to site.pp and netboot.cfgPapauloperations/puppetproduction+6 -1
install_server: simplify custom prometheus.cfgFilippo Giunchedioperations/puppetproduction+8 -98
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
 ResolvedPapaulT294302 Q2:(Need By: TBD) rack/setup/install prometheus200[56]

Event Timeline

RobH created this task.Oct 25 2021, 8:28 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 25 2021, 8:28 PM
RobH assigned this task to fgiunchedi.Oct 25 2021, 8:28 PM
RobH moved this task from Backlog to Racking Tasks on the ops-codfw board.
RobH added subscribers: Papaul, fgiunchedi.

@fgiunchedi,

I had to make some assumptions in the racking details section of this task. Can you please review it all, correct as needed, and when done assign this task to @Papaul? Thanks!

RobH added a parent task: Unknown Object (Task).Oct 25 2021, 8:29 PM
RobH unsubscribed.
Maintenance_bot added a project: SRE.Oct 25 2021, 8:45 PM
fgiunchedi reassigned this task from fgiunchedi to Papaul.Oct 26 2021, 8:28 AM
fgiunchedi updated the task description. (Show Details)
gerritbot added a comment.Oct 26 2021, 8:31 AM

Change 734564 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] install_server: simplify custom prometheus.cfg

https://gerrit.wikimedia.org/r/734564

gerritbot added a project: Patch-For-Review.Oct 26 2021, 8:31 AM
gerritbot added a comment.Oct 27 2021, 7:54 AM

Change 734564 merged by Filippo Giunchedi:

[operations/puppet@production] install_server: simplify custom prometheus.cfg

https://gerrit.wikimedia.org/r/734564

Maintenance_bot removed a project: Patch-For-Review.Oct 27 2021, 8:10 AM
Papaul updated the task description. (Show Details)Nov 17 2021, 1:49 AM
Papaul updated the task description. (Show Details)Nov 19 2021, 1:46 AM
Papaul updated the task description. (Show Details)Nov 19 2021, 8:26 PM
gerritbot added a comment.Nov 19 2021, 10:50 PM

Change 740276 had a related patch set uploaded (by Papaul; author: Papaul):

[operations/puppet@production] Add prometheus200[56] to site.pp and netboot.cfg

https://gerrit.wikimedia.org/r/740276

gerritbot added a project: Patch-For-Review.Nov 19 2021, 10:50 PM
gerritbot added a comment.Nov 19 2021, 10:53 PM

Change 740276 merged by Papaul:

[operations/puppet@production] Add prometheus200[56] to site.pp and netboot.cfg

https://gerrit.wikimedia.org/r/740276

ops-monitoring-bot added a comment.Nov 19 2021, 10:59 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host prometheus2005.codfw.wmnet with OS bullseye

Papaul updated the task description. (Show Details)Nov 19 2021, 11:03 PM
Maintenance_bot removed a project: Patch-For-Review.Nov 19 2021, 11:10 PM
gerritbot added a comment.Nov 19 2021, 11:14 PM

Change 740277 had a related patch set uploaded (by Papaul; author: Papaul):

[operations/puppet@production] Fix partman for prometheus200[56]

https://gerrit.wikimedia.org/r/740277

gerritbot added a project: Patch-For-Review.Nov 19 2021, 11:14 PM

Change 740277 merged by Papaul:

[operations/puppet@production] Fix partman for prometheus200[56]

https://gerrit.wikimedia.org/r/740277

ops-monitoring-bot added a comment.Nov 19 2021, 11:24 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host prometheus2005.codfw.wmnet with OS bullseye executed with errors:

  • prometheus2005 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Nov 19 2021, 11:25 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host prometheus2005.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Nov 19 2021, 11:52 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host prometheus2005.codfw.wmnet with OS bullseye executed with errors:

  • prometheus2005 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
Papaul added a comment.Nov 19 2021, 11:57 PM

prometheus2005 in row B

`

    ┌───────────┤ [!!] Download debconf preconfiguration file ├────────────┐
    │                                                                      │
    │ The netmask is used to determine which machines are local to your    │
    │ network.  Consult your network administrator if you do not know the  │
    │ value.  The netmask should be entered as four numbers separated by   │
    │ periods.                                                             │
    │                                                                      │
    │ Netmask:                                                             │
    │                                                                      │
    │ 255.255.255.0_______________________________________________________ │
    │                                                                      │
    │     <Go Back>                                         <Continue>     │
    │
Maintenance_bot removed a project: Patch-For-Review.Nov 20 2021, 12:10 AM
gerritbot added a comment.Nov 22 2021, 9:33 AM

Change 740531 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] install_server: fix partman config for new prometheus hosts

https://gerrit.wikimedia.org/r/740531

gerritbot added a project: Patch-For-Review.Nov 22 2021, 9:33 AM
fgiunchedi added a comment.Nov 22 2021, 9:34 AM

Thanks @Papaul for taking care of this! I've noticed a typo in netboot.cfg for prometheus which I think might explain the error you are seeing, please check the related patch above

gerritbot added a comment.Nov 22 2021, 10:05 AM

Change 740531 merged by Filippo Giunchedi:

[operations/puppet@production] install_server: fix partman config for new prometheus hosts

https://gerrit.wikimedia.org/r/740531

Maintenance_bot removed a project: Patch-For-Review.Nov 22 2021, 10:10 AM
Papaul added a comment.Nov 22 2021, 12:45 PM

@fgiunchedi Thanks for the fix.

ops-monitoring-bot added a comment.Nov 22 2021, 1:25 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host prometheus2005.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Nov 22 2021, 1:58 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host prometheus2005.codfw.wmnet with OS bullseye completed:

  • prometheus2005 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202111221325_pt1979_1461051_prometheus2005.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
gerritbot added a comment.Nov 22 2021, 2:46 PM

Change 740596 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] install_server: tweak reserved space for prometheus

https://gerrit.wikimedia.org/r/740596

gerritbot added a project: Patch-For-Review.Nov 22 2021, 2:46 PM
gerritbot added a comment.Nov 22 2021, 2:48 PM

Change 740596 merged by Filippo Giunchedi:

[operations/puppet@production] install_server: tweak reserved space for prometheus

https://gerrit.wikimedia.org/r/740596

ops-monitoring-bot added a comment.Nov 22 2021, 2:54 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host prometheus2006.codfw.wmnet with OS bullseye

Papaul updated the task description. (Show Details)Nov 22 2021, 3:09 PM
ops-monitoring-bot added a comment.Nov 22 2021, 3:27 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host prometheus2006.codfw.wmnet with OS bullseye completed:

  • prometheus2006 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202111221454_pt1979_1472331_prometheus2006.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
Papaul updated the task description. (Show Details)Nov 22 2021, 3:29 PM
Papaul closed this task as Resolved.Nov 22 2021, 3:32 PM

This is complete on on end.

gerritbot added a comment.Nov 22 2021, 3:32 PM

Change 740633 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] install_server: tweak reserved space for prometheus

https://gerrit.wikimedia.org/r/740633

fgiunchedi mentioned this in T296199: Prometheus hardware refresh (+ Bullseye upgrade).Nov 22 2021, 3:54 PM
gerritbot added a comment.Nov 23 2021, 11:39 AM

Change 740633 merged by Filippo Giunchedi:

[operations/puppet@production] install_server: tweak reserved space for prometheus

https://gerrit.wikimedia.org/r/740633

• lmata moved this task from Inbox to Done on the SRE Observability (FY2021/2022-Q2) board.Jul 18 2023, 5:03 PM
Maintenance_bot removed a project: Patch-For-Review.Jul 18 2023, 5:12 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits