Page MenuHomePhabricator
Create Task
Maniphest T341999

Create 'titan' role and put new hosts in service
Closed, ResolvedPublic
Actions

Description

This task tracks putting in production titan role, and related hosts, and have them in rotation together with thanos-fe hosts as part of serving thanos-query.discovery.wmnet

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
install_server: use raid0 for titanFilippo Giunchedioperations/puppetproduction+1 -1
titan: move pyrra off thanos roleFilippo Giunchedioperations/puppetproduction+3 -2
cumin: add titan aliasesFilippo Giunchedioperations/puppetproduction+3 -0
alerting_host: add titan to thanos-query hostsFilippo Giunchedioperations/puppetproduction+4 -0
hieradata: add titan hosts to thanos frontendsFilippo Giunchedioperations/puppetproduction+4 -0
conftool-data: add titan hosts to thanosFilippo Giunchedioperations/puppetproduction+4 -0
site: apply role to titan hostsFilippo Giunchedioperations/puppetproduction+3 -4
prometheus: create /etc/prometheus when neededFilippo Giunchedioperations/puppetproduction+11 -0
New role: titanFilippo Giunchedioperations/puppetproduction+79 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

fgiunchedi created this task.Jul 17 2023, 2:19 PM
gerritbot added a comment.Jul 17 2023, 2:22 PM

Change 938866 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] New role: titan

https://gerrit.wikimedia.org/r/938866

gerritbot added a project: Patch-For-Review.Jul 17 2023, 2:22 PM

Change 938867 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] prometheus: create /etc/prometheus when needed

https://gerrit.wikimedia.org/r/938867

fgiunchedi mentioned this in T341488: Split Thanos components from thanos-fe hosts into titan hosts.Jul 17 2023, 2:26 PM
gerritbot added a comment.Jul 18 2023, 7:24 AM

Change 938866 merged by Filippo Giunchedi:

[operations/puppet@production] New role: titan

https://gerrit.wikimedia.org/r/938866

gerritbot added a comment.Jul 18 2023, 7:24 AM

Change 938867 merged by Filippo Giunchedi:

[operations/puppet@production] prometheus: create /etc/prometheus when needed

https://gerrit.wikimedia.org/r/938867

Maintenance_bot removed a project: Patch-For-Review.Jul 18 2023, 7:30 AM
fgiunchedi moved this task from Backlog to Doing on the User-fgiunchedi board.Jul 18 2023, 7:30 AM
fgiunchedi moved this task from Doing to Up next on the User-fgiunchedi board.
• lmata triaged this task as Medium priority.Jul 18 2023, 5:28 PM
• lmata moved this task from Inbox to Prioritized on the Observability-Metrics board.
• lmata added a project: SRE Observability (FY2023/2024-Q1).
• lmata moved this task from Inbox to Up next on the SRE Observability (FY2023/2024-Q1) board.
• lmata subscribed.

Since this is part of core work would this be better fitted as "high" priority?

fgiunchedi raised the priority of this task from Medium to High.Jul 19 2023, 8:32 AM

Sure why not, {{done}}

fgiunchedi moved this task from Up next to Doing on the User-fgiunchedi board.Sep 12 2023, 8:07 AM
gerritbot added a comment.Sep 12 2023, 8:10 AM

Change 956782 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] site: apply role to titan hosts

https://gerrit.wikimedia.org/r/956782

gerritbot added a project: Patch-For-Review.Sep 12 2023, 8:10 AM
gerritbot added a comment.Sep 12 2023, 8:18 AM

Change 956782 merged by Filippo Giunchedi:

[operations/puppet@production] site: apply role to titan hosts

https://gerrit.wikimedia.org/r/956782

Maintenance_bot removed a project: Patch-For-Review.Sep 12 2023, 8:30 AM
gerritbot added a comment.Sep 12 2023, 8:30 AM

Change 956788 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] hieradata: add titan hosts to thanos frontends

https://gerrit.wikimedia.org/r/956788

gerritbot added a project: Patch-For-Review.Sep 12 2023, 8:30 AM
gerritbot added a comment.Sep 12 2023, 9:44 AM

Change 956788 merged by Filippo Giunchedi:

[operations/puppet@production] hieradata: add titan hosts to thanos frontends

https://gerrit.wikimedia.org/r/956788

gerritbot added a comment.Sep 12 2023, 9:48 AM

Change 956800 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] conftool-data: add titan hosts to thanos

https://gerrit.wikimedia.org/r/956800

gerritbot added a comment.Sep 12 2023, 11:33 AM

Change 956800 merged by Filippo Giunchedi:

[operations/puppet@production] conftool-data: add titan hosts to thanos

https://gerrit.wikimedia.org/r/956800

Stashbot added a comment.Sep 12 2023, 11:43 AM

Mentioned in SAL (#wikimedia-operations) [2023-09-12T11:43:39Z] <godog> pool titan hosts alongside thanos-fe for thanos-query / thanos-web services - T341999

Stashbot added a comment.Sep 12 2023, 11:57 AM

Mentioned in SAL (#wikimedia-operations) [2023-09-12T11:57:27Z] <godog> pool thanos[12]001 for thanos.w.o - T341999

gerritbot added a comment.Sep 12 2023, 12:02 PM

Change 956840 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] cumin: add titan aliases

https://gerrit.wikimedia.org/r/956840

gerritbot added a comment.Sep 12 2023, 12:30 PM

Change 956866 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] alerting_host: add titan to thanos-query hosts

https://gerrit.wikimedia.org/r/956866

gerritbot added a comment.Sep 12 2023, 12:30 PM

Change 956867 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] titan: move pyrra off thanos role

https://gerrit.wikimedia.org/r/956867

gerritbot added a comment.Sep 12 2023, 12:43 PM

Change 956866 merged by Filippo Giunchedi:

[operations/puppet@production] alerting_host: add titan to thanos-query hosts

https://gerrit.wikimedia.org/r/956866

gerritbot added a comment.Sep 12 2023, 1:14 PM

Change 956840 merged by Filippo Giunchedi:

[operations/puppet@production] cumin: add titan aliases

https://gerrit.wikimedia.org/r/956840

fgiunchedi closed this task as Resolved.Sep 12 2023, 1:24 PM
fgiunchedi claimed this task.

New hosts are in service, resolving

gerritbot added a comment.Sep 12 2023, 1:26 PM

Change 956867 merged by Filippo Giunchedi:

[operations/puppet@production] titan: move pyrra off thanos role

https://gerrit.wikimedia.org/r/956867

Maintenance_bot removed a project: Patch-For-Review.Sep 12 2023, 1:31 PM
fgiunchedi reopened this task as Open.Sep 12 2023, 1:42 PM

I was a little too hasty here, I forgot we need raid0 on these hosts to be able to store blocks to be compacted, will need to reimage the hosts

gerritbot added a comment.Sep 12 2023, 1:44 PM

Change 956887 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] install_server: use raid0 for titan

https://gerrit.wikimedia.org/r/956887

gerritbot added a project: Patch-For-Review.Sep 12 2023, 1:44 PM
gerritbot added a comment.Sep 12 2023, 1:51 PM

Change 956887 merged by Filippo Giunchedi:

[operations/puppet@production] install_server: use raid0 for titan

https://gerrit.wikimedia.org/r/956887

Maintenance_bot removed a project: Patch-For-Review.Sep 12 2023, 2:11 PM
ops-monitoring-bot added a comment.Sep 13 2023, 7:39 AM

Cookbook cookbooks.sre.hosts.reimage was started by filippo@cumin1001 for host titan1001.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Sep 13 2023, 7:46 AM

Cookbook cookbooks.sre.hosts.reimage was started by filippo@cumin1001 for host titan2001.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Sep 13 2023, 7:51 AM

Cookbook cookbooks.sre.hosts.reimage started by filippo@cumin1001 for host titan1001.eqiad.wmnet with OS bookworm executed with errors:

  • titan1001 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Sep 13 2023, 7:54 AM

Cookbook cookbooks.sre.hosts.reimage was started by filippo@cumin1001 for host titan1001.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Sep 13 2023, 8:07 AM

Cookbook cookbooks.sre.hosts.reimage started by filippo@cumin1001 for host titan1001.eqiad.wmnet with OS bookworm executed with errors:

  • titan1001 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Sep 13 2023, 8:08 AM

Cookbook cookbooks.sre.hosts.reimage was started by filippo@cumin1001 for host titan1001.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Sep 13 2023, 8:14 AM

Cookbook cookbooks.sre.hosts.reimage started by filippo@cumin1001 for host titan2001.codfw.wmnet with OS bookworm executed with errors:

  • titan2001 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Sep 13 2023, 8:14 AM

Cookbook cookbooks.sre.hosts.reimage was started by filippo@cumin1001 for host titan2001.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Sep 13 2023, 8:47 AM

Cookbook cookbooks.sre.hosts.reimage started by filippo@cumin1001 for host titan1001.eqiad.wmnet with OS bookworm completed:

  • titan1001 (WARN)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309130825_filippo_2981305_titan1001.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (row E/F)
ops-monitoring-bot added a comment.Sep 13 2023, 9:11 AM

Cookbook cookbooks.sre.hosts.reimage started by filippo@cumin1001 for host titan2001.codfw.wmnet with OS bookworm completed:

  • titan2001 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309130833_filippo_3001079_titan2001.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Sep 13 2023, 9:14 AM

Cookbook cookbooks.sre.hosts.reimage was started by filippo@cumin1001 for host titan2002.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Sep 13 2023, 9:16 AM

Cookbook cookbooks.sre.hosts.reimage was started by filippo@cumin1001 for host titan1002.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Sep 13 2023, 10:06 AM

Cookbook cookbooks.sre.hosts.reimage started by filippo@cumin1001 for host titan2002.codfw.wmnet with OS bookworm executed with errors:

  • titan2002 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Sep 13 2023, 10:06 AM

Cookbook cookbooks.sre.hosts.reimage started by filippo@cumin1001 for host titan1002.eqiad.wmnet with OS bookworm executed with errors:

  • titan1002 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Sep 13 2023, 10:06 AM

Cookbook cookbooks.sre.hosts.reimage was started by filippo@cumin1001 for host titan2002.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Sep 13 2023, 10:06 AM

Cookbook cookbooks.sre.hosts.reimage was started by filippo@cumin1001 for host titan1002.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Sep 13 2023, 10:26 AM

Cookbook cookbooks.sre.hosts.reimage started by filippo@cumin1001 for host titan2002.codfw.wmnet with OS bookworm executed with errors:

  • titan2002 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Sep 13 2023, 10:27 AM

Cookbook cookbooks.sre.hosts.reimage was started by filippo@cumin1001 for host titan2002.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Sep 13 2023, 10:34 AM

Cookbook cookbooks.sre.hosts.reimage started by filippo@cumin1001 for host titan2002.codfw.wmnet with OS bookworm executed with errors:

  • titan2002 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Sep 13 2023, 10:34 AM

Cookbook cookbooks.sre.hosts.reimage was started by filippo@cumin1001 for host titan2002.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Sep 13 2023, 10:40 AM

Cookbook cookbooks.sre.hosts.reimage started by filippo@cumin1001 for host titan1002.eqiad.wmnet with OS bookworm completed:

  • titan1002 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309131024_filippo_3151773_titan1002.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Sep 13 2023, 11:11 AM

Cookbook cookbooks.sre.hosts.reimage started by filippo@cumin1001 for host titan2002.codfw.wmnet with OS bookworm completed:

  • titan2002 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309131054_filippo_3167758_titan2002.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
fgiunchedi closed this task as Resolved.Sep 13 2023, 12:17 PM

Hosts reimaged with raid0, resolving

• lmata moved this task from Up next to Done on the SRE Observability (FY2023/2024-Q1) board.Oct 9 2023, 4:28 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits