Page MenuHomePhabricator
Create Task
Maniphest T335424

kafkamon: upgrade to bullseye
Closed, ResolvedPublic
Actions

Description

Tracking task for upgrading kafkamon hosts to bullseye

High level checklist:

  • Create bullseye VMs
    • kafkamon1003
    • kafkamon2003
  • Create kafka::monitoring_bullseye role
  • Validate config on bullseye
  • Cut prometheus configs over to kafka::monitoring_bullseye class
  • Retire buster kafkamon VMs

Details

SubjectRepoBranchLines +/-
kafkamon: cleanup buster classesoperations/puppetproduction+1 -20
kafkamon: cut over to bullseye exportersoperations/puppetproduction+4 -4
kafkamon: add kafkamon[12]003 to fw allow listoperations/puppetproduction+4 -0
kafkamon: transition to firewall definitionoperations/puppetproduction+14 -3
kafkamon: add monitoring bullseye clustersoperations/puppetproduction+7 -0
kafkamon: add monitoring bullseye yamloperations/puppetproduction+20 -0
kafkamon: add bullseye role and node assignmentsoperations/puppetproduction+16 -0
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 Openandrea.denisseT324725 Observability Bookworm upgrades
 ResolvedherronT335424 kafkamon: upgrade to bullseye
 ResolvedherronT335426 2 VMs for kafkamon

Event Timeline

herron removed andrea.denisse as the assignee of this task.Apr 26 2023, 4:19 PM
herron created this task.
gerritbot added a comment.Apr 26 2023, 4:58 PM

Change 912341 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] kafkamon: add bullseye role and node assignments

https://gerrit.wikimedia.org/r/912341

gerritbot added a project: Patch-For-Review.Apr 26 2023, 4:58 PM
gerritbot added a comment.Apr 27 2023, 3:46 PM

Change 912341 merged by Herron:

[operations/puppet@production] kafkamon: add bullseye role and node assignments

https://gerrit.wikimedia.org/r/912341

Maintenance_bot removed a project: Patch-For-Review.Apr 27 2023, 4:10 PM
gerritbot added a comment.Apr 27 2023, 5:28 PM

Change 912930 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] kafkamon: add monitoring bullseye yaml

https://gerrit.wikimedia.org/r/912930

gerritbot added a project: Patch-For-Review.Apr 27 2023, 5:28 PM

Change 912930 merged by Herron:

[operations/puppet@production] kafkamon: add monitoring bullseye yaml

https://gerrit.wikimedia.org/r/912930

Maintenance_bot removed a project: Patch-For-Review.Apr 27 2023, 5:30 PM
gerritbot added a comment.Apr 27 2023, 5:33 PM

Change 912932 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] kafkamon: add monitoring bullseye clusters

https://gerrit.wikimedia.org/r/912932

gerritbot added a project: Patch-For-Review.Apr 27 2023, 5:33 PM

Change 912932 merged by Herron:

[operations/puppet@production] kafkamon: add monitoring bullseye clusters

https://gerrit.wikimedia.org/r/912932

Maintenance_bot removed a project: Patch-For-Review.Apr 27 2023, 6:10 PM
gerritbot added a comment.Apr 27 2023, 9:04 PM

Change 912979 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] kafkamon: transition to firewall definition

https://gerrit.wikimedia.org/r/912979

gerritbot added a project: Patch-For-Review.Apr 27 2023, 9:04 PM
lmata edited projects, added SRE Observability (FY2022/2023-Q4); removed SRE Observability (FY2022/2023-Q3).May 2 2023, 1:44 PM
gerritbot added a comment.May 2 2023, 5:00 PM

Change 912979 merged by Herron:

[operations/puppet@production] kafkamon: transition to firewall definition

https://gerrit.wikimedia.org/r/912979

Maintenance_bot removed a project: Patch-For-Review.May 2 2023, 5:10 PM
herron updated the task description. (Show Details)May 3 2023, 1:22 PM
gerritbot added a comment.May 3 2023, 1:51 PM

Change 914787 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] kafkamon: add kafkamon[12]003 to fw allow list

https://gerrit.wikimedia.org/r/914787

gerritbot added a project: Patch-For-Review.May 3 2023, 1:51 PM
gerritbot added a comment.May 3 2023, 4:03 PM

Change 914787 merged by Herron:

[operations/puppet@production] kafkamon: add kafkamon[12]003 to fw allow list

https://gerrit.wikimedia.org/r/914787

Maintenance_bot removed a project: Patch-For-Review.May 3 2023, 4:10 PM
herron updated the task description. (Show Details)May 3 2023, 5:37 PM
gerritbot added a comment.May 3 2023, 5:40 PM

Change 914876 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] kafkamon: cut over to bullseye exporters

https://gerrit.wikimedia.org/r/914876

gerritbot added a project: Patch-For-Review.May 3 2023, 5:40 PM
Stashbot added a comment.May 4 2023, 1:48 PM

Mentioned in SAL (#wikimedia-operations) [2023-05-04T13:48:26Z] <herron> switching to bullseye kafka monitoring hosts T335424

gerritbot added a comment.May 4 2023, 1:49 PM

Change 914876 merged by Herron:

[operations/puppet@production] kafkamon: cut over to bullseye exporters

https://gerrit.wikimedia.org/r/914876

Maintenance_bot removed a project: Patch-For-Review.May 4 2023, 2:10 PM
herron closed subtask T335426: 2 VMs for kafkamon as Resolved.May 4 2023, 2:18 PM
herron updated the task description. (Show Details)
gerritbot added a comment.May 4 2023, 2:45 PM

Change 915694 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] kafkamon: cleanup buster classes

https://gerrit.wikimedia.org/r/915694

gerritbot added a project: Patch-For-Review.May 4 2023, 2:45 PM
gerritbot added a comment.May 4 2023, 3:57 PM

Change 915694 merged by Herron:

[operations/puppet@production] kafkamon: cleanup buster classes

https://gerrit.wikimedia.org/r/915694

herron closed this task as Resolved.May 4 2023, 4:02 PM
herron claimed this task.
herron triaged this task as Medium priority.
herron updated the task description. (Show Details)

done!

Maintenance_bot removed a project: Patch-For-Review.May 4 2023, 4:10 PM
lmata added a comment.May 5 2023, 2:14 PM
In T335424#8827165, @herron wrote:

done!

thank you!

fgiunchedi reopened this task as Open.May 23 2023, 12:25 PM
fgiunchedi added a subscriber: fgiunchedi.

I noticed the puppet failed for kafkamon*002 (since 20 days!) and indeed the old VMs are still up:

(1) kafkamon2002.codfw.wmnet                                                                              
----- OUTPUT of 'uptime' -----                                                                            
 12:24:03 up 159 days, 20:04,  1 user,  load average: 0.69, 0.28, 0.23                        
(1) kafkamon1002.eqiad.wmnet                                                                              
----- OUTPUT of 'uptime' -----                                                                            
 12:24:03 up 159 days, 20:12,  0 users,  load average: 1.24, 0.78, 0.73
herron added a comment.May 23 2023, 1:45 PM

Good catch thanks, I had to take a look through my history on cumin1001 because I remember decomming these. Turns out ran the cookbook with the -d (dry run) flag enabled 🤦‍♂️ Will re-run these decoms now.

ops-monitoring-bot added a comment.May 23 2023, 1:55 PM

cookbooks.sre.hosts.decommission executed by herron@cumin1001 for hosts: kafkamon1002.eqiad.wmnet

  • kafkamon1002.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster eqiad to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster eqiad to Netbox
ops-monitoring-bot added a comment.May 23 2023, 2:05 PM

cookbooks.sre.hosts.decommission executed by herron@cumin1001 for hosts: kafkamon2002.codfw.wmnet

  • kafkamon2002.codfw.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster codfw to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster codfw to Netbox
  • COMMON_STEPS (FAIL)
    • Failed to run the sre.dns.netbox cookbook, run it manually

ERROR: some step on some host failed, check the bolded items above

herron added a comment.EditedMay 23 2023, 2:09 PM
In T335424#8873679, @ops-monitoring-bot wrote:

cookbooks.sre.hosts.decommission executed by herron@cumin1001 for hosts: kafkamon2002.codfw.wmnet

  • kafkamon2002.codfw.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster codfw to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster codfw to Netbox
  • COMMON_STEPS (FAIL)
    • Failed to run the sre.dns.netbox cookbook, run it manually

ERROR: some step on some host failed, check the bolded items above

I think we're good here. The VM was decommed, but the netbox dns cookbook ran failed due to multiple people running it at the same time. The dns record was removed, just not by this cookbook run

herron closed this task as Resolved.May 23 2023, 2:10 PM
lmata moved this task from Inbox to Done on the SRE Observability (FY2022/2023-Q4) board.Jul 18 2023, 4:59 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL