Page MenuHomePhabricator
Create Task
Maniphest T261489

Upgrade debmonitor to Buster
Closed, ResolvedPublic
Actions

Description

Upgrade debmonitor to Buster. Since those are VMs it's probably simplest to create new VMs, setup a Buster installation in parallel and then switch over.

The clients are detecting the debmonitor server via a discovery record. The docker-report image injection script also uses debmonitor-client, so doesn't need a separate change.

Details

SubjectRepoBranchLines +/-
Remove debmonitor1001/2001operations/dnsmaster+0 -4
Remove debmonitor1001/2001operations/puppetproduction+1 -15
cr: update debmonitor IPs in firewall rules (#2)operations/homer/publicmaster+5 -2
cr: update debmonitor IPs in firewall rulesoperations/homer/publicmaster+5 -5
Switch debmonitor to debmonitor1002operations/dnsmaster+2 -2
Add debmonitor1002 as debmonitor serveroperations/puppetproduction+2 -9
Add debmonitor::server role to debmonitor2002operations/puppetproduction+2 -1
Create artifacts for 0.2.7 on Busteroperations/software/debmonitor/deploymaster+0 -0
Customize query in gerrit

Related Objects

Event Timeline

MoritzMuehlenhoff created this task.Aug 28 2020, 10:04 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 28 2020, 10:04 AM
gerritbot added a comment.Aug 31 2020, 7:10 AM

Change 623229 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Add debmonitor::server role to debmonitor2002

https://gerrit.wikimedia.org/r/623229

gerritbot added a project: Patch-For-Review.Aug 31 2020, 7:10 AM
MoritzMuehlenhoff triaged this task as Medium priority.Aug 31 2020, 7:13 AM
MoritzMuehlenhoff updated the task description. (Show Details)
gerritbot added a comment.Aug 31 2020, 11:17 AM

Change 623335 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/software/debmonitor/deploy@master] Create artifacts for 0.2.7 on Buster

https://gerrit.wikimedia.org/r/623335

gerritbot added a comment.Aug 31 2020, 11:44 AM

Change 623335 merged by Muehlenhoff:
[operations/software/debmonitor/deploy@master] Create artifacts for 0.2.7 on Buster

https://gerrit.wikimedia.org/r/623335

gerritbot added a comment.Aug 31 2020, 12:23 PM

Change 623229 merged by Muehlenhoff:
[operations/puppet@production] Add debmonitor::server role to debmonitor2002

https://gerrit.wikimedia.org/r/623229

Maintenance_bot removed a project: Patch-For-Review.Aug 31 2020, 1:10 PM
gerritbot added a comment.Sep 3 2020, 1:53 PM

Change 624066 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Add debmonitor1002 as debmonitor server

https://gerrit.wikimedia.org/r/624066

gerritbot added a project: Patch-For-Review.Sep 3 2020, 1:53 PM
gerritbot added a comment.Sep 3 2020, 2:16 PM

Change 624066 merged by Muehlenhoff:
[operations/puppet@production] Add debmonitor1002 as debmonitor server

https://gerrit.wikimedia.org/r/624066

Maintenance_bot removed a project: Patch-For-Review.Sep 3 2020, 3:11 PM
gerritbot added a comment.Sep 7 2020, 7:29 AM

Change 625591 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/dns@master] Switch debmonitor to debmonitor1002

https://gerrit.wikimedia.org/r/625591

gerritbot added a project: Patch-For-Review.Sep 7 2020, 7:29 AM
gerritbot added a comment.Sep 7 2020, 9:24 AM

Change 625591 merged by Muehlenhoff:
[operations/dns@master] Switch debmonitor to debmonitor1002

https://gerrit.wikimedia.org/r/625591

Maintenance_bot removed a project: Patch-For-Review.Sep 7 2020, 10:10 AM
MoritzMuehlenhoff added a comment.Sep 7 2020, 10:25 AM

debmonitor.wikimedia.org is now served by debmonitor1002 running Buster and everything is working well. I'm keeping the old VMs throughout the week before tearing them down.

gerritbot added a comment.Sep 8 2020, 8:35 AM

Change 625833 had a related patch set uploaded (by Volans; owner: Volans):
[operations/homer/public@master] cr: update debmonitor IPs in firewall rules

https://gerrit.wikimedia.org/r/625833

gerritbot added a project: Patch-For-Review.Sep 8 2020, 8:35 AM
gerritbot added a comment.Sep 8 2020, 8:39 AM

Change 625833 merged by jenkins-bot:
[operations/homer/public@master] cr: update debmonitor IPs in firewall rules

https://gerrit.wikimedia.org/r/625833

Stashbot added a comment.Sep 8 2020, 8:45 AM

Mentioned in SAL (#wikimedia-operations) [2020-09-08T08:45:01Z] <volans> running homer 'cr*eqiad*' commit "Update debmonitor IPs, T261489"

gerritbot added a comment.Sep 8 2020, 9:00 AM

Change 625840 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/dns@master] Remove debmonitor1001/2001

https://gerrit.wikimedia.org/r/625840

gerritbot added a comment.Sep 8 2020, 9:00 AM

Change 625841 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Remove debmonitor1001/2001

https://gerrit.wikimedia.org/r/625841

Volans mentioned this in rOHPU0ecbb052ced3: cr: update debmonitor IPs in firewall rules.Sep 8 2020, 9:03 AM
gerritbot added a comment.Sep 8 2020, 9:24 AM

Change 625844 had a related patch set uploaded (by Volans; owner: Volans):
[operations/homer/public@master] cr: update debmonitor IPs in firewall rules (#2)

https://gerrit.wikimedia.org/r/625844

gerritbot added a comment.Sep 8 2020, 9:30 AM

Change 625844 merged by jenkins-bot:
[operations/homer/public@master] cr: update debmonitor IPs in firewall rules (#2)

https://gerrit.wikimedia.org/r/625844

Stashbot added a comment.Sep 8 2020, 9:37 AM

Mentioned in SAL (#wikimedia-operations) [2020-09-08T09:37:51Z] <volans> running homer 'cr*eqiad*' commit "Update debmonitor IPs (#2), T261489"

ops-monitoring-bot added a comment.Sep 14 2020, 11:27 AM

cookbooks.sre.hosts.decommission executed by volans@cumin1001 for hosts: debmonitor2001.codfw.wmnet

  • debmonitor2001.codfw.wmnet (WARN)
    • Downtimed host on Icinga
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.codfw.wmnet to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Site codfw DNS records not yet migrated to the automatic system, manual patch required
ops-monitoring-bot added a comment.Sep 14 2020, 11:36 AM

cookbooks.sre.hosts.decommission executed by jmm@cumin1001 for hosts: debmonitor1001.eqiad.wmnet

  • debmonitor1001.eqiad.wmnet (WARN)
    • Downtimed host on Icinga
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.eqiad.wmnet to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Site eqiad DNS records not yet migrated to the automatic system, manual patch required
gerritbot added a comment.Sep 14 2020, 11:44 AM

Change 625841 merged by Muehlenhoff:
[operations/puppet@production] Remove debmonitor1001/2001

https://gerrit.wikimedia.org/r/625841

ops-monitoring-bot added a comment.Sep 14 2020, 12:09 PM

cookbooks.sre.hosts.decommission executed by volans@cumin1001 for hosts: debmonitor2001.codfw.wmnet

  • debmonitor2001.codfw.wmnet (FAIL)
    • Failed downtime host on Icinga (likely already removed)
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.codfw.wmnet to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Site codfw DNS records not yet migrated to the automatic system, manual patch required

ERROR: some step on some host failed, check the bolded items above

gerritbot added a comment.Sep 14 2020, 12:44 PM

Change 625840 merged by Muehlenhoff:
[operations/dns@master] Remove debmonitor1001/2001

https://gerrit.wikimedia.org/r/625840

MoritzMuehlenhoff closed this task as Resolved.Sep 14 2020, 12:46 PM

The old stretch instances (debmonitor1001/2001) have been removed.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits