Page MenuHomePhabricator
Create Task
Maniphest T297595

Upgrade netflow VMs to Bullseye
Closed, ResolvedPublic
Actions

Description

While working on T263277 I hit a sfacctd bug that is solved in its Bullseye package.

Easiest at this point is to re-create the netflow* VMs to Bullseye as this will to happen at some point anyway.

Details

SubjectRepoBranchLines +/-
Remove old netflow hosts from kafka jumbo acloperations/puppetproduction+0 -5
Update all netflow collectors to new VMsoperations/homer/publicmaster+6 -6
Add DHCP for new netflow VMsoperations/puppetproduction+16 -17
Add new netflow hosts to Kafka jumbo ACLoperations/puppetproduction+5 -0
Update netflow collector for codfw/eqdfw to netflow2002operations/homer/publicmaster+2 -2
Include new netflow VMs in site.ppoperations/puppetproduction+1 -1
Add netflow2002 to DHCPoperations/puppetproduction+6 -0
Make netboot.cfg generic for netflow VMsoperations/puppetproduction+1 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

ayounsi triaged this task as Medium priority.Dec 13 2021, 12:24 PM
ayounsi created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 13 2021, 12:24 PM
ayounsi added a parent task: T263277: Collect netflow data for internal traffic.Dec 13 2021, 12:24 PM
gerritbot added a comment.Dec 13 2021, 12:30 PM

Change 746853 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/puppet@production] Make netboot.cfg generic for netflow VMs

https://gerrit.wikimedia.org/r/746853

gerritbot added a project: Patch-For-Review.Dec 13 2021, 12:30 PM
gerritbot added a comment.Dec 13 2021, 12:39 PM

Change 746853 merged by Ayounsi:

[operations/puppet@production] Make netboot.cfg generic for netflow VMs

https://gerrit.wikimedia.org/r/746853

Maintenance_bot added a project: SRE.Dec 13 2021, 12:45 PM
gerritbot added a comment.Dec 13 2021, 1:08 PM

Change 746862 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/puppet@production] Add netflow2002 to DHCP

https://gerrit.wikimedia.org/r/746862

gerritbot added a comment.Dec 13 2021, 1:09 PM

Change 746862 merged by Ayounsi:

[operations/puppet@production] Add netflow2002 to DHCP

https://gerrit.wikimedia.org/r/746862

Maintenance_bot removed a project: Patch-For-Review.Dec 13 2021, 1:11 PM
gerritbot added a comment.Dec 13 2021, 1:22 PM

Change 746863 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/puppet@production] Include new netflow VMs in site.pp

https://gerrit.wikimedia.org/r/746863

gerritbot added a project: Patch-For-Review.Dec 13 2021, 1:22 PM

Change 746863 merged by Ayounsi:

[operations/puppet@production] Include new netflow VMs in site.pp

https://gerrit.wikimedia.org/r/746863

Stashbot added a comment.Dec 13 2021, 2:34 PM

Mentioned in SAL (#wikimedia-operations) [2021-12-13T14:34:33Z] <moritzm> imported fastnetmon 1.1.7+deb11u1 for bullseye-wikimedia https://phabricator.wikimedia.org/T297595

gerritbot added a comment.Dec 14 2021, 8:21 AM

Change 747047 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/homer/public@master] Update netflow collector for codfw/eqdfw to netflow2002

https://gerrit.wikimedia.org/r/747047

gerritbot added a comment.Dec 14 2021, 8:23 AM

Change 747047 merged by Ayounsi:

[operations/homer/public@master] Update netflow collector for codfw/eqdfw to netflow2002

https://gerrit.wikimedia.org/r/747047

gerritbot added a comment.Dec 14 2021, 8:48 AM

Change 747050 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/puppet@production] Add new netflow hosts to Kafka jumbo ACL

https://gerrit.wikimedia.org/r/747050

gerritbot added a comment.Dec 14 2021, 8:55 AM

Change 747050 merged by Ayounsi:

[operations/puppet@production] Add new netflow hosts to Kafka jumbo ACL

https://gerrit.wikimedia.org/r/747050

gerritbot added a comment.Dec 14 2021, 9:05 AM

Change 747052 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/puppet@production] Add DHCP for new netflow VMs

https://gerrit.wikimedia.org/r/747052

gerritbot added a comment.Dec 14 2021, 9:07 AM

Change 747052 merged by Ayounsi:

[operations/puppet@production] Add DHCP for new netflow VMs

https://gerrit.wikimedia.org/r/747052

gerritbot added a comment.Dec 14 2021, 9:50 AM

Change 747056 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/homer/public@master] Update all netflow collectors to new VMs

https://gerrit.wikimedia.org/r/747056

gerritbot added a comment.Dec 14 2021, 9:53 AM

Change 747056 merged by jenkins-bot:

[operations/homer/public@master] Update all netflow collectors to new VMs

https://gerrit.wikimedia.org/r/747056

gerritbot added a comment.Dec 14 2021, 10:19 AM

Change 747063 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/puppet@production] Remove old netflow hosts from kafka jumbo acl

https://gerrit.wikimedia.org/r/747063

ops-monitoring-bot added a comment.Dec 14 2021, 10:22 AM

cookbooks.sre.hosts.decommission executed by ayounsi@cumin1001 for hosts: netflow2001.codfw.wmnet

  • netflow2001.codfw.wmnet (FAIL)
    • Downtimed host on Icinga
    • Host steps raised exception: Error while performing request to RAPI

ERROR: some step on some host failed, check the bolded items above

gerritbot added a comment.Dec 14 2021, 10:23 AM

Change 747063 merged by Ayounsi:

[operations/puppet@production] Remove old netflow hosts from kafka jumbo acl

https://gerrit.wikimedia.org/r/747063

ops-monitoring-bot added a comment.Dec 14 2021, 10:36 AM

cookbooks.sre.hosts.decommission executed by ayounsi@cumin1001 for hosts: netflow4001.ulsfo.wmnet

  • netflow4001.ulsfo.wmnet (PASS)
    • Downtimed host on Icinga
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.ulsfo.wmnet to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.ulsfo.wmnet to Netbox
ops-monitoring-bot added a comment.Dec 14 2021, 12:01 PM

cookbooks.sre.hosts.decommission executed by ayounsi@cumin1001 for hosts: netflow2001.codfw.wmnet

  • netflow2001.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.codfw.wmnet to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.codfw.wmnet to Netbox
ops-monitoring-bot added a comment.Dec 14 2021, 12:24 PM

cookbooks.sre.hosts.decommission executed by ayounsi@cumin1001 for hosts: netflow1001.eqiad.wmnet

  • netflow1001.eqiad.wmnet (PASS)
    • Downtimed host on Icinga
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.eqiad.wmnet to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.eqiad.wmnet to Netbox
ops-monitoring-bot added a comment.Dec 14 2021, 12:44 PM

cookbooks.sre.hosts.decommission executed by ayounsi@cumin1001 for hosts: netflow3001.esams.wmnet

  • netflow3001.esams.wmnet (PASS)
    • Downtimed host on Icinga
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.esams.wmnet to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.esams.wmnet to Netbox
ops-monitoring-bot added a comment.Dec 14 2021, 12:57 PM

cookbooks.sre.hosts.decommission executed by ayounsi@cumin1001 for hosts: netflow5001.eqsin.wmnet

  • netflow5001.eqsin.wmnet (PASS)
    • Downtimed host on Icinga
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.eqsin.wmnet to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.eqsin.wmnet to Netbox
ayounsi closed this task as Resolved.Dec 14 2021, 2:15 PM
ayounsi claimed this task.

All done!

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL