Page MenuHomePhabricator
Create Task
Maniphest T327862

Use mgmt_junos on all network devices
Closed, ResolvedPublic
Actions

Description

This feature is already used on all the L3 devices, for consistency and leaner automation (currently behind flags) we should roll it to all the L2 devices as well once they're upgrades (only eqiad/codfw prod switches are left).

Details

SubjectRepoBranchLines +/-
gNMI: don't use mgmt_junos on asw1-eqsin and faswoperations/homer/publicmaster+2 -0
Introduce mgmt_junos variableoperations/homer/publicmaster+18 -20
mgmt_junos: create default route if no mgmt_junos configuredoperations/homer/publicmaster+8 -0
Configure mgmt_junos on L2 switchesoperations/homer/publicmaster+15 -34
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedayounsiT327862 Use mgmt_junos on all network devices
 OpenNoneT316539 Upgrade network devices to Junos 20+
 ResolvedayounsiT316532 Upgrade POPs asw to Junos 21
 ResolvedayounsiT323094 asw1-eqsin: VC mastership change
 ResolvedNoneT295690 Upgrade core routers to Junos 21+
 ResolvedPapaulT316529 Upgrade management routers and switches to Junos 21
 ResolvedPapaulT295691 Upgrade pfw to Junos 20+
 ResolvedPapaulT316542 Upgrade fasw to Junos 21
 OpencmooneyT316544 Upgrade cloudsw1-c8-eqiad and cloudsw1-d5-eqiad to Junos 20+
 OpendcaroT297083 [ceph] Getting rack level HA
 ResolvedRequestCmjohnsonT303058 hw troubleshooting: move cloudcephmon1003.eqiad.wmnet from rack B2 to rack C8
 ResolvedCmjohnsonT304096 move cloudcephmon1002.eqiad.wmnet from rack B4 to rack D5
 ResolvednskaggsT329498 [ceph] Move cloudcephosd1001 (b7) and cloudcephosd1002 (b4) to rack e4
ResolvedBUG REPORTdcaroT329535 Cloud Ceph outage 2023-02-13
 In ProgressdcaroT329709 [cookbooks.ceph] Add a cookbook to drain a ceph osd in a safe manner
 ResolveddcaroT329711 [ceph] Add monitoring for inter-osd/mon/cloudvirt connectivity
 OpendcaroT329778 [ceph] Investigate if there's a way to degrade instead of failing when jumbo frames are being dropped in the network
 ResolvedcmooneyT329799 Add network-layer protections to avoid inadvertently lowering IRB MTU
 ResolvednskaggsT329502 [ceph] Move cloudcephosd1003 (b2) to rack e4 and cloudcephosd1004 (c8) to rack f4
ResolvednskaggsT329504 [ceph] Move cloudcephosd1005 (c8) and cloudcephosd1010 (d5) to rack f4
ResolveddcaroT329507 [ceph] Test crush tree with rack level HA on codfw
 ResolvedRequestPapaulT330754 hw troubleshooting: Link hard down (probably cable) for cloudcephosd2002-dev.codfw.wmnet
 ResolveddcaroT331141 Change crushmap in eqiad to have rack HA
 In ProgressdcaroT331145 [cookbooks] adapt to having an extra level of buckets in the crushmap
 OpenNoneT330733 Move coludcephmon1001 from B7 to rack F4
 OpendcaroT331636 [cookbooks.ceph] create a script to get the list of rbd images affected by stuck/inactive PGs
 OpenNoneT336845 puppet: profile::auto_restarts::service: have a way to don't deploy the systemd timers
 ResolvedayounsiT327248 eqiad/codfw virtual-chassis upgrades
 ResolvedayounsiT327925 codfw row A switches upgrade
 ResolvedMarosteguiT327997 Switchover s1 master (db2103 -> db2112)
 ResolvedMarosteguiT327998 Switchover s2 master (db2104 -> db2107)
 ResolvedMarosteguiT327999 Switchover s3 master (db2105 -> db2127)
 ResolvedMarosteguiT328000 Switchover s7 master (db2121 -> db2118)
 ResolvedMarosteguiT328001 Switchover x2 master (db2142 -> db2144)
 ResolvedayounsiT327991 codfw row B switches upgrade
 ResolvedMarosteguiT328022 Switchover s4 master (db2110 -> db2140)
 ResolvedMarosteguiT328023 Switchover s5 master (db2123 -> db2113)
 ResolvedMarosteguiT328024 Switchover s8 master (db2161 -> db2165)
 ResolvedcmooneyT329073 eqiad row A switches upgrade
 ResolvedMarosteguiT329141 Switchover m3 master db1164 -> db1159
 ResolvedMarosteguiT329143 Move db1164 to m1
 ResolvedMarosteguiT329259 Switchover m1 master (db1176 -> db1164)
 ResolvedMarosteguiT331382 Move db1101 to m3 and make it master
 ResolvedMarosteguiT331384 Switchover m3 master db1159 -> db1101
 ResolvedMarosteguiT331387 Switchover m3 master db1101 -> db1159
 ResolvedMarosteguiT331626 Mailman hasn't delivered emails since 2023-03-07 14 UTC (was: reviewer-bot is not working)
 ResolvedayounsiT330165 eqiad row B switches upgrade
 InvalidMarosteguiT330977 Move db1183 to m1
 ResolvedMarosteguiT331510 Switchover m1 master (db1164 -> db1101)
 ResolvedMarosteguiT331511 Move db1101 to m1
 ResolvedMarosteguiT330847 Switchover m5 master (db1183 -> db1176)
 ResolvedayounsiT331882 eqiad row C switches upgrade
 ResolvedLadsgroupT333123 Switchover m1 master (db1101 -> db1164)
 ResolvedherronT333478 failover alert1001 to alert2001
 ResolvedherronT333837 failover alert2001 to alert1001
 DeclinedherronT333838 alerting_host: Reduced availability for job icinga-am after failover event
 OpenherronT333855 vopsbot needed manual restart after alerting hosts failover
 ResolvedcmooneyT333377 eqiad row D switches upgrade
 ResolvedayounsiT334049 codfw row C switches upgrade
 ResolvedayounsiT335042 codfw row D switches upgrade

Event Timeline

ayounsi triaged this task as Low priority.Jan 25 2023, 8:04 AM
ayounsi created this task.
Restricted Application added a project: Infrastructure-Foundations. · View Herald TranscriptJan 25 2023, 8:04 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
ayounsi added a subtask: T316539: Upgrade network devices to Junos 20+.Jan 25 2023, 8:04 AM
Maintenance_bot added a project: SRE.Jan 25 2023, 8:29 AM
Peachey88 updated the task description. (Show Details)Jan 25 2023, 8:51 AM
ayounsi moved this task from Backlog to Next quarter on the netops board.Feb 20 2023, 3:33 PM
ayounsi moved this task from Next quarter to This quarter on the netops board.Apr 5 2023, 12:18 PM
gerritbot added a comment.May 16 2023, 3:39 PM

Change 920311 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/homer/public@master] Configure mgmt_junos on L2 switches

https://gerrit.wikimedia.org/r/920311

gerritbot added a project: Patch-For-Review.May 16 2023, 3:39 PM
gerritbot added a comment.May 22 2023, 4:53 PM

Change 920311 merged by jenkins-bot:

[operations/homer/public@master] Configure mgmt_junos on L2 switches

https://gerrit.wikimedia.org/r/920311

Maintenance_bot removed a project: Patch-For-Review.May 22 2023, 5:10 PM
gerritbot added a comment.May 22 2023, 6:53 PM

Change 922161 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/homer/public@master] Introduce mgmt_junos variable

https://gerrit.wikimedia.org/r/922161

gerritbot added a project: Patch-For-Review.May 22 2023, 6:53 PM
ayounsi added a comment.May 22 2023, 6:57 PM

the fasw and asw1-eqsin switches didn't create the mgmt_junos routing instance as they should have.

https://gerrit.wikimedia.org/r/922161 works around the issue by adding a (hopefully) temporary variable to control the mgmt_junos routing instance.
Filtering on device type was not possible (as they're all asw) and filtering on device name/prefix was getting ugly as they're multiple devices impacted. This also allows to cleanly disable mgmt_junos on mr devices and on devices with no mgmt network (eg. knams routers)

ayounsi mentioned this in T332395: Upgrade asw1-eqsin.May 22 2023, 6:59 PM
gerritbot added a comment.May 22 2023, 7:50 PM

Change 922161 merged by jenkins-bot:

[operations/homer/public@master] Introduce mgmt_junos variable

https://gerrit.wikimedia.org/r/922161

gerritbot added a comment.May 22 2023, 7:58 PM

Change 922164 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/homer/public@master] mgmt_junos: create default route if no mgmt_junos configured

https://gerrit.wikimedia.org/r/922164

gerritbot added a comment.May 22 2023, 7:59 PM

Change 922164 merged by jenkins-bot:

[operations/homer/public@master] mgmt_junos: create default route if no mgmt_junos configured

https://gerrit.wikimedia.org/r/922164

Maintenance_bot removed a project: Patch-For-Review.May 22 2023, 8:11 PM
ayounsi closed this task as Resolved.May 22 2023, 8:24 PM
ayounsi claimed this task.

Going to close this task as this is as far as we can go due to the fasw switches not being easily upgraded.

ayounsi reopened this task as Open.Aug 30 2023, 10:45 AM

Re-opening as the fasw got upgraded since, so we can enable mgmt_junos

Stashbot added a comment.Aug 30 2023, 10:57 AM

Mentioned in SAL (#wikimedia-operations) [2023-08-30T10:57:39Z] <XioNoX> enable mgmt_junos on fasw-c-codfw - T327862

ayounsi closed this task as Resolved.Aug 30 2023, 12:03 PM

Nevermind, still doesn't work on the fasw.

gerritbot added a comment.Aug 30 2023, 12:05 PM

Change 953591 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/homer/public@master] gNMI: don't use mgmt_junos on asw1-eqsin and fasw

https://gerrit.wikimedia.org/r/953591

gerritbot added a project: Patch-For-Review.Aug 30 2023, 12:05 PM
gerritbot added a comment.Aug 30 2023, 12:07 PM

Change 953591 merged by jenkins-bot:

[operations/homer/public@master] gNMI: don't use mgmt_junos on asw1-eqsin and fasw

https://gerrit.wikimedia.org/r/953591

Maintenance_bot removed a project: Patch-For-Review.Aug 30 2023, 12:10 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL