Page MenuHomePhabricator
Create Task
Maniphest T191667

Juniper HA audit
Closed, ResolvedPublic
Actions

Description

tl;dr;

  • GRES should be enabled on all Juniper dual-RE devices

Because of Junos limitation with our current configuration, disable GRES from on dual-RE routers, see T191371
Test then re-enable GRES on:

  • cr1-eqiad
  • cr2-eqiad
  • cr1-eqsin
  • cr2-esams
  • Nonstop bridging should NOT be enabled on any devices (only useful for STP, which we don't use)

Remove from:

  • asw2-a-eqiad.mgmt.eqiad.wmnet
  • asw2-b-eqiad.mgmt.eqiad.wmnet
  • asw2-c-eqiad.mgmt.eqiad.wmnet
  • asw2-d-eqiad.mgmt.eqiad.wmnet
  • asw-a-codfw.mgmt.codfw.wmnet
  • asw-b-codfw.mgmt.codfw.wmnet
  • asw-c-codfw.mgmt.codfw.wmnet
  • asw-d-codfw.mgmt.codfw.wmnet
  • fasw-c-eqiad.mgmt.eqiad.wmnet
  • fasw-c-codfw.mgmt.codfw.wmnet
  • asw2-esams.mgmt.esams.wmnet
  • asw1-eqsin.mgmt.eqsin.wmnet
  • Nonstop active routing should be enabled on all switches with > 1 RE (handles LACP, BFD, OSPF, BGP, VRRP)
  • asw1-eqsin.mgmt.eqsin.wmnet
  • Nonstop active routing should NOT be enabled on any router
  • graceful-restart should be enabled on all devices where NSR is not configured (nonstop active routing and graceful-restart are mutually exclusive)
  • Write matching homer changes to make it systematic.

More details on https://www.juniper.net/documentation/en_US/junos/topics/concept/high-availability-features-in-junos-introducing.html

Details

SubjectRepoBranchLines +/-
Remove nonstop-bridging from switchesoperations/homer/publicmaster+4 -0
add graceful-restart to CRsoperations/homer/publicmaster+1 -0
Add graceful-switchover to multiple RE devicesoperations/homer/publicmaster+14 -1
Revert "Depolling eqsin due to router issue"operations/dnsmaster+0 -2
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedayounsiT191667 Juniper HA audit
 ResolvedayounsiT247073 Configure management-instance on routers with Junos > 17.3
Restricted Task
Restricted Task

Event Timeline

ayounsi triaged this task as Medium priority.Apr 6 2018, 9:01 PM
ayounsi created this task.
Restricted Application added a project: SRE. · View Herald TranscriptApr 6 2018, 9:01 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
ayounsi mentioned this in T191940: Investigate 2018-04-10 global traffic drop.Apr 10 2018, 11:50 PM
ayounsi updated the task description. (Show Details)Apr 11 2018, 1:37 AM
gerritbot subscribed.Apr 11 2018, 4:05 PM

Change 425552 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/dns@master] Revert "Depolling eqsin due to router issue"

https://gerrit.wikimedia.org/r/425552

gerritbot added a project: Patch-For-Review.Apr 11 2018, 4:05 PM
gerritbot added a comment.Apr 11 2018, 4:07 PM

Change 425552 merged by BBlack:
[operations/dns@master] Revert "Depolling eqsin due to router issue"

https://gerrit.wikimedia.org/r/425552

ayounsi added a comment.Apr 13 2018, 1:20 AM

From JTAC, the nonstop-routing issue most likely have been caused by a Junos bug where the following commit sometimes enables nonstop-routing before disabling graceful-restart, while they are mutually exclusive.

[edit routing-options]
-   graceful-restart;
+   nonstop-routing;

Suggested approach by JTAC is to disable graceful-restart and enable nonstop-routing in two different commits.

ayounsi updated the task description. (Show Details)Jun 5 2018, 12:45 PM
ayounsi mentioned this in T191371: Enabling graceful-switchover causes core dumps on cr1-codfw.
Krinkle added a project: Wikimedia-Incident.Jul 19 2018, 6:59 AM
Krinkle moved this task from Active investigation to Follow-up prevention on the Wikimedia-Incident board.
• Phabricator_maintenance moved this task from Backlog to Acknowledged on the SRE board.Jan 26 2019, 9:55 PM
Ladsgroup removed a project: Patch-For-Review.May 28 2019, 3:37 PM
ayounsi updated the task description. (Show Details)Feb 18 2020, 3:04 PM
ayounsi added a comment.EditedMar 6 2020, 12:23 PM

Next step here as this is something that needs to be squared away.

Decide what should be configured for which type of devices, respectively:

  • Dual-REs routers: GRES + GR or NSR
    • The GRES bug should be fixed with T247073
    • On the paper, NSR better for RE crash. As it syncs the states, a failover after a crash should be seamless. On the other hand, real life experience shows that syncing states is brittle and things (RE included, don't fail as we want them to)
    • In addition it's not clear if NSR would help in the case where we change the TCP-MSS of an interface as it's mutually exclusive with graceful-restart (see T246338)
    • From the doc "NSR is advantageous in networks in which neighbor routers (or switches) do not support graceful restart protocol extensions." Most likely only a few IXPs peers don't support GR (data point: ~90 out of 400 in AMS-IX, 30 out of 190 in Equinix Ashburn)
    • ISSU is not mature enough
    • For those reasons I think that GRES + graceful-restart is the way to go for now
  • All other statements from the description still stand
  • Once agreed, audit all devices to make sure we capture what needs to change
  • If a device isn't running a compatible Junos version, document it so we take care of it when possible
CDanis mentioned this in T246338: Add graceful-restart to cr2-esams.Mar 6 2020, 8:34 PM
ayounsi added a subtask: T247073: Configure management-instance on routers with Junos > 17.3.Mar 26 2020, 12:41 PM
ayounsi updated the task description. (Show Details)Mar 26 2020, 1:06 PM
gerritbot added a comment.Apr 28 2020, 12:58 PM

Change 592938 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/homer/public@master] Add graceful-switchover multiple RE devices

https://gerrit.wikimedia.org/r/592938

gerritbot added a project: Patch-For-Review.Apr 28 2020, 12:58 PM
gerritbot added a comment.Apr 28 2020, 5:42 PM

Change 577564 had a related patch set uploaded (by Ayounsi; owner: CDanis):
[operations/homer/public@master] add graceful-restart to CRs

https://gerrit.wikimedia.org/r/577564

Krinkle edited projects, added Sustainability (Incident Followup); removed Wikimedia-Incident.Apr 28 2020, 9:50 PM
ayounsi closed subtask T247073: Configure management-instance on routers with Junos > 17.3 as Resolved.Jun 3 2020, 12:10 PM
gerritbot added a comment.Jul 2 2020, 9:16 AM

Change 592938 merged by jenkins-bot:
[operations/homer/public@master] Add graceful-switchover to multiple RE devices

https://gerrit.wikimedia.org/r/c/operations/homer/public/ /592938

gerritbot added a comment.Jul 2 2020, 9:17 AM

Change 577564 merged by jenkins-bot:
[operations/homer/public@master] add graceful-restart to CRs

https://gerrit.wikimedia.org/r/c/operations/homer/public/ /577564

ayounsi mentioned this in rOHPU195ce3e172ef: add graceful-restart to CRs.Jul 2 2020, 9:18 AM
ayounsi updated the task description. (Show Details)Jul 2 2020, 9:59 AM
gerritbot added a comment.Jul 2 2020, 11:44 AM

Change 609139 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/homer/public@master] Remove nonstop-bridging from switches

https://gerrit.wikimedia.org/r/c/operations/homer/public/ /609139

ayounsi updated the task description. (Show Details)Jul 2 2020, 11:50 AM
ayounsi updated the task description. (Show Details)Aug 3 2020, 1:10 PM
Stashbot added a comment.Aug 3 2020, 1:11 PM

Mentioned in SAL (#wikimedia-operations) [2020-08-03T13:11:52Z] <XioNoX> remove nonstop-bridging from asw-a-codfw - T191667

Stashbot added a comment.Aug 3 2020, 1:12 PM

Mentioned in SAL (#wikimedia-operations) [2020-08-03T13:12:58Z] <XioNoX> remove nonstop-bridging from asw-b-codfw - T191667

Stashbot added a comment.Aug 3 2020, 1:14 PM

Mentioned in SAL (#wikimedia-operations) [2020-08-03T13:14:25Z] <XioNoX> remove nonstop-bridging from asw-c-codfw - T191667

Stashbot added a comment.Aug 3 2020, 1:15 PM

Mentioned in SAL (#wikimedia-operations) [2020-08-03T13:15:40Z] <XioNoX> remove nonstop-bridging from asw-d-codfw - T191667

ayounsi updated the task description. (Show Details)Aug 3 2020, 1:18 PM
ayounsi updated the task description. (Show Details)Aug 3 2020, 2:05 PM
Stashbot added a comment.Aug 3 2020, 2:27 PM

Mentioned in SAL (#wikimedia-operations) [2020-08-03T14:27:45Z] <XioNoX> remove nonstop-bridging from fasw-c-codfw - T191667

ayounsi updated the task description. (Show Details)Aug 3 2020, 3:18 PM
Stashbot added a comment.Aug 4 2020, 7:28 AM

Mentioned in SAL (#wikimedia-operations) [2020-08-04T07:28:10Z] <XioNoX> remove nonstop-bridging from asw2-esams - T191667

Stashbot added a comment.Aug 4 2020, 7:29 AM

Mentioned in SAL (#wikimedia-operations) [2020-08-04T07:29:11Z] <XioNoX> remove nonstop-bridging from eqiad asw2 switches - T191667

Stashbot added a comment.Aug 4 2020, 7:32 AM

Mentioned in SAL (#wikimedia-operations) [2020-08-04T07:32:30Z] <XioNoX> remove nonstop-bridging from fasw-c-eqiad switches - T191667

gerritbot added a comment.Aug 4 2020, 7:36 AM

Change 609139 merged by jenkins-bot:
[operations/homer/public@master] Remove nonstop-bridging from switches

https://gerrit.wikimedia.org/r/609139

ayounsi closed this task as Resolved.Aug 4 2020, 7:37 AM
ayounsi updated the task description. (Show Details)
jenkins-bot mentioned this in rOHPU4d66d0d66cf4: Remove nonstop-bridging from switches.Aug 4 2020, 7:38 AM
Maintenance_bot removed a project: Patch-For-Review.Aug 4 2020, 8:10 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits