Page MenuHomePhabricator
Create Task
Maniphest T261866

Route cloud-hosts1-b-eqiad vlan through cloudsw
Closed, ResolvedPublic
Actions

Description

Stage 1 of https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/EnhancementProposals/2020_Network_refresh

Scheduled for Thursday Sept. 3rd, 9am UTC.

  • Force VRRP master on cr1-eqiad
cr1-eqiad
[edit interfaces ae2 unit 1118 family inet address 10.64.20.2/24 vrrp-group 118]
+        priority 200;
[edit interfaces ae2 unit 1118 family inet6 address 2620:0:861:118:fe00::1/64 vrrp-inet6-group 118]
+        priority 200;

show vrrp interface ae2.1118 | match State:

  • Move inet/inet6 configuration on cr2-eqiad from ae2.1118 to xe-3/0/4.1118

Enable: https://netbox.wikimedia.org/dcim/interfaces/2146/
Enable: https://netbox.wikimedia.org/dcim/interfaces/7663/
Rename: https://netbox.wikimedia.org/dcim/interfaces/9010/ to xe-3/0/4.1118
Run Homer
cloudsw1-d5-eqiad# delete interfaces xe-0/0/0 disable

  • Check if IP is working as expected

Reachability from cr1:ae2.1118 to cr2:xe-3/0/4.1118
cr1-eqiad> ping 10.64.20.3 source 10.64.20.2
VRRP state sharing between cr1 and cr2
cr1-eqiad> show vrrp interface ae2.1118

  • Move VRRP mastership to cr2
cr1-eqiad
[edit interfaces ae2 unit 1118 family inet address 10.64.20.2/24 vrrp-group 118]
+        priority 70;
[edit interfaces ae2 unit 1118 family inet6 address 2620:0:861:118:fe00::1/64 vrrp-inet6-group 118]
+        priority 70;
  • Check reachability of cloud-hosts devices
  • Move inet/inet6 configuration on cr1-eqiad from ae2.1118 to xe-3/0/4.1118

Enable https://netbox.wikimedia.org/dcim/interfaces/7651/
Enable https://netbox.wikimedia.org/dcim/interfaces/2082/
Rename https://netbox.wikimedia.org/dcim/interfaces/9033/ to xe-3/0/4.1118
Remove VRRP group from vlan 1118 VRRP IPs (See T260363)
Edit homer-public to change the interface using labs-in6
Run Homer
cloudsw1-c8-eqiad# delete interfaces xe-0/0/0 disable

  • Check if IP is working as expected
  • Cleanup (update OSFP/bootp)

Details

SubjectRepoBranchLines +/-
Only use VRRP bandwidth-threshold on ae linksoperations/homer/publicmaster+1 -1
Rename ae2.1118 to xe-3/0/4.1118operations/homer/publicmaster+5 -5
Customize query in gerrit

Related Objects

Event Timeline

ayounsi triaged this task as Medium priority.Sep 2 2020, 1:45 PM
ayounsi created this task.
Restricted Application added a project: SRE. · View Herald TranscriptSep 2 2020, 1:45 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Stashbot added a comment.Sep 3 2020, 9:01 AM

Mentioned in SAL (#wikimedia-operations) [2020-09-03T09:01:51Z] <XioNoX> force ae2.1118 VRRP master on cr1-eqiad - T261866

Stashbot added a comment.Sep 3 2020, 9:06 AM

Mentioned in SAL (#wikimedia-operations) [2020-09-03T09:06:19Z] <XioNoX> move vlan 1118 from ae2.1118 to xe-3/0/4.1118 cr2-eqiad - T261866

Stashbot added a comment.Sep 3 2020, 9:13 AM

Mentioned in SAL (#wikimedia-operations) [2020-09-03T09:13:30Z] <XioNoX> rolled back: move vlan 1118 from ae2.1118 to xe-3/0/4.1118 cr2-eqiad - T261866

Stashbot added a comment.Sep 3 2020, 9:31 AM

Mentioned in SAL (#wikimedia-cloud) [2020-09-03T09:31:50Z] <arturo> downtime cloud* servers for 30 mins (T261866)

Stashbot added a comment.Sep 3 2020, 9:31 AM

Mentioned in SAL (#wikimedia-cloud) [2020-09-03T09:31:57Z] <arturo> icinga downtime cloud* servers for 30 mins (T261866)

Stashbot added a comment.Sep 3 2020, 9:38 AM

Mentioned in SAL (#wikimedia-operations) [2020-09-03T09:38:18Z] <XioNoX> move vlan 1118 IPv6 from ae2.1118 to xe-3/0/4.1118 cr2-eqiad - T261866

Stashbot added a comment.Sep 3 2020, 9:46 AM

Mentioned in SAL (#wikimedia-operations) [2020-09-03T09:46:31Z] <XioNoX> move vlan 1118 IPv4 from ae2.1118 to xe-3/0/4.1118 cr2-eqiad - T261866

Stashbot added a comment.Sep 3 2020, 9:48 AM

Mentioned in SAL (#wikimedia-operations) [2020-09-03T09:48:01Z] <XioNoX> move VRRP master from cr1-eqiad:ae2.1118 to cr2-eqiad:xe-3/0/4.1118 - T261866

Stashbot added a comment.Sep 3 2020, 9:56 AM

Mentioned in SAL (#wikimedia-operations) [2020-09-03T09:56:43Z] <XioNoX> move vlan 1118 from ae2.1118 to xe-3/0/4.1118 cr2-eqiad - T261866

Stashbot added a comment.Sep 3 2020, 9:57 AM

Mentioned in SAL (#wikimedia-operations) [2020-09-03T09:57:05Z] <XioNoX> rectification: move vlan 1118 from ae2.1118 to xe-3/0/4.1118 on cr1-eqiad - T261866

gerritbot added a comment.Sep 3 2020, 10:02 AM

Change 623988 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/homer/public@master] Rename ae2.1118 to xe-3/0/4.1118

https://gerrit.wikimedia.org/r/623988

gerritbot added a project: Patch-For-Review.Sep 3 2020, 10:02 AM
Stashbot added a comment.Sep 3 2020, 10:07 AM

Mentioned in SAL (#wikimedia-operations) [2020-09-03T10:07:47Z] <XioNoX> re-apply vlan 1118 firewall filter and update OSPF/bootp on cr1/2-eqiad - T261866

gerritbot added a comment.Sep 3 2020, 10:09 AM

Change 623995 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/homer/public@master] Only use VRRP bandwidth-threshold on ae links

https://gerrit.wikimedia.org/r/623995

ayounsi updated the task description. (Show Details)Sep 3 2020, 10:11 AM
gerritbot added a comment.Sep 3 2020, 10:13 AM

Change 623988 merged by jenkins-bot:
[operations/homer/public@master] Rename ae2.1118 to xe-3/0/4.1118

https://gerrit.wikimedia.org/r/623988

gerritbot added a comment.Sep 3 2020, 10:13 AM

Change 623995 merged by jenkins-bot:
[operations/homer/public@master] Only use VRRP bandwidth-threshold on ae links

https://gerrit.wikimedia.org/r/623995

ayounsi closed this task as Resolved.Sep 3 2020, 10:18 AM

There has been 1 issue: the cr2-eqiad facing interface on cloudsw1-d5 was miss-configured (configured as L3 instead of L2 trunk interface).

Even though VRRP master for that vlan was on cr1-eqiad, all the other vlans had their default GW on cr2-eqiad, so traffic from (for example) icinga, was trying to enter vlan 1118 through the cr2<->cloudsw1-d5 link and got blackholed.

Fixing the initial issue then taking a more careful approach by moving v6 before v4 allowed to complete this change.

Maintenance_bot removed a project: Patch-For-Review.Sep 3 2020, 11:10 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL