Page MenuHomePhabricator
Create Task
Maniphest T257573

Remove multicast
Closed, ResolvedPublic
Actions

Description

If I understand T250781, multicast is no more. Which seems to be confirmed by LibreNMS.

If so, we can then proceed with cleaning up everything related to multicast:

  • Ensure purged doesn't subscribe to any multicast group
  • Confirm that all multicast listeners are gone (eg. using cumin with netstat -g -n | grep -v 224.0.0.1 | grep -v ff0 | tail -n +4
  • Remove igmp-snooping from switches
  • Remove PIM config from routers
  • Close related tasks (eg. T201039, T167842)
  • Update doc https://wikitech.wikimedia.org/wiki/Multicast_IP_Addresses

Details

SubjectRepoBranchLines +/-
Reclaim PIM-RP IPs from the multicast godsoperations/dnsmaster+1 -2
Remove PIM related CR stanzaoperations/homer/publicmaster+0 -57
Remove IGMP snooping stanzaoperations/homer/publicmaster+0 -14
Remove IGMP snooping from all access switchesoperations/homer/publicmaster+3 -6
purged: stop passing -mcast_addrsoperations/puppetproduction+1 -8
Make -mcast_addrs optionaloperations/software/purgedmaster+13 -6
Customize query in gerrit

Related Objects

Event Timeline

ayounsi triaged this task as Low priority.Jul 9 2020, 1:16 PM
ayounsi created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 9 2020, 1:16 PM
ayounsi updated the task description. (Show Details)Jul 9 2020, 1:31 PM
• ema awarded a token.Jul 10 2020, 7:58 AM
gerritbot added a comment.Jul 15 2020, 9:44 AM

Change 612836 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] purged: stop passing -mcast_addrs

https://gerrit.wikimedia.org/r/612836

gerritbot added a project: Patch-For-Review.Jul 15 2020, 9:44 AM
• ema added a subscriber: • Pchelolo.Jul 15 2020, 9:49 AM

When it comes to purged, I just confirmed with rate(purged_htcp_packets_total[5m]) > 0 that indeed it stopped receiving multicast HTCP purges as mentioned by @Pchelolo on 2020-07-08 (one week ago).

Screenshot from 2020-07-15 11-46-26.png (1×2 px, 264 KB)

• ema moved this task from Backlog to Network on the Traffic board.Jul 15 2020, 9:51 AM
Joe subscribed.Jul 15 2020, 9:56 AM
gerritbot added a comment.Jul 15 2020, 10:05 AM

Change 612840 had a related patch set uploaded (by Ema; owner: Ema):
[operations/software/purged@master] Make -mcast_addrs optional

https://gerrit.wikimedia.org/r/612840

gerritbot added a comment.Jul 15 2020, 2:13 PM

Change 612840 merged by Ema:
[operations/software/purged@master] Make -mcast_addrs optional

https://gerrit.wikimedia.org/r/612840

Stashbot added a comment.Jul 15 2020, 2:30 PM

Mentioned in SAL (#wikimedia-operations) [2020-07-15T14:30:21Z] <ema> upload purged 0.17 to buster-wikimedia T257573

Stashbot added a comment.Jul 15 2020, 2:37 PM

Mentioned in SAL (#wikimedia-operations) [2020-07-15T14:36:54Z] <ema> A:cp: upgrade purged to 0.17 T257573

gerritbot added a comment.Jul 15 2020, 2:38 PM

Change 612836 merged by Ema:
[operations/puppet@production] purged: stop passing -mcast_addrs

https://gerrit.wikimedia.org/r/612836

Maintenance_bot removed a project: Patch-For-Review.Jul 15 2020, 3:11 PM
ayounsi updated the task description. (Show Details)Jul 16 2020, 6:33 AM
gerritbot added a comment.Jul 16 2020, 6:59 AM

Change 613081 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/homer/public@master] Remove IGMP snooping from all access switches

https://gerrit.wikimedia.org/r/613081

gerritbot added a project: Patch-For-Review.Jul 16 2020, 6:59 AM
gerritbot added a comment.Jul 16 2020, 7:01 AM

Change 613082 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/homer/public@master] Remove IGMP snooping stanza

https://gerrit.wikimedia.org/r/613082

akosiaris subscribed.Jul 16 2020, 7:14 AM
Stashbot added a comment.Jul 16 2020, 8:03 AM

Mentioned in SAL (#wikimedia-operations) [2020-07-16T08:03:36Z] <XioNoX> remove PIM config from ulsfo routers - T257573

Stashbot added a comment.Jul 16 2020, 8:07 AM

Mentioned in SAL (#wikimedia-operations) [2020-07-16T08:07:31Z] <XioNoX> remove PIM config from codfw routers - T257573

Stashbot added a comment.Jul 16 2020, 8:09 AM

Mentioned in SAL (#wikimedia-operations) [2020-07-16T08:09:58Z] <XioNoX> remove PIM config from eqsin routers - T257573

Stashbot added a comment.Jul 16 2020, 8:11 AM

Mentioned in SAL (#wikimedia-operations) [2020-07-16T08:11:58Z] <XioNoX> remove PIM config from esams routers - T257573

Stashbot added a comment.Jul 16 2020, 8:14 AM

Mentioned in SAL (#wikimedia-operations) [2020-07-16T08:14:15Z] <XioNoX> remove PIM config from eqiad routers - T257573

Stashbot added a comment.Jul 16 2020, 8:15 AM

Mentioned in SAL (#wikimedia-operations) [2020-07-16T08:15:03Z] <XioNoX> remove PIM config from eqord/eqdfw/knams routers - T257573

ayounsi added a comment.EditedJul 16 2020, 8:16 AM

Confirmed with run clear multicast statistics then run show multicast statistics that counters stay at 0. (checked before removing PIM)

gerritbot added a comment.Jul 16 2020, 8:20 AM

Change 613091 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/homer/public@master] Remove PIM related CR stanza

https://gerrit.wikimedia.org/r/613091

gerritbot added a comment.Jul 16 2020, 8:22 AM

Change 613081 merged by jenkins-bot:
[operations/homer/public@master] Remove IGMP snooping from all access switches

https://gerrit.wikimedia.org/r/613081

Stashbot added a comment.Jul 16 2020, 8:24 AM

Mentioned in SAL (#wikimedia-operations) [2020-07-16T08:24:16Z] <XioNoX> remove igmp-snooping from access switches - T257573

gerritbot added a comment.Jul 16 2020, 8:31 AM

Change 613082 merged by jenkins-bot:
[operations/homer/public@master] Remove IGMP snooping stanza

https://gerrit.wikimedia.org/r/613082

ayounsi mentioned this in T201039: IGMP snooping breaks IPv6 ND on Junos 14.1X53-D46.Jul 16 2020, 8:32 AM
gerritbot added a comment.Jul 16 2020, 8:35 AM

Change 613091 merged by jenkins-bot:
[operations/homer/public@master] Remove PIM related CR stanza

https://gerrit.wikimedia.org/r/613091

Stashbot added a comment.Jul 16 2020, 8:35 AM

Mentioned in SAL (#wikimedia-operations) [2020-07-16T08:35:25Z] <XioNoX> Remove PIM/IGMP related CR stanza (acls) - T257573

ayounsi closed this task as Resolved.Jul 16 2020, 8:55 AM
ayounsi claimed this task.
ayounsi updated the task description. (Show Details)

No more multicast/PIM in the infra.

Vgutierrez awarded a token.Jul 20 2020, 12:26 PM
faidon reopened this task as Open.Jul 22 2020, 6:12 PM
faidon subscribed.

We still seem to have remnants of PIM-RP:

faidon@re0.cr2-codfw> show configuration | display set | match 208.80.153.194             
set interfaces lo0 unit 0 family inet address 208.80.153.194/32
$ dig +short @ns0.wikimedia.org -x 208.80.153.194
pim-rp.wikimedia.org.
$ dig +short @ns0.wikimedia.org -x 2620:0:860:ffff::3
pim-rp.wikimedia.org.
Stashbot added a comment.Jul 23 2020, 8:40 AM

Mentioned in SAL (#wikimedia-operations) [2020-07-23T08:40:51Z] <XioNoX> remove pim-rp IPs from last routers - T257573

gerritbot added a comment.Jul 23 2020, 8:45 AM

Change 615673 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/dns@master] Reclaim PIM-RP IPs from the multicast gods

https://gerrit.wikimedia.org/r/615673

gerritbot added a comment.Jul 23 2020, 8:46 AM

Change 615673 merged by Ayounsi:
[operations/dns@master] Reclaim PIM-RP IPs from the multicast gods

https://gerrit.wikimedia.org/r/615673

ayounsi closed this task as Resolved.Jul 23 2020, 8:50 AM

Indeed, cleaned up as well:

rancid-configs/configs$ ag 208.80.153.194
cr1-codfw.wikimedia.org
1661:                address 208.80.153.194/32;

cr2-codfw.wikimedia.org
1635:                address 208.80.153.194/32;
rancid-configs/configs$ ag 2620:0:860:ffff::3
cr1-codfw.wikimedia.org
1671:                address 2620:0:860:ffff::3/128;

cr1-eqiad.wikimedia.org
2038:                address 2620:0:860:ffff::3/128;

cr2-codfw.wikimedia.org
1645:                address 2620:0:860:ffff::3/128;

cr2-eqiad.wikimedia.org
2044:                inactive: address 2620:0:860:ffff::3/128;
BBlack moved this task from Network to Done on the Traffic board.Oct 8 2021, 6:00 PM
Restricted Application added a project: Infrastructure-Foundations. · View Herald TranscriptOct 8 2021, 6:00 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL