Page MenuHomePhabricator

Remove multicast
Closed, ResolvedPublic

Description

If I understand T250781, multicast is no more. Which seems to be confirmed by LibreNMS.

If so, we can then proceed with cleaning up everything related to multicast:

  • Ensure purged doesn't subscribe to any multicast group
  • Confirm that all multicast listeners are gone (eg. using cumin with netstat -g -n | grep -v 224.0.0.1 | grep -v ff0 | tail -n +4
  • Remove igmp-snooping from switches
  • Remove PIM config from routers
  • Close related tasks (eg. T201039, T167842)
  • Update doc https://wikitech.wikimedia.org/wiki/Multicast_IP_Addresses

Event Timeline

ayounsi created this task.

Change 612836 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] purged: stop passing -mcast_addrs

https://gerrit.wikimedia.org/r/612836

When it comes to purged, I just confirmed with rate(purged_htcp_packets_total[5m]) > 0 that indeed it stopped receiving multicast HTCP purges as mentioned by @Pchelolo on 2020-07-08 (one week ago).

Screenshot from 2020-07-15 11-46-26.png (1×2 px, 264 KB)

Change 612840 had a related patch set uploaded (by Ema; owner: Ema):
[operations/software/purged@master] Make -mcast_addrs optional

https://gerrit.wikimedia.org/r/612840

Change 612840 merged by Ema:
[operations/software/purged@master] Make -mcast_addrs optional

https://gerrit.wikimedia.org/r/612840

Mentioned in SAL (#wikimedia-operations) [2020-07-15T14:30:21Z] <ema> upload purged 0.17 to buster-wikimedia T257573

Change 612836 merged by Ema:
[operations/puppet@production] purged: stop passing -mcast_addrs

https://gerrit.wikimedia.org/r/612836

Change 613081 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/homer/public@master] Remove IGMP snooping from all access switches

https://gerrit.wikimedia.org/r/613081

Change 613082 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/homer/public@master] Remove IGMP snooping stanza

https://gerrit.wikimedia.org/r/613082

Mentioned in SAL (#wikimedia-operations) [2020-07-16T08:03:36Z] <XioNoX> remove PIM config from ulsfo routers - T257573

Mentioned in SAL (#wikimedia-operations) [2020-07-16T08:07:31Z] <XioNoX> remove PIM config from codfw routers - T257573

Mentioned in SAL (#wikimedia-operations) [2020-07-16T08:09:58Z] <XioNoX> remove PIM config from eqsin routers - T257573

Mentioned in SAL (#wikimedia-operations) [2020-07-16T08:11:58Z] <XioNoX> remove PIM config from esams routers - T257573

Mentioned in SAL (#wikimedia-operations) [2020-07-16T08:14:15Z] <XioNoX> remove PIM config from eqiad routers - T257573

Mentioned in SAL (#wikimedia-operations) [2020-07-16T08:15:03Z] <XioNoX> remove PIM config from eqord/eqdfw/knams routers - T257573

Confirmed with run clear multicast statistics then run show multicast statistics that counters stay at 0. (checked before removing PIM)

Change 613091 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/homer/public@master] Remove PIM related CR stanza

https://gerrit.wikimedia.org/r/613091

Change 613081 merged by jenkins-bot:
[operations/homer/public@master] Remove IGMP snooping from all access switches

https://gerrit.wikimedia.org/r/613081

Mentioned in SAL (#wikimedia-operations) [2020-07-16T08:24:16Z] <XioNoX> remove igmp-snooping from access switches - T257573

Change 613082 merged by jenkins-bot:
[operations/homer/public@master] Remove IGMP snooping stanza

https://gerrit.wikimedia.org/r/613082

Change 613091 merged by jenkins-bot:
[operations/homer/public@master] Remove PIM related CR stanza

https://gerrit.wikimedia.org/r/613091

Mentioned in SAL (#wikimedia-operations) [2020-07-16T08:35:25Z] <XioNoX> Remove PIM/IGMP related CR stanza (acls) - T257573

ayounsi claimed this task.
ayounsi updated the task description. (Show Details)

No more multicast/PIM in the infra.

faidon added a subscriber: faidon.

We still seem to have remnants of PIM-RP:

faidon@re0.cr2-codfw> show configuration | display set | match 208.80.153.194             
set interfaces lo0 unit 0 family inet address 208.80.153.194/32
$ dig +short @ns0.wikimedia.org -x 208.80.153.194
pim-rp.wikimedia.org.
$ dig +short @ns0.wikimedia.org -x 2620:0:860:ffff::3
pim-rp.wikimedia.org.

Mentioned in SAL (#wikimedia-operations) [2020-07-23T08:40:51Z] <XioNoX> remove pim-rp IPs from last routers - T257573

Change 615673 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/dns@master] Reclaim PIM-RP IPs from the multicast gods

https://gerrit.wikimedia.org/r/615673

Change 615673 merged by Ayounsi:
[operations/dns@master] Reclaim PIM-RP IPs from the multicast gods

https://gerrit.wikimedia.org/r/615673

Indeed, cleaned up as well:

rancid-configs/configs$ ag 208.80.153.194
cr1-codfw.wikimedia.org
1661:                address 208.80.153.194/32;

cr2-codfw.wikimedia.org
1635:                address 208.80.153.194/32;
rancid-configs/configs$ ag 2620:0:860:ffff::3
cr1-codfw.wikimedia.org
1671:                address 2620:0:860:ffff::3/128;

cr1-eqiad.wikimedia.org
2038:                address 2620:0:860:ffff::3/128;

cr2-codfw.wikimedia.org
1645:                address 2620:0:860:ffff::3/128;

cr2-eqiad.wikimedia.org
2044:                inactive: address 2620:0:860:ffff::3/128;