Page MenuHomePhabricator

drmrs/esams/magru LVS : remove cross-rack links
Closed, ResolvedPublic

Description

Now that IPIP has been rolled out to all the POPs we should be able to simplify the POPs network topology by removing now unnecessary configuration/links.

Steps along the lines of:

  1. Depool the site
  2. Disable the x-rack switch port
  3. Check monitoring (make sure health checks are still working)
  4. Repool the site
  5. Keep on that state for a while for monitoring and to be able to re-enabled it quickly it any issue
  6. Open a remote hands ticket to fully remove the cable

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Don't be too aggressive with this one, we could need to rollback at some point, let's wait a few weeks at the very least

Of course ! not planning on doing it today :) The task is there to not forget.

Doing a bit of an audit here to assess the current situation, we have the following cables in place which need to be removed:

@Vgutierrez all of these have been moved to Liberica right? In other words there is no going back now, and we should move to disable the switch ports and have the cables removed?

It's not clear to me why the reimage of lvs6003 removed that cable in Netbox but not for the others, possibly some race to do with when the puppetdb import script ran.

Vgutierrez raised the priority of this task from Low to Medium.Jun 11 2025, 8:25 AM
Vgutierrez moved this task from Backlog to Scheduled work for this quarter on the Traffic board.

all PoPs have been running liberica with IPIP encapsulation for a while now and we are already starting the migration to Katran there, so we can start removing these cables

Mentioned in SAL (#wikimedia-operations) [2025-06-11T11:34:01Z] <XioNoX> disable lvs7003 secondary link switch port - T367731

Mentioned in SAL (#wikimedia-operations) [2025-06-11T11:42:02Z] <XioNoX> disable lvs3010 secondary link switch port - T367731

Mentioned in SAL (#wikimedia-operations) [2025-06-11T12:41:32Z] <XioNoX> disable lvs7002 secondary link switch port - T367731

Mentioned in SAL (#wikimedia-operations) [2025-06-11T12:43:42Z] <XioNoX> disable lvs7001 secondary link switch port - T367731

Mentioned in SAL (#wikimedia-operations) [2025-06-11T12:51:17Z] <XioNoX> disable lvs3009 secondary link switch port - T367731

Mentioned in SAL (#wikimedia-operations) [2025-06-11T12:54:52Z] <XioNoX> disable lvs3008 secondary link switch port - T367731

Mentioned in SAL (#wikimedia-operations) [2025-06-11T13:00:19Z] <XioNoX> disable lvs6002 secondary link switch port - T367731

ayounsi claimed this task.
ayounsi updated the task description. (Show Details)

All done!

Nice work @ayounsi!

I would have thought dc-ops need to follow up with remote hands requests to remove the cables though?