eqiad:

codfw:

@Papaul maybe I'm missing some limitation, but there is any reason to just be using two switches per row on codfw for LVS networking links? Adding a third switch per row would be great for availability purposes, could we do that?

@Vgutierrez the only limitation i cans see is the number of NIC ports on each server. Each server has 4 NIC's each NIC connected to 1 row on 1 switch. If we do keep the existing configuration then we will have to add another 4 ports NIC in each server so that each server can have 2 connections on each row.

@Papaul nope.. the idea would be replace some of the current links with new ones to additional switches

@Vgutierrez if my understanding is right you want for example lvsX NIC 1 to switch asw-a2 NIC 2 to switch asw-a7 (2 switches in ROW A) and NIC 3 to switch asw-b2 and NIC 4 to swirch asw-b7 (2 switch in ROW B) and so on ?

the current problem is that both lvs2007 (primary for high-traffic1) and lvs2010 (secondary) get row A traffic from the very same switch, so if that switch fails we lose row A in both primary and secondary LVS. This also happens between lvs2009 and lvs2010 regarding row A, lvs2008, lvs2009 and lvs2010 for row B. lvs2008, lvs2009 and lvs2010 for row C and lvs2008, lvs2009 and lvs2010 for row D as well.

So it looks like that we should move lvs2010 row A, B and C to another switch on those rows and move lvs2008 and lvs2009 row D connections to another switch on row D

Ok understood. Please provide me with the configuration you want in a table like above for each server which NIC connects to which switch and i can do my site audit on how best to re-route those links.

In T286881#7242985, @Vgutierrez wrote:

Host	Row	Host iface	switch iface	switch name
lvs2007	A	ens2f0np0	xe-2/0/45	A2
lvs2008	A	ens2f1np1	xe-7/0/45	A7
lvs2009	A	ens2f1np1	xe-2/0/43	A2
lvs2010	A	ens2f1np1	Switch A4	A4
lvs2007	B	ens2f1np1	xe-7/0/45	A7
lvs2008	B	ens2f0np0	xe-2/0/45	B2
lvs2009	B	ens3f0np0	xe-2/0/43	B2
lvs2010	B	ens3f0np0	Switch B4	B4
lvs2007	C	ens3f0np0	xe-7/0/45	C7
lvs2008	C	ens3f0np0	xe-2/0/45	C2
lvs2009	C	ens2f0np0	xe-2/0/44	C2
lvs2010	C	ens3f1np1	Switch C4	C4
lvs2007	D	ens3f1np1	xe-7/0/45	C7
lvs2008	D	ens3f1np1	Switch D4	D4
lvs2009	D	ens3f1np1	Switch D4	D4
lvs2010	D	ens2f0np0	xe-2/0/44	D2

@Vgutierrez Base on the table you provided lvs2007 has it’s 2 first NIC’s connected to row A( asw-a2 and asw-a7) and the other 2 NIC ‘s connected to row c ( asw-c7 and asw C7) can you please double check this. Thanks

@Papaul that's a mistake on my side, thanks for spotting it, the second NIC ens2f1np1 is actually connected to B7

@Vgutierrez thank you. What about lvs2007 ens3f1np1? Actually it is connected to d7 and you want it to be moved to C7 or lvs2007 ens3f0np0 is already connected to C7

@Papaul same thing.. lvs2007 ens3f1np1 is connected to D7, the only desired changes are the new links against A4, B4, C4 and D4

@Vgutierrez thank you I have all the information needed. I will do my site audit and get back with you next week to setup a day and time to start moving those links

@Vgutierrez do you have time today 10am CT to move only
lvs2010 ens2f1np1 xe-2/0/44 asw-a4-codfw from A2 to a4 xe-4/0/47

@Papaul we need to coordinate with @ayounsi or @cmooney to let them configure the ports on asw-a4-codfw. For me it's basically a NOOP on lvs2010, I just need to check that the interfaces are up and working after you change it

I can set up the interface on asw-a4-codfw

even better then :)

Mentioned in SAL (#wikimedia-operations) [2021-08-05T14:52:37Z] <vgutierrez> depool lvs2008 - T286881

Mentioned in SAL (#wikimedia-operations) [2021-08-05T14:53:52Z] <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on lvs2008.codfw.wmnet with reason: T286881

Icinga downtime set by vgutierrez@cumin1001 for 1:00:00 1 host(s) and their services with reason: T286881

lvs2008.codfw.wmnet

Mentioned in SAL (#wikimedia-operations) [2021-08-05T14:53:59Z] <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on lvs2008.codfw.wmnet with reason: T286881

Mentioned in SAL (#wikimedia-operations) [2021-08-05T15:10:13Z] <vgutierrez> pool lvs2008 - T286881

Mentioned in SAL (#wikimedia-operations) [2021-08-05T15:11:07Z] <vgutierrez> depool lvs2009 - T286881

Mentioned in SAL (#wikimedia-operations) [2021-08-05T15:11:16Z] <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on lvs2009.codfw.wmnet with reason: T286881

Icinga downtime set by vgutierrez@cumin1001 for 1:00:00 1 host(s) and their services with reason: T286881

lvs2009.codfw.wmnet

Mentioned in SAL (#wikimedia-operations) [2021-08-05T15:11:22Z] <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on lvs2009.codfw.wmnet with reason: T286881

Mentioned in SAL (#wikimedia-operations) [2021-08-05T15:28:37Z] <vgutierrez> pool lvs2009 - T286881

The swap of Traffic for Traffic-Icebox in this ticket's set of tags was based on a bulk action for all tickets that aren't are neither part of our current planned work nor clearly a recent, higher-priority emergent issue. This is simply one step in a larger task cleanup effort. Further triage of these tickets (and especially, organizing future potential project ideas from them into a new medium) will occur afterwards! For more detail, have a look at the extended explanation on the main page of Traffic-Icebox . Thank you!

@Vgutierrez What actions, if any, are still required for this ticket?

After double checking on netbox it looks like we have some standing issues:
lvs1017 and lvs1020 get connectivity to rows B and C on the same switch:

• Row B:
  • lvs1017 is connected to asw2-b4-eqiad on port xe-4/0/15
  • lvs1020 is connected to asw2-b4-eqiad on port xe-4/0/34
• Row C:
  • lvs1017 is connected to asw2-c4-eqiad on port xe-4/0/32
  • lvs1020 is connected to asw2-c4-eqiad on port xe-4/0/5

everything else looks good.

In T286881#8254203, @BCornwall wrote:

@Vgutierrez What actions, if any, are still required for this ticket?

we need to coordinate with DCops, probably with @Jclark-ctr, to rearrange these connections

@wiki_willy could you help us prioritizing the remaining work on eqiad? this needs to be fixed ASAP

@Vgutierrez Would these 2 changes work for what is needed? If not we would have to order replacement cables longer lengths to reach other racks.

Row B:
lvs1017 is connected to asw2-b4-eqiad on port xe-4/0/15 Cableid 4801 asw2-b2-eqiad xe-2/0/42
lvs1020 is connected to asw2-b4-eqiad on port xe-4/0/34 Cableid 3931
Row C:
lvs1017 is connected to asw2-c4-eqiad on port xe-4/0/32 Cableid 4802
lvs1020 is connected to asw2-c4-eqiad on port xe-4/0/5 Cableid 3918 asw2-c2-eqiad xe-2/0/40

@Jclark-ctr as long as both lvs1017 and lvs1020 don't get connectivity from the same switch on a single row is ok. So those look good

@Jclark-ctr please let me or @BCornwall know when it would be a good time for you to perform the change

@Vgutierrez will schedule for next week i will not be on site today unless @Cmjohnson is available today i will have to get with netops for configuring the port for move

@Jclark-ctr if you want, you can also ping me for the port configuration.

@Vgutierrez Would like to move connections can You depooled Servers so connection can move this afternoon??

@Jclark-ctr we need to do lvs1020 first, and after it's done we can depool lvs1017, but we cannot depool both lvs instances at the same time

Mentioned in SAL (#wikimedia-operations) [2022-10-13T16:50:30Z] <sukhe> disable Puppet and stop Pybal on lvs1020: T286881

Mentioned in SAL (#wikimedia-operations) [2022-10-13T17:10:31Z] <sukhe> disable Puppet and stop Pybal on lvs1017: T286881

Moved two connections cableid stayed the same
lvs1017 is connected to asw2-b4-eqiad on old port xe-4/0/15 Cableid 4801, New asw2-b2-eqiad xe-2/0/42
lvs1020 is connected to asw2-c4-eqiad on old port xe-4/0/5 Cableid 3918, New asw2-c2-eqiad xe-2/0/40