Connect labtestvirt2003 eth1 and eth2 interface(s) to switch fabric
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• chasemp
	Dec 18 2017, 6:25 PM

Description

When labtestvirt2003 was setup in T166237: rack/setup/install labtestvirt2003 I must have failed to note that it needed the eth1 interface connected. Our 'labvirts' traditionally use eth0 for the host and management traffic, and eth1 for tenants. Additionally we are close to piloting eth2 for vxlan use cases so seems ideal to get it done at one time :)

eth0 is setup and good to go but at the moment no tenants have connectivity as eth1 is not connected.

eth0 status checked with lldpcli show neighbors and

interface-range vlan-labs-hosts1-b-codfw {
    member ge-1/0/11;

eth1 is 30:e1:71:60:e9:7d

Related Objects
Search...

Status	Assigned	Task
Open	None	T53494 Use Beta cluster as a true canary for code deployments (epic)
Open	None	T87220 Minimize infrastructure differences between Beta Cluster and production
Open	None	T196662 Set up LVS in beta like prod
Resolved	bd808	T166396 Program 1 Outcome 4: VPS hosting
Resolved	None	T167293 Nova-network to Neutron migration
Resolved	Papaul	T183167 Connect labtestvirt2003 eth1 and eth2 interface(s) to switch fabric

Event Timeline

• chasemp triaged this task as Medium priority.Dec 18 2017, 6:25 PM

• chasemp created this task.

Restricted Application added a project: SRE. · View Herald TranscriptDec 18 2017, 6:25 PM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

@Papaul,

please document which ports you connect to so we can update the switch config accordingly. thanks!

RobH moved this task from Backlog to Up Next on the ops-codfw board.Jan 3 2018, 10:10 PM

@RobH

This was not on my dashboard so I missed it. I will get on it when back at the DC tomorrow.

@RobH
Row B rack B1
labtestvirt2003 ge-1/0/12

Papaul reassigned this task from Papaul to RobH.Jan 4 2018, 4:46 PM

synced up with @Papaul via irc:

labtestvirt2003:eth1:ge-1/0/12
labtestvirt2003:eth2:ge-1/0/14

In T183167#3876000, @RobH wrote:

synced up with @Papaul via irc:

labtestvirt2003:eth1:ge-1/0/12
labtestvirt2003:eth2:ge-1/0/14

I'm a bit confused by this:

asw-b-codfw# show interfaces ge-1/0/14
description labtestnet2002-eth1;
enable;

Mentioned in SAL (#wikimedia-operations) [2018-02-05T21:45:22Z] <chasemp> asw-b-codfw# rollback 0 pending questions on T183167

Ok, so what is on this task doesn't match what is on the switch stack. Unfortuantely, I don't recall if we ever set this correctly, or if anything has changed since this was last looked at a month ago.

Right now, the switch stack shows nothing on ge-1/0/12 and labtestnet2002-eth1 on eth2:ge-1/0/14.

@Papaul:

Due to the confusion on this, I need you to audit and paste on this task the following port information:

What network ports do the following plug into:

labtestvirt2002:eth0
labtestvirt2002:eth1
labtestvirt2002:eth2
labtestvirt2003:eth0
labtestvirt2003:eth1
labtestvirt2003:eth2

Also, can you include the cable serial numbers for each? It'd be nice to start appending this on like we do for cross-connects.

Please provide the above info and assign back to me, thank you!

labtestvirt2003:eth2

is also in the list above

In T183167#3947773, @chasemp wrote:

labtestvirt2003:eth2

is also in the list above

Indeed, I've corrected my comment to list the following:

labtestvirt2002:eth0
labtestvirt2002:eth1
labtestvirt2003:eth0
labtestvirt2003:eth1
labtestvirt2003:eth2

labtestvirt2002:eth0 = ge-5/0/17 (ID=2187)
labtestvirt2002:eth1 = ge-5/0/ 31 (ID=11518)

labtestvirt2003:eth0 = ge -1/0/ 11 (ID=11856)
labtestvirt2003:eth1 = ge-1/0/ 12 (ID= 11890)
labtestvirt2003:eth2 = ge-1/0/ 14 (ID= 11990)

@Papaul -- I'm still a bit confused. Really appreciate you sanity checking all this, I think something is still off

In T183167#3949338, @Papaul wrote:

labtestvirt2002:eth0 = ge-5/0/17 (ID=2187)
labtestvirt2002:eth1 = ge-5/0/ 31 (ID=11518)

ge-5/0/17 shows as labtestvirt2001 (instead of 2002 as noted) for me

labtestvirt2001:~# ip addr show eth0 | grep ether | awk '{print $2}'
90:b1:1c:2d:7e:b9

rush@asw-b-codfw> show ethernet-switching table 90:b1:1c:2d:7e:b9

MAC flags (S - static MAC, D - dynamic MAC, L - locally learned, P - Persistent static
           SE - statistics enabled, NM - non configured MAC, R - remote PE MAC, O - ovsdb MAC)


Ethernet switching table : 12 entries, 12 learned
Routing instance : default-switch
    Vlan                MAC                 MAC         Age    Logical
    name                address             flags              interface
    labs-hosts1-b-codfw 90:b1:1c:2d:7e:b9   D             -   ge-5/0/17.0

In the switch:

set interfaces ge-5/0/17 description labtestvirt2001-eth0
set interfaces ge-5/0/31 description labtestvirt2001-eth1

I think this is mixing up labvirts 2001 and 2002 or labels are wrong or something

One other issue I have at the moment is

set interfaces ge-5/0/8 description labtestmetal2001-eth0
set interfaces ge-5/0/30 description labtestmetal2001-eth1

set interfaces ge-8/0/0 description labtestmetal2001-eth0
set interfaces ge-8/0/3 description labtestmetal2001-eth1

I know there was confusion because we had an old server named this and it was renamed and then a new server was bought with this name. I think that:

set interfaces ge-8/0/0 description labtestmetal2001-eth0 is correct
set interfaces ge-8/0/3 description labtestmetal2001-eth1 is correct

and

set interfaces ge-5/0/8 description labtestmetal2001-eth0is labtestvirt2002
set interfaces ge-5/0/30 description labtestmetal2001-eth1 is labtestvirt2002

based on

root@labtestvirt2002:~# ip addr show eth0 | grep ether | awk '{print $2}'
78:2b:cb:49:00:15

rush@asw-b-codfw> show ethernet-switching table 78:2b:cb:49:00:15

MAC flags (S - static MAC, D - dynamic MAC, L - locally learned, P - Persistent static
           SE - statistics enabled, NM - non configured MAC, R - remote PE MAC, O - ovsdb MAC)


Ethernet switching table : 12 entries, 12 learned
Routing instance : default-switch
    Vlan                MAC                 MAC         Age    Logical
    name                address             flags              interface
    labs-hosts1-b-codfw 78:2b:cb:49:00:15   D             -   ge-5/0/8.0

Could you verify if you traced cables for labtestvirt2001 or labvirt2002 in the above? (I think you traced labtestvirt2001 accidentaly instead of labtestvirt2002)

Could you verify ge-5/0/8 and ge-5/0/30 is indeed labtestvirt2002?

Ok, synced up with Chase via IRC:

@Papaul:

Your cable trace of:

In T183167#3949338, @Papaul wrote:

labtestvirt2002:eth0 = ge-5/0/17 (ID=2187)
labtestvirt2002:eth1 = ge-5/0/ 31 (ID=11518)

labtestvirt2003:eth0 = ge -1/0/ 11 (ID=11856)
labtestvirt2003:eth1 = ge-1/0/ 12 (ID= 11890)
labtestvirt2003:eth2 = ge-1/0/ 14 (ID= 11990)

Shows that labtestvirt2002 is using the ports that are currently assigned to labtestvirt2001. In addition to that, when checking on the switch for the mac addresses in the switching table, it shows the MAC for labtestvirt2002 on port ge-5/0/8.

I don't like to rely solely on the mac address table since it won't give me the secondary NICs (since they dont really pxe or push traffic) and won't give me the cable serial numbers.

I need you to review your work on this a second time, and check for the ports for the two new systems, and also verify the labtestvirt2001, since your earlier update now puts it into question.

Summary:

Carefully re-check the ports for the following systems. Please keep in mind we can verify the primary interface (eth0) for each system, but not eth1+ as easily (unless we force it to do a test failed PXE boot.) Your last update stated that labtestvirt2002:eth0 was in a port that we have confirmed via MAC to be in use for labtestvirt2001. So please review all of the following servers and provide the ports and cable numbers for each.

labtestvirt2001:eth0 (MAC 90:b1:1c:2d:7e:b9) = asw-b-codfw:ge-5/0/17 (port confirmed via ethernet switching table on switch) Please provide the cable #.
labtestvirt2001:eth0 (MAC 90:b1:1c:2d:7e:ba) = asw-b-codfw:

labtestvirt2002:eth0 (MAC 78:2b:cb:49:00:15) = asw-b-codfw:ge-5/0/8 (port confirmed via ethernet switching table on switch) Please provide the cable #.
labtestvirt2002:eth1 (MAC 78:2b:cb:49:00:16) = asw-b-codfw:

labtestvirt2003:eth0 (MAC 30:e1:71:60:e9:7c) = asw-b-codfw:ge-1/0/11 (port confirmed via ethernet switching table on switch) Please provide the cable #.
labtestvirt2003:eth0 (MAC 30:e1:71:60:e9:7d) = asw-b-codfw:
labtestvirt2003:eth0 (MAC 30:e1:71:60:e9:7e) = asw-b-codfw:

All the information I provide this morning on labtestvirt2002 indeed are correct

labtestvirt2001:eth0 = ge-5/0/8 (ID=1080)
labtestvirt2001:eth1 = ge-5/0/30 (ID=11516)

We sorted things out in real time and the definitive is:

labtestvirt2001:eth0 = ge-5/0/17 (ID=2187)
labtestvirt2001:eth1 = ge-5/0/31 (ID=11518)

labtestvirt2002:eth0 = ge-5/0/8 (ID=1080)
labtestvirt2002:eth1 = ge-5/0/30 (ID=11516)

labtestvirt2003:eth0 = ge -1/0/ 11 (ID=11856)
labtestvirt2003:eth1 = ge-1/0/12 (ID= 11890)
labtestvirt2003:eth2 = ge-1/0/14 (ID= 11990)

labtestmetal2001:eth0 =       ge-8/0/0 ( ID = 1147)  
labtestmetal2002:eth1 =       ge-8/0/3 (ID =   1247)

Papaul closed this task as Resolved.Feb 7 2018, 12:50 AM

In T183167#3950688, @chasemp wrote:

We sorted things out in real time and the definitive is:

labtestvirt2001:eth0 = ge-5/0/17 (ID=2187)
labtestvirt2001:eth1 = ge-5/0/31 (ID=11518)

labtestvirt2002:eth0 = ge-5/0/8 (ID=1080)
labtestvirt2002:eth1 = ge-5/0/30 (ID=11516)

labtestvirt2003:eth0 = ge -1/0/ 11 (ID=11856)
labtestvirt2003:eth1 = ge-1/0/12 (ID= 11890)

labtestmetal2001:eth0 =       ge-8/0/0 ( ID = 1147)  
labtestmetal2002:eth1 =       ge-8/0/3 (ID =   1247)

I have fixed up the config and descriptions for these.

I did have one port for which I wanted to double(triple?) check because it involves a server not sanity checked here:

labtestvirt2003:eth2 = ge-1/0/14 (ID= 11990)

This currently says

set interfaces ge-1/0/14 description labtestnet2002-eth1

@Papaul can you check the connections for labtestnet2002 to be sure what ports it has connected? ge-1/0/14 the description is in conflict, and if it is labtestvirt2003:eth2 then what is eth1 for labtestnet2002 connected to?

@chasemp
labtestnet2002:eth0 = ge-1/0/13
labtestvirt2003:eth2 = ge-1/0/14

In T183167#3967741, @Papaul wrote:

@chasemp
labtestnet2002:eth0 = ge-1/0/13
labtestvirt2003:eth2 = ge-1/0/14

Thanks @Papaul

@Papaul confirmed that physically labtestnet2002 only has eth0 connected.

[edit interfaces ge-1/0/14]
-   description labtestnet2002-eth1;
+   description labtestvirt2003-eth2;

I have fixed this up so now:

set interfaces ge-1/0/11 description labtestvirt2003-eth0
set interfaces ge-1/0/12 description labtestvirt2003-eth1
set interfaces ge-1/0/14 description labtestvirt2003-eth2
set interfaces ge-5/0/8 description labtestvirt2002-eth0
set interfaces ge-5/0/17 description labtestvirt2001-eth0
set interfaces ge-5/0/30 description labtestvirt2002-eth1
set interfaces ge-5/0/31 description labtestvirt2001-eth1

set interfaces ge-1/0/13 description labtestnet2002-eth0
set interfaces ge-5/0/21 description labtestnet2001-eth0
set interfaces ge-5/0/32 description labtestnet2001-eth1

closing now until we surface something that doesn't make any sense :P

Connect labtestvirt2003 eth1 and eth2 interface(s) to switch fabricClosed, ResolvedPublicActions

Description

Related ObjectsSearch...

Event Timeline

Connect labtestvirt2003 eth1 and eth2 interface(s) to switch fabric
Closed, ResolvedPublic
Actions

Related Objects
Search...