Manage frack switches with Netbox
Open, LowPublic
Actions

Assigned To

None

Authored By

	ayounsi
	Nov 26 2020, 7:01 AM

Description

Similarly to what we did for all the prod servers (T250429), I'd like to explore the possibility of doing the same with the Frack infrastructure.
The goal is to have everything documented in Netbox so all workflows and configs are derived from it (eg. no more manual switch config).

There are 2 things to check:

how to do the initial import, for that we can either rely on:
1. Switch interfaces descriptions (not very reliable)
2. LLDP, but it looks like that none of the frack servers have LLDP enabled?
Adapt the automation scripts (or write new ones) to support the frack particularities:
1. No IP provisioning (looks easy (if frack, skip IPs)
2. Bonded server NICs (more tricky)

I think a good first step if you agree would be to enable LLDP on the servers. As it's useful for troubleshooting regardless of automation.

Details

	Subject	Repo	Branch	Lines +/-
	lldp: add new per interface neighbours fact	operations/puppet	production	+2 -1

Customize query in gerrit

Related Objects

Mentioned Here: T250429: Homer: Netbox driven switch interfaces

Event Timeline

ayounsi created this task.Nov 26 2020, 7:01 AM

Restricted Application added a project: SRE. · View Herald TranscriptNov 26 2020, 7:01 AM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Jgreen moved this task from Triage to Backlog on the fundraising-tech-ops board.Nov 30 2020, 3:36 PM

@ayounsi LLDP should be possible for us. We are at the start of the busy time for fundraising so it'll will probably be a few weeks or most likely January when we would feel safe rolling this out. I'll get the puppet setup and test in our VM setup until then.

Thanks! let me know if I can help!

Another useful thing would be to duplicate the production "networking" puppet fact, to have LLDP, IPs, etc. exposed there as well.

In T268802#6662467, @ayounsi wrote:

Another useful thing would be to duplicate the production "networking" puppet fact, to have LLDP, IPs, etc. exposed there as well.

In relation to the lldp facts all yo would need is to copy lldp.rb to the correct folder under $some_module in your puppet repo e.g modules/$some_module/lib/lib/facter/lldp.rb

The fact file produces both legacy and structured facts. the latter needs at least facter3 for them to be exposed. I belive that facter 2 will still be able to use lldp.rb above and will just ignore the structured facts . you can test this manually by copying the above file to sudo puppet config print factpath and sudo run facter -p lldp

$ cp lldp.rb /var/lib/puppet/lib/facter/lldp.rb
$ sudo facter -p lldp
{
  ens5 => {
    neighbor => "ganeti1007.eqiad.wmnet",
    port => "fe:16:61:67:4d:d8",
    descr => "tap4"
  }
}

jbond triaged this task as Medium priority.Dec 9 2020, 11:57 AM

@jbond Thanks for the pointers. I have started testing this in our VM setup and it looks like getting lldp in place should be easy to do.

I do have a question about the use of facter though. In my testing with lldbctl I see multiple neighbors for an interface. When testing through facter with the production lldp.rb, only the last neighbor on the interface is returned. Is this expected and by design?

Thanks for any info and no rush needed on response.

In T268802#6681000, @Dwisehaupt wrote:

I do have a question about the use of facter though. In my testing with lldbctl I see multiple neighbors for an interface.

Although this is not a bug it is a bit unusual AFAIK you should only really see multiple neighbours if the interface is bonded or plugged into something like a hub as oppose to a switch. Could be related to how the vSwitch is configured?

When testing through facter with the production lldp.rb, only the last neighbor on the interface is returned. Is this expected and by design?

This is arguably a bug in the lldp fact however its not one we see in production and fixing it may be more difficult then expected (need to check current use cases)

are you able to provide me the output of the following commands so i can investigate further

sudo lldpctl -f xml
sudo facter -p lldp

As a side note to @ayounsi i just noticed that for ganeti and cloud VMs we can use the lldp neighbour to determine which host its on.

Yeah, our local VM setups are definitely hub like so that makes sense. We aren't currently using vSwitch and manage our different networks by using a host that simulates a PFW to some degree.

If this is difficult to address in the lldp fact then don't worry too much. I doubt we would be hitting this when we roll to the proper setup.

Here is the output from the commands:

(vb)dallas@frpm1001:~$ sudo lldpctl -f xml

<?xml version="1.0" encoding="UTF-8"?>
<lldp label="LLDP neighbors">
 <interface label="Interface" name="enp0s3" via="LLDP" rid="1" age="0 day, 01:44:56">
  <chassis label="Chassis">
   <id label="ChassisID" type="mac">18:66:da:63:2f:1c</id>
   <name label="SysName">frauth1001.frack.eqiad.wmnet</name>
   <descr label="SysDescr">Debian GNU/Linux 10 (buster) Linux 4.19.0-10-amd64 #1 SMP Debian 4.19.132-1 (2020-07-24) x86_64</descr>
   <mgmt-ip label="MgmtIP">10.64.40.68</mgmt-ip>
   <capability label="Capability" type="Bridge" enabled="off"/>
   <capability label="Capability" type="Router" enabled="off"/>
   <capability label="Capability" type="Wlan" enabled="off"/>
   <capability label="Capability" type="Station" enabled="on"/>
  </chassis>
  <port label="Port">
   <id label="PortID" type="mac">18:66:da:63:2f:1c</id>
   <descr label="PortDescr">eth0</descr>
   <ttl label="TTL">120</ttl>
   <auto-negotiation label="PMD autoneg" supported="yes" enabled="yes">
    <advertised label="Adv" type="10Base-T" hd="yes" fd="yes"/>
    <advertised label="Adv" type="100Base-TX" hd="yes" fd="yes"/>
    <advertised label="Adv" type="1000Base-T" hd="no" fd="yes"/>
    <current label="MAU oper type">1000BaseTFD - Four-pair Category 5 UTP, full duplex mode</current>
   </auto-negotiation>
  </port>
 </interface>
 <interface label="Interface" name="enp0s3" via="LLDP" rid="2" age="0 day, 01:44:39">
  <chassis label="Chassis">
   <id label="ChassisID" type="mac">94:18:82:89:10:38</id>
   <name label="SysName">frlog1001.frack.eqiad.wmnet</name>
   <descr label="SysDescr">Debian GNU/Linux 10 (buster) Linux 4.19.0-10-amd64 #1 SMP Debian 4.19.132-1 (2020-07-24) x86_64</descr>
   <mgmt-ip label="MgmtIP">10.64.40.72</mgmt-ip>
   <capability label="Capability" type="Bridge" enabled="off"/>
   <capability label="Capability" type="Router" enabled="off"/>
   <capability label="Capability" type="Wlan" enabled="off"/>
   <capability label="Capability" type="Station" enabled="on"/>
  </chassis>
  <port label="Port">
   <id label="PortID" type="mac">94:18:82:89:10:38</id>
   <descr label="PortDescr">enp0s3</descr>
   <ttl label="TTL">120</ttl>
   <auto-negotiation label="PMD autoneg" supported="yes" enabled="yes">
    <advertised label="Adv" type="10Base-T" hd="yes" fd="yes"/>
    <advertised label="Adv" type="100Base-TX" hd="yes" fd="yes"/>
    <advertised label="Adv" type="1000Base-T" hd="no" fd="yes"/>
    <current label="MAU oper type">1000BaseTFD - Four-pair Category 5 UTP, full duplex mode</current>
   </auto-negotiation>
  </port>
 </interface>
 <interface label="Interface" name="enp0s3" via="LLDP" rid="3" age="0 day, 01:44:36">
  <chassis label="Chassis">
   <id label="ChassisID" type="mac">d0:94:66:97:a2:24</id>
   <name label="SysName">frav1002.frack.eqiad.wmnet</name>
   <descr label="SysDescr">Debian GNU/Linux 10 (buster) Linux 4.19.0-10-amd64 #1 SMP Debian 4.19.132-1 (2020-07-24) x86_64</descr>
   <mgmt-ip label="MgmtIP">10.64.40.74</mgmt-ip>
   <capability label="Capability" type="Bridge" enabled="off"/>
   <capability label="Capability" type="Router" enabled="off"/>
   <capability label="Capability" type="Wlan" enabled="off"/>
   <capability label="Capability" type="Station" enabled="on"/>
  </chassis>
  <port label="Port">
   <id label="PortID" type="mac">d0:94:66:97:a2:24</id>
   <descr label="PortDescr">enp0s3</descr>
   <ttl label="TTL">120</ttl>
   <auto-negotiation label="PMD autoneg" supported="yes" enabled="yes">
    <advertised label="Adv" type="10Base-T" hd="yes" fd="yes"/>
    <advertised label="Adv" type="100Base-TX" hd="yes" fd="yes"/>
    <advertised label="Adv" type="1000Base-T" hd="no" fd="yes"/>
    <current label="MAU oper type">1000BaseTFD - Four-pair Category 5 UTP, full duplex mode</current>
   </auto-negotiation>
  </port>
 </interface>
</lldp>

(vb)dallas@frpm1001:~$ sudo facter -p lldp

{
  enp0s3 => {
    neighbor => "frav1002.frack.eqiad.wmnet",
    port => "d0:94:66:97:a2:24",
    descr => "enp0s3"
  }
}

Change 649592 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] lldp: add new per interface neighbours fact

https://gerrit.wikimedia.org/r/649592

gerritbot added a project: Patch-For-Review.Dec 15 2020, 9:33 AM

@Dwisehaupt I have created a patch to add a new per interface neighbours fact. The output looks as follows

$ sudo facter -p lldp                                                                                                                      
{
  eno1 => {
    neighbors => [
      "asw2-d-eqiad"
    ],
    neighbor => "asw2-d-eqiad",
    port => "ge-6/0/5",
    descr => "sretest1002:eno1 {#}",
    mtu => 9192,
    vlans => {
      untagged_vlan => 1020,
      mode => "access"
    }
  }
}

@jbond Thanks. I pulled the patch in and it runs great in the VM.

(vb)dallas@frpm1001:~$ sudo facter -p lldp
{
  enp0s3 => {
    neighbors => [
      "frlog1001.frack.eqiad.wmnet",
      "frav1002.frack.eqiad.wmnet",
      "frauth1001.frack.eqiad.wmnet"
    ],
    neighbor => "frauth1001.frack.eqiad.wmnet",
    port => "18:66:da:63:2f:1c",
    descr => "eth0"
  }
}

Dwisehaupt claimed this task.Jan 6 2021, 6:10 PM

Dwisehaupt moved this task from Backlog to Up Next on the fundraising-tech-ops board.

This has been pushed to the frack puppet instance and will be rolling across hosts in the next few minutes.

Dwisehaupt closed this task as Resolved.Jan 22 2021, 8:58 PM

Dwisehaupt moved this task from In Progress to Done on the fundraising-tech-ops board.

Thanks for taking care of that.
Re-opening as the scope of this task is larger than the Puppet change.

Jgreen moved this task from Done to Watching on the fundraising-tech-ops board.Jan 26 2021, 4:08 PM

Change 649592 merged by Jbond:

[operations/puppet@production] lldp: add new per interface neighbours fact

https://gerrit.wikimedia.org/r/649592

Maintenance_bot removed a project: Patch-For-Review.Apr 7 2021, 1:10 PM

cmooney subscribed.Jun 4 2021, 3:26 PM

Aklapper added a project: Infrastructure-Foundations.Jun 21 2021, 8:59 PM

joanna_borun lowered the priority of this task from Medium to Low.Mon, Apr 8, 2:51 PM

Manage frack switches with NetboxOpen, LowPublicActions

Description

Details

Related Objects

Event Timeline

Manage frack switches with Netbox
Open, LowPublic
Actions