Page MenuHomePhabricator

Manage frack switches with Netbox
Open, MediumPublic

Description

Similarly to what we did for all the prod servers (T250429), I'd like to explore the possibility of doing the same with the Frack infrastructure.
The goal is to have everything documented in Netbox so all workflows and configs are derived from it (eg. no more manual switch config).

There are 2 things to check:

  1. how to do the initial import, for that we can either rely on:
    1. Switch interfaces descriptions (not very reliable)
    2. LLDP, but it looks like that none of the frack servers have LLDP enabled?
  2. Adapt the automation scripts (or write new ones) to support the frack particularities:
    1. No IP provisioning (looks easy (if frack, skip IPs)
    2. Bonded server NICs (more tricky)

I think a good first step if you agree would be to enable LLDP on the servers. As it's useful for troubleshooting regardless of automation.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

@ayounsi LLDP should be possible for us. We are at the start of the busy time for fundraising so it'll will probably be a few weeks or most likely January when we would feel safe rolling this out. I'll get the puppet setup and test in our VM setup until then.

Thanks! let me know if I can help!

Another useful thing would be to duplicate the production "networking" puppet fact, to have LLDP, IPs, etc. exposed there as well.

Another useful thing would be to duplicate the production "networking" puppet fact, to have LLDP, IPs, etc. exposed there as well.

In relation to the lldp facts all yo would need is to copy lldp.rb to the correct folder under $some_module in your puppet repo e.g modules/$some_module/lib/lib/facter/lldp.rb

The fact file produces both legacy and structured facts. the latter needs at least facter3 for them to be exposed. I belive that facter 2 will still be able to use lldp.rb above and will just ignore the structured facts . you can test this manually by copying the above file to sudo puppet config print factpath and sudo run facter -p lldp

$ cp lldp.rb /var/lib/puppet/lib/facter/lldp.rb
$ sudo facter -p lldp
{
  ens5 => {
    neighbor => "ganeti1007.eqiad.wmnet",
    port => "fe:16:61:67:4d:d8",
    descr => "tap4"
  }
}
jbond triaged this task as Medium priority.Dec 9 2020, 11:57 AM

@jbond Thanks for the pointers. I have started testing this in our VM setup and it looks like getting lldp in place should be easy to do.

I do have a question about the use of facter though. In my testing with lldbctl I see multiple neighbors for an interface. When testing through facter with the production lldp.rb, only the last neighbor on the interface is returned. Is this expected and by design?

Thanks for any info and no rush needed on response.

I do have a question about the use of facter though. In my testing with lldbctl I see multiple neighbors for an interface.

Although this is not a bug it is a bit unusual AFAIK you should only really see multiple neighbours if the interface is bonded or plugged into something like a hub as oppose to a switch. Could be related to how the vSwitch is configured?

When testing through facter with the production lldp.rb, only the last neighbor on the interface is returned. Is this expected and by design?

This is arguably a bug in the lldp fact however its not one we see in production and fixing it may be more difficult then expected (need to check current use cases)

are you able to provide me the output of the following commands so i can investigate further

  • sudo lldpctl -f xml
  • sudo facter -p lldp

As a side note to @ayounsi i just noticed that for ganeti and cloud VMs we can use the lldp neighbour to determine which host its on.

Yeah, our local VM setups are definitely hub like so that makes sense. We aren't currently using vSwitch and manage our different networks by using a host that simulates a PFW to some degree.

If this is difficult to address in the lldp fact then don't worry too much. I doubt we would be hitting this when we roll to the proper setup.

Here is the output from the commands:

(vb)dallas@frpm1001:~$ sudo lldpctl -f xml

<?xml version="1.0" encoding="UTF-8"?>
<lldp label="LLDP neighbors">
 <interface label="Interface" name="enp0s3" via="LLDP" rid="1" age="0 day, 01:44:56">
  <chassis label="Chassis">
   <id label="ChassisID" type="mac">18:66:da:63:2f:1c</id>
   <name label="SysName">frauth1001.frack.eqiad.wmnet</name>
   <descr label="SysDescr">Debian GNU/Linux 10 (buster) Linux 4.19.0-10-amd64 #1 SMP Debian 4.19.132-1 (2020-07-24) x86_64</descr>
   <mgmt-ip label="MgmtIP">10.64.40.68</mgmt-ip>
   <capability label="Capability" type="Bridge" enabled="off"/>
   <capability label="Capability" type="Router" enabled="off"/>
   <capability label="Capability" type="Wlan" enabled="off"/>
   <capability label="Capability" type="Station" enabled="on"/>
  </chassis>
  <port label="Port">
   <id label="PortID" type="mac">18:66:da:63:2f:1c</id>
   <descr label="PortDescr">eth0</descr>
   <ttl label="TTL">120</ttl>
   <auto-negotiation label="PMD autoneg" supported="yes" enabled="yes">
    <advertised label="Adv" type="10Base-T" hd="yes" fd="yes"/>
    <advertised label="Adv" type="100Base-TX" hd="yes" fd="yes"/>
    <advertised label="Adv" type="1000Base-T" hd="no" fd="yes"/>
    <current label="MAU oper type">1000BaseTFD - Four-pair Category 5 UTP, full duplex mode</current>
   </auto-negotiation>
  </port>
 </interface>
 <interface label="Interface" name="enp0s3" via="LLDP" rid="2" age="0 day, 01:44:39">
  <chassis label="Chassis">
   <id label="ChassisID" type="mac">94:18:82:89:10:38</id>
   <name label="SysName">frlog1001.frack.eqiad.wmnet</name>
   <descr label="SysDescr">Debian GNU/Linux 10 (buster) Linux 4.19.0-10-amd64 #1 SMP Debian 4.19.132-1 (2020-07-24) x86_64</descr>
   <mgmt-ip label="MgmtIP">10.64.40.72</mgmt-ip>
   <capability label="Capability" type="Bridge" enabled="off"/>
   <capability label="Capability" type="Router" enabled="off"/>
   <capability label="Capability" type="Wlan" enabled="off"/>
   <capability label="Capability" type="Station" enabled="on"/>
  </chassis>
  <port label="Port">
   <id label="PortID" type="mac">94:18:82:89:10:38</id>
   <descr label="PortDescr">enp0s3</descr>
   <ttl label="TTL">120</ttl>
   <auto-negotiation label="PMD autoneg" supported="yes" enabled="yes">
    <advertised label="Adv" type="10Base-T" hd="yes" fd="yes"/>
    <advertised label="Adv" type="100Base-TX" hd="yes" fd="yes"/>
    <advertised label="Adv" type="1000Base-T" hd="no" fd="yes"/>
    <current label="MAU oper type">1000BaseTFD - Four-pair Category 5 UTP, full duplex mode</current>
   </auto-negotiation>
  </port>
 </interface>
 <interface label="Interface" name="enp0s3" via="LLDP" rid="3" age="0 day, 01:44:36">
  <chassis label="Chassis">
   <id label="ChassisID" type="mac">d0:94:66:97:a2:24</id>
   <name label="SysName">frav1002.frack.eqiad.wmnet</name>
   <descr label="SysDescr">Debian GNU/Linux 10 (buster) Linux 4.19.0-10-amd64 #1 SMP Debian 4.19.132-1 (2020-07-24) x86_64</descr>
   <mgmt-ip label="MgmtIP">10.64.40.74</mgmt-ip>
   <capability label="Capability" type="Bridge" enabled="off"/>
   <capability label="Capability" type="Router" enabled="off"/>
   <capability label="Capability" type="Wlan" enabled="off"/>
   <capability label="Capability" type="Station" enabled="on"/>
  </chassis>
  <port label="Port">
   <id label="PortID" type="mac">d0:94:66:97:a2:24</id>
   <descr label="PortDescr">enp0s3</descr>
   <ttl label="TTL">120</ttl>
   <auto-negotiation label="PMD autoneg" supported="yes" enabled="yes">
    <advertised label="Adv" type="10Base-T" hd="yes" fd="yes"/>
    <advertised label="Adv" type="100Base-TX" hd="yes" fd="yes"/>
    <advertised label="Adv" type="1000Base-T" hd="no" fd="yes"/>
    <current label="MAU oper type">1000BaseTFD - Four-pair Category 5 UTP, full duplex mode</current>
   </auto-negotiation>
  </port>
 </interface>
</lldp>
(vb)dallas@frpm1001:~$ sudo facter -p lldp

{
  enp0s3 => {
    neighbor => "frav1002.frack.eqiad.wmnet",
    port => "d0:94:66:97:a2:24",
    descr => "enp0s3"
  }
}

Change 649592 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] lldp: add new per interface neighbours fact

https://gerrit.wikimedia.org/r/649592

@Dwisehaupt I have created a patch to add a new per interface neighbours fact. The output looks as follows

$ sudo facter -p lldp                                                                                                                      
{
  eno1 => {
    neighbors => [
      "asw2-d-eqiad"
    ],
    neighbor => "asw2-d-eqiad",
    port => "ge-6/0/5",
    descr => "sretest1002:eno1 {#}",
    mtu => 9192,
    vlans => {
      untagged_vlan => 1020,
      mode => "access"
    }
  }
}

@jbond Thanks. I pulled the patch in and it runs great in the VM.

(vb)dallas@frpm1001:~$ sudo facter -p lldp
{
  enp0s3 => {
    neighbors => [
      "frlog1001.frack.eqiad.wmnet",
      "frav1002.frack.eqiad.wmnet",
      "frauth1001.frack.eqiad.wmnet"
    ],
    neighbor => "frauth1001.frack.eqiad.wmnet",
    port => "18:66:da:63:2f:1c",
    descr => "eth0"
  }
}

This has been pushed to the frack puppet instance and will be rolling across hosts in the next few minutes.

Dwisehaupt moved this task from In Progress to Done on the fundraising-tech-ops board.
ayounsi removed Dwisehaupt as the assignee of this task.

Thanks for taking care of that.
Re-opening as the scope of this task is larger than the Puppet change.

Change 649592 merged by Jbond:

[operations/puppet@production] lldp: add new per interface neighbours fact

https://gerrit.wikimedia.org/r/649592