Page MenuHomePhabricator
Create Task
Maniphest T221388

Test dhcp-option 82
Closed, ResolvedPublic0 Estimated Story Points
Actions

Description

Added the following to asw2-b-eqiad:

[edit vlan cloud-hosts1-b-eqiad]
+    forwarding-options {
+        dhcp-security {
+            option-82 {
+                remote-id {
+                    host-name;
+                }
+            }
+        }
+    }

Ran tcpdump on install1002 sudo tcpdump -s0 -w dhcp-82.pcap port 67

We can see the new option being added:

Option: (82) Agent Information Option
    Length: 60
    Option 82 Suboption: (1) Agent Circuit ID
        Length: 32
        Agent Circuit ID: 78652d342f302f33392e303a636c6f75642d686f73747331...
        (Which translate to xe-4/0/39.0:cloud-hosts1-b-eqiad)
    Option 82 Suboption: (2) Agent Remote ID
        Length: 24
        Agent Remote ID: 617377322d622d65716961643a78652d342f302f33392e30
       (Which translates to (asw2-b-eqiad:xe-4/0/39.0)

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
dhcp: fix typo in opt82 file pathoperations/software/spicerackmaster+3 -3
install_server: standardize DHCP includesoperations/puppetproduction+17 -12
dhcp: Add module for manipulating dynamic DHCP entriesoperations/software/spicerackmaster+604 -1
Option 82: don't use use-vlan-idoperations/homer/publicmaster+0 -1
Option 82: use-vlan-idoperations/homer/publicmaster+1 -0
dhcp: specify owner and group for option 82 configoperations/puppetproduction+2 -0
dhcpd: move sretest1002 to option 82operations/puppetproduction+0 -6
dhcpd: create and include files for option 82operations/puppetproduction+14 -1
Customize query in gerrit

Related Objects

Event Timeline

ayounsi created this task.Apr 18 2019, 4:12 PM
ayounsi closed this task as Resolved.
ayounsi triaged this task as Low priority.
Restricted Application added a project: SRE. · View Herald TranscriptApr 18 2019, 4:12 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
ayounsi mentioned this in T269855: Manage DHCP from Netbox.Dec 10 2020, 2:23 PM
Volans reopened this task as Open.Feb 3 2021, 3:32 PM
Volans claimed this task.

Re-opening as we're aiming to implement it this quarter.

Stashbot added a comment.Feb 3 2021, 3:32 PM

Mentioned in SAL (#wikimedia-operations) [2021-02-03T15:32:42Z] <volans> disabling puppet on install1003 for a quick test for T221388

Stashbot added a comment.Feb 3 2021, 4:13 PM

Mentioned in SAL (#wikimedia-operations) [2021-02-03T16:13:14Z] <volans> enabled puppet on install1003 after the test T221388

Volans added a comment.Feb 3 2021, 4:16 PM

I tested the config with:

host sretest1001 {
    host-identifier option agent.circuit-id "ge-3/0/15.0:private1-d-eqiad";
     fixed-address sretest1001.eqiad.wmnet;
}

And it seemed to work as expected. I need to perform a more in depth test as I want to reimage sretest1001 with the same config so to ensure that this works across the whole reimage process, but looks promising.
Another bits to look at is the Junos side of the configuration.
The tested configuration was:

ayounsi@asw2-d-eqiad# show | compare 
[edit vlans private1-d-eqiad]
+    forwarding-options {
+        dhcp-security {
+            option-82 {
+                remote-id {
+                    host-name;
+                }
+            }
+        }
+    }

We can check if we could have the switch hostname as part of the injected data.

Volans added a comment.EditedFeb 3 2021, 4:52 PM

Adding the circuit-id prefix host-name setting and removing the remote-id that we're not gonna use, the circuit ID includes the switch hostname too, so becoming asw2-d-eqiad:ge-3/0/15.0:private1-d-eqiad. That should be enough to be able to set the DHCP with in a unique manner.

References used:
https://www.juniper.net/documentation/en_US/junos/topics/topic-map/example-setting-up-dhcp-option82-no-relay.html#id-setting-up-dhcp-option-82-on-the-switch-with-no-relay-els

http://www.miquels.cistron.nl/isc-dhcpd/

Volans added a comment.Feb 3 2021, 6:18 PM

The config for the above test used:

circuit-id {
    prefix {
        host-name;
    }
}
gerritbot added a comment.Feb 10 2021, 4:05 PM

Change 663233 had a related patch set uploaded (by Volans; owner: Volans):
[operations/puppet@production] dhcpd: create and include files for option 82

https://gerrit.wikimedia.org/r/663233

gerritbot added a project: Patch-For-Review.Feb 10 2021, 4:05 PM

Change 663234 had a related patch set uploaded (by Volans; owner: Volans):
[operations/puppet@production] dhcpd: move sretest1002 to option 82

https://gerrit.wikimedia.org/r/663234

gerritbot added a comment.Feb 11 2021, 11:28 AM

Change 663233 merged by Volans:
[operations/puppet@production] dhcpd: create and include files for option 82

https://gerrit.wikimedia.org/r/663233

gerritbot added a comment.Feb 11 2021, 11:30 AM

Change 663234 merged by Volans:
[operations/puppet@production] dhcpd: move sretest1002 to option 82

https://gerrit.wikimedia.org/r/663234

Maintenance_bot removed a project: Patch-For-Review.Feb 11 2021, 12:10 PM
Volans added a comment.Feb 11 2021, 1:28 PM

With the above patches merged, and with:

root@install1003:/etc/dhcp# cat opt82-entries.ttyS1-115200
host sretest1002 {
    host-identifier option agent.circuit-id "asw2-d-eqiad:ge-6/0/5.0:private1-d-eqiad";
    fixed-address sretest1002.eqiad.wmnet;
}

The DHCP requests from sretest1002 seems to work fine. I'll shortly test also a reimage.

BBlack subscribed.Feb 11 2021, 1:40 PM

I'm probably not up to date on concrete plans built on top of this, but it seems like having the numeric vlan id might be useful metadata here in addition to the abstract name of the vlan (e.g. scenarios where we might do vlan trunking on the main interface of the host and need to see or match that primary-vlan number in some interface setup scripts?)

ayounsi added a comment.Feb 11 2021, 1:44 PM

From the doc:

Specify that the circuit ID suboption value contains the VLAN ID rather than the VLAN name (the default):

[edit vlans vlan-name forwarding-options dhcp-security option-82]
user@switch# set circuit-id use-vlan-id

So it seems possible.

Volans added a comment.Feb 11 2021, 1:54 PM
In T221388#6822703, @BBlack wrote:

I'm probably not up to date on concrete plans built on top of this, but it seems like having the numeric vlan id might be useful metadata here in addition to the abstract name of the vlan (e.g. scenarios where we might do vlan trunking on the main interface of the host and need to see or match that primary-vlan number in some interface setup scripts?)

@BBlack The option on the JunOS side allows to pick either the name or the ID, not both. In terms of assured uniqueness we can surely use the ID if the name can be duplicated (I don't think that Netbox enforces it).

ops-monitoring-bot added a comment.Mar 1 2021, 5:46 PM

Script wmf-auto-reimage was launched by volans on cumin1001.eqiad.wmnet for hosts:

sretest1002.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202103011745_volans_18022_sretest1002_eqiad_wmnet.log.

ops-monitoring-bot added a comment.Mar 1 2021, 6:07 PM

Completed auto-reimage of hosts:

['sretest1002.eqiad.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Mar 2 2021, 9:52 AM

Script wmf-auto-reimage was launched by volans on cumin1001.eqiad.wmnet for hosts:

sretest1002.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202103020952_volans_22946_sretest1002_eqiad_wmnet.log.

ops-monitoring-bot added a comment.Mar 2 2021, 10:10 AM

Completed auto-reimage of hosts:

['sretest1002.eqiad.wmnet']

and were ALL successful.

gerritbot added a comment.Mar 3 2021, 11:27 AM

Change 668041 had a related patch set uploaded (by Volans; owner: Volans):
[operations/puppet@production] dhcp: specify owner and group for option 82 config

https://gerrit.wikimedia.org/r/668041

gerritbot added a project: Patch-For-Review.Mar 3 2021, 11:27 AM
gerritbot added a comment.Mar 3 2021, 1:34 PM

Change 668041 merged by Volans:
[operations/puppet@production] dhcp: specify owner and group for option 82 config

https://gerrit.wikimedia.org/r/668041

Maintenance_bot removed a project: Patch-For-Review.Mar 3 2021, 2:10 PM
gerritbot added a comment.Mar 24 2021, 12:13 PM

Change 674574 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/homer/public@master] Option 82: use-vlan-id

https://gerrit.wikimedia.org/r/674574

gerritbot added a project: Patch-For-Review.Mar 24 2021, 12:13 PM
gerritbot added a comment.Mar 25 2021, 6:53 AM

Change 674574 merged by jenkins-bot:
[operations/homer/public@master] Option 82: use-vlan-id

https://gerrit.wikimedia.org/r/674574

ayounsi mentioned this in rOHPU61376ea9fcf0: Option 82: use-vlan-id.Mar 25 2021, 6:55 AM
gerritbot added a comment.Mar 31 2021, 12:24 AM

Change 675932 had a related patch set uploaded (by CRusnov; author: CRusnov):

[operations/software/spicerack@master] dhcp: Add module for manipulating dynamic DHCP entries

https://gerrit.wikimedia.org/r/675932

jbond subscribed.EditedApr 26 2021, 1:35 PM
In T221388#6822771, @Volans wrote:
In T221388#6822703, @BBlack wrote:

I'm probably not up to date on concrete plans built on top of this, but it seems like having the numeric vlan id might be useful metadata here in addition to the abstract name of the vlan (e.g. scenarios where we might do vlan trunking on the main interface of the host and need to see or match that primary-vlan number in some interface setup scripts?)

@BBlack The option on the JunOS side allows to pick either the name or the ID, not both. In terms of assured uniqueness we can surely use the ID if the name can be duplicated (I don't think that Netbox enforces it).

I have just tested this and it seems that there is currently an issue in that the DHCPOFFER uses the vlan name but the DHCPREQUEST uses the vlan-id this causes isc-dhcp to fail.

Currently the swtiches send a DHCPOFFER which use the vlan-name, this means that if one specifics the vlan-id in DHCP config e.g. asw2-d-eqiad:ge-6/0/5.0:1020 then isc-dhcp dosn't even send an DHCPOFFER and you see the following in the logs

DHCPDISCOVER from 4c:d9:8f:60:af:47 via 10.64.48.3: network 10.64.48.0/22: no free leases

If we use the vlan name in isc-dhcp config e.g. asw2-d-eqiad:ge-6/0/5.0:private1-d-eqiad then we do recive a DHCPOFFER however the DHCPREQUEST which is sent by the switches now use the vlan-id this causes the following error in the logs

DHCPREQUEST for 10.64.48.139 (208.80.154.32) from 4c:d9:8f:60:af:47 via 10.64.48.3: unknown lease 10.64.48.1

I couldn't logon to the switch so not sure if this is a config issues, perhaps not updating all the bits however it smells a bit more like a bug to me (i.e. you shouldn't be able to configure this behaviour as its borked)

I have left a tcpdump created from trying to image sretest1002 in install1003:/home/jbond/dhcp_sretest1002.pcap however see below for the main bits

DHCPDISCOVER
14:50:21.675682 IP (tos 0x0, ttl 64, id 13544, offset 0, flags [none], proto UDP (17), length 633)
    10.64.48.3.bootps > install1003.wikimedia.org.bootps: [udp sum ok] BOOTP/DHCP, Request from 4c:d9:8f:60:af:47 (oui Unknown), length 605, hops 1, xid 0x9060af47, secs 4, Flags [Broadcast] (0x8000)
          Gateway-IP 10.64.48.3
          Client-Ethernet-Address 4c:d9:8f:60:af:47 (oui Unknown)
          Vendor-rfc1048 Extensions
            Magic Cookie 0x63825363
    ....
            Agent-Information Option 82, length 54: 
              Circuit-ID SubOption 1, length 40: asw2-d-eqiad:ge-6/0/5.0:private1-d-eqiad
              Remote-ID SubOption 2, length 10: ge-6/0/5.0
            END Option 255, length 0
            PAD Option 0, length 0, occurs 156
            xT21690 Option 21690, length 70: [|rfc1048 70]
DHCPOFFER
14:50:21.686454 IP (tos 0x0, ttl 64, id 40970, offset 0, flags [DF], proto UDP (17), length 467)
    install1003.wikimedia.org.bootps > 10.64.48.3.bootps: [bad udp cksum 0xa684 -> 0xf4a3!] BOOTP/DHCP, Reply, length 439, hops 1, xid 0x9060af47, secs 4, Flags [Broadcast] (0x8000)
          Your-IP 10.64.48.139
          Server-IP install1003.wikimedia.org
          Gateway-IP 10.64.48.3
          Client-Ethernet-Address 4c:d9:8f:60:af:47 (oui Unknown)
          file "lpxelinux.0"
          Vendor-rfc1048 Extensions
            Magic Cookie 0x63825363
...
            Agent-Information Option 82, length 54: 
              Circuit-ID SubOption 1, length 40: asw2-d-eqiad:ge-6/0/5.0:private1-d-eqiad
              Remote-ID SubOption 2, length 10: ge-6/0/5.0
            END Option 255, length 0
DHCPREQUEST
14:50:25.769773 IP (tos 0x0, ttl 64, id 15669, offset 0, flags [none], proto UDP (17), length 620)
    10.64.48.3.bootps > install1003.wikimedia.org.bootps: [udp sum ok] BOOTP/DHCP, Request from 4c:d9:8f:60:af:47 (oui Unknown), length 592, hops 1, xid 0x9060af47, secs 4, Flags [Broadcast] (0x8000)
          Gateway-IP 10.64.48.3
          Client-Ethernet-Address 4c:d9:8f:60:af:47 (oui Unknown)
          Vendor-rfc1048 Extensions
            Magic Cookie 0x63825363
...
            Agent-Information Option 82, length 42: 
              Circuit-ID SubOption 1, length 28: asw2-d-eqiad:ge-6/0/5.0:1020
              Remote-ID SubOption 2, length 10: ge-6/0/5.0
            END Option 255, length 0
            PAD Option 0, length 0, occurs 200

for now i think we should switch back to using the default vlan name

gerritbot added a comment.Apr 26 2021, 1:36 PM

Change 682629 had a related patch set uploaded (by Jbond; author: Jbond):

[operations/homer/public@master] Option 82: don't use use-vlan-id

https://gerrit.wikimedia.org/r/682629

gerritbot added a comment.Apr 26 2021, 1:49 PM

Change 682629 merged by jenkins-bot:

[operations/homer/public@master] Option 82: don't use use-vlan-id

https://gerrit.wikimedia.org/r/682629

jenkins-bot mentioned this in rOHPUc9e7b42ea666: Option 82: don't use use-vlan-id.Apr 26 2021, 1:50 PM
jbond added a comment.Apr 26 2021, 3:28 PM

Just to confirm after removing use-vlan-id re-imaging of sretest1002 worked fine

Volans added a comment.Apr 26 2021, 3:42 PM

Thanks for the confirmation and sorry about the trouble @jbond

gerritbot added a comment.May 6 2021, 4:06 PM

Change 675932 merged by CRusnov:

[operations/software/spicerack@master] dhcp: Add module for manipulating dynamic DHCP entries

https://gerrit.wikimedia.org/r/675932

Volans mentioned this in T217074: Reduce Manual Steps in Provisioning by 4.May 17 2021, 9:37 AM
cmooney subscribed.Jun 4 2021, 9:29 AM
Aklapper added a project: Infrastructure-Foundations.Jun 21 2021, 9:00 PM
gerritbot added a comment.Sep 27 2021, 9:12 AM

Change 723996 had a related patch set uploaded (by Volans; author: Volans):

[operations/puppet@production] install_server: standardize DHCP includes

https://gerrit.wikimedia.org/r/723996

Stashbot added a comment.Sep 27 2021, 11:02 AM

Mentioned in SAL (#wikimedia-operations) [2021-09-27T11:02:20Z] <volans> disabling puppet on install hosts to deploy 723996 - T221388

gerritbot added a comment.Sep 27 2021, 11:02 AM

Change 723996 merged by Volans:

[operations/puppet@production] install_server: standardize DHCP includes

https://gerrit.wikimedia.org/r/723996

Stashbot added a comment.Sep 27 2021, 11:09 AM

Mentioned in SAL (#wikimedia-operations) [2021-09-27T11:09:31Z] <volans> re-enabled puppet on install hosts after deployment of g/723996 - T221388

gerritbot added a comment.Sep 28 2021, 6:59 AM

Change 724337 had a related patch set uploaded (by Volans; author: Volans):

[operations/software/spicerack@master] dhcp: fix typo in opt82 file path

https://gerrit.wikimedia.org/r/724337

gerritbot added a comment.Sep 28 2021, 7:10 AM

Change 724337 merged by jenkins-bot:

[operations/software/spicerack@master] dhcp: fix typo in opt82 file path

https://gerrit.wikimedia.org/r/724337

Volans closed this task as Resolved.Oct 11 2021, 7:56 AM

The test of the option 82 has been successful and we're switching to this system for all physical hosts DHCP settings in T269855.
In the final setup we're using:

  • The VLAN name instead of the ID for the issue reported above
  • The primary IPv4 address that is configured in Netbox instead of the FQDN to remove a direct dependency between DCHP and DNS and because the DNS configuration is also coming from Netbox data.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits