Page MenuHomePhabricator
Create Task
Maniphest T190424

modify labs-hosts1-vlans for http load of installer kernel
Closed, ResolvedPublic
Actions

Description

So the installer dhcp lease file was recently changed to start loading ALL kernel images over http, rather than tftp. The change is a good one, since http is far less error prone on high latency connections.

However, the two labs-hosts1-vlans: cloud-hosts1-b-codfw and cloud-hosts1-b-eqiad have extra ACLs in place for more restrictions. Can we please have the http port(s) allowed access to this vlan, similar to tftp, so we can load the kernel images for these hosts identically? (Presently we had to shift labvirt* hosts back to tftp load of kernel via the dhcpd lease file.)

Details

ProjectBranchLines +/-Subject
operations/dnsmaster+3 -0adding cloudvirt1024b dns entry
Customize query in gerrit

Related Objects

Event Timeline

RobH triaged this task as Medium priority.Mar 22 2018, 4:28 PM
RobH created this task.
Restricted Application added a project: SRE. · View Herald TranscriptMar 22 2018, 4:28 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
ayounsi assigned this task to RobH.Mar 23 2018, 4:06 PM
ayounsi added a subscriber: ayounsi.

For ACLs, please be more specific, ideally mentioning a source/destination IP(s) and port(s).

Taking a random labvirt* host as well as some well known ports used by http (I don't know which one is correct) flows seem to be going through.

ayounsi@labvirt1015:~$ nc -zv install1002.wikimedia.org 8080
Connection to install1002.wikimedia.org 8080 port [tcp/http-alt] succeeded!
ayounsi@labvirt1015:~$ nc -zv install1002.wikimedia.org 80
Connection to install1002.wikimedia.org 80 port [tcp/http] succeeded!
RobH added a subscriber: faidon.Mar 23 2018, 4:18 PM

so the symptoms of this were us trying to PXE boot labvirt1021 (10.64.20.40) and labvirt1022 (10.64.20.41). During the PXE boot, it gets the DHCP offer from install1002 (208.80.154.22), and then we see install1002 attempt to send the kernel over http, and it eventually times out. If we change the dhcp file to load via tftp, it loads the kernel fine.

@faidon had suggested that it may likely be due to be a router ACLs, and it seemed to support that since it worked when we changed it. Perhaps he may have some insight?

RobH reassigned this task from RobH to faidon.Mar 23 2018, 4:18 PM
RobH reassigned this task from faidon to ayounsi.Jun 6 2018, 7:16 PM

So, this bit me again today, and it would be nice to make these tftp loads via http like the rest of production.

I'm going to change this assignment to @ayounsi since he is our day to day network admin.

ayounsi added a comment.Jun 7 2018, 7:40 AM

It looks like the dhcp bit is something like:

option pxelinux.pathprefix "http://apt.wikimedia.org/tftpboot/jessie-installer/";
filename "lpxelinux.0";

From labvirt1021 I can reach that file properly:
labvirt1021:~$ curl -v http://apt.wikimedia.org/tftpboot/jessie-installer/pxelinux.0

So it doesn't seem to be a firewall issue.

@RobH
1/ are there logs of the failure?
2/ Is there a host we can use to test it?
3/ Can the host resolve apt.wikimedia.org during its PXE boot? Or can we try to hardcode 208.80.154.22?

ayounsi reassigned this task from ayounsi to RobH.Jul 13 2018, 11:25 PM
MoritzMuehlenhoff added a subscriber: MoritzMuehlenhoff.Aug 31 2018, 10:33 AM

@ayounsi : I can still reproduce this with an installation of cloudvirt1023, I can see in syslog that atftpd is serving lpxelinux.0 to 10.64.20.42 and I can see on the serial console that the PXE boot firmware doesn't get a reply. Ping me when you have some to debug this?

MoritzMuehlenhoff added a comment.Sep 5 2018, 5:03 PM

Arzhel and I had a look and this doesn't seem to be ACL-related, the tftp packets are flowing in both directions. This is possibly a bug in the firmware, see https://phabricator.wikimedia.org/T199125#4560182

When the firmware is updated, I'll retest the installation and close if confirmed.

RobH added a comment.Sep 6 2018, 7:01 PM

Ok, so we've been discussing this in IRC.

when trying to use cloudvirt1023 in the labs-hosts1-b-eqiad vlan, if it has NO specific entry for the kernel in its dhcp stanza, it uses the default HTTP load of stretch. It then fails, as it sends the file over http, and it never arrives to cloudvirt1023.

I modified the dhcp file to load it over tftp instead, and it works!
host cloudvirt1023 {

hardware ethernet D0:94:66:61:17:C7;
fixed-address cloudvirt1023.eqiad.wmnet;
option pxelinux.pathprefix "stretch-installer/";
filename "stretch-installer/debian-installer/amd64/pxelinux.0";

}

Then I booted into the installer jsut fine, where it hit the no disk driver issue for the Perc H740p.

gerritbot added a subscriber: gerritbot.Sep 6 2018, 7:06 PM

Change 458563 had a related patch set uploaded (by RobH; owner: RobH):
[operations/dns@master] adding cloudvirt1024b dns entry

https://gerrit.wikimedia.org/r/458563

gerritbot added a project: Patch-For-Review.Sep 6 2018, 7:06 PM
RobH added a comment.EditedSep 6 2018, 7:27 PM

Ok, I'm going to outline all the troubleshooting steps below that I've done to demonstrate that the issue is inherently one with the vlan ACL rules, and not likely a kernel issue.

The testbed for this testing was cloudvirt1024.

We'll start with the 'working default' for cloudvirts.

cloudvirt1024.eqiad.wmnet is in the labs-hosts1-b-eqiad vlan/subnet with the IP address of 10.64.20.43.

I'll paste the network port info with each test to show the inheritence of the vlan being tested:

Ok, I'm going to outline all the troubleshooting steps below that I've done to demonstrate that the issue is inherently one with the vlan ACL rules, and not likely a kernel issue.

The testbed for this testing was cloudvirt1024.

We'll start with the 'working default' for cloudvirts.

cloudvirt1024.eqiad.wmnet is in the labs-hosts1-b-eqiad vlan/subnet with the IP address of 10.64.20.43. loading stretch over tftp successfully

robh@asw2-b-eqiad> show interfaces descriptions | grep cloudvirt1024 
ge-8/0/22       up    up   cloudvirt1024 eth0
ge-8/0/24       up    up   cloudvirt1024 eth1

{master:2}
robh@asw2-b-eqiad> edit 
Entering configuration mode

{master:2}[edit]
robh@asw2-b-eqiad# show interfaces ge-8/0/22 | display inheritance 
description "cloudvirt1024 eth0";
##
## '9192' was expanded from interface-range 'vlan-cloud-hosts1-b-eqiad'
##
mtu 9192;
##
## '0' was expanded from interface-range 'vlan-cloud-hosts1-b-eqiad'
##
unit 0 {
    ##
    ## 'ethernet-switching' was expanded from interface-range 'vlan-cloud-hosts1-b-eqiad'
    ##
    family ethernet-switching {
        ##
        ## 'access' was expanded from interface-range 'vlan-cloud-hosts1-b-eqiad'
        ##
        interface-mode access;
        ##
        ## 'vlan' was expanded from interface-range 'vlan-cloud-hosts1-b-eqiad'
        ##
        vlan {
            ##
            ## 'cloud-hosts1-b-eqiad' was expanded from interface-range 'vlan-cloud-hosts1-b-eqiad'
            ##
            members cloud-hosts1-b-eqiad;
        }
    }
}

The dhcp lease file has the PXE load specifically overridden from the default to use TFTP only (not HTTP), and the result is the file is served from the install server, and received and loaded by cloudvirt1024 (I watched it load the installer up and show installer on screen.)

Excerpt of the dhcp file:

host cloudvirt1024 {
    hardware ethernet D0:94:66:61:15:75;
    fixed-address cloudvirt1024.eqiad.wmnet;
    option pxelinux.pathprefix "stretch-installer/";
    filename "stretch-installer/debian-installer/amd64/pxelinux.0";
}

Output of the dhcp logs:

Sep  6 19:24:48 install1002 dhcpd: DHCPDISCOVER from d0:94:66:61:15:75 via 10.64.20.3
Sep  6 19:24:48 install1002 dhcpd: DHCPOFFER on 10.64.20.43 to d0:94:66:61:15:75 via 10.64.20.3
Sep  6 19:24:48 install1002 dhcpd: DHCPDISCOVER from d0:94:66:61:15:75 via 10.64.20.2
Sep  6 19:24:48 install1002 dhcpd: DHCPOFFER on 10.64.20.43 to d0:94:66:61:15:75 via 10.64.20.2
Sep  6 19:24:52 install1002 dhcpd: DHCPREQUEST for 10.64.20.43 (208.80.154.22) from d0:94:66:61:15:75 via 10.64.20.2
Sep  6 19:24:52 install1002 dhcpd: DHCPACK on 10.64.20.43 to d0:94:66:61:15:75 via 10.64.20.2
Sep  6 19:24:52 install1002 dhcpd: DHCPREQUEST for 10.64.20.43 (208.80.154.22) from d0:94:66:61:15:75 via 10.64.20.3
Sep  6 19:24:52 install1002 dhcpd: DHCPACK on 10.64.20.43 to d0:94:66:61:15:75 via 10.64.20.3
Sep  6 19:24:52 install1002 atftpd[507]: Serving stretch-installer/debian-installer/amd64/pxelinux.0 to 10.64.20.43:2070
Sep  6 19:24:52 install1002 atftpd[507]: Serving stretch-installer/debian-installer/amd64/pxelinux.0 to 10.64.20.43:2071
Sep  6 19:24:52 install1002 atftpd[507]: Serving stretch-installer/ldlinux.c32 to 10.64.20.43:49152
Sep  6 19:24:52 install1002 atftpd[507]: Serving stretch-installer/pxelinux.cfg/ttyS1-115200 to 10.64.20.43:49153
Sep  6 19:24:52 install1002 atftpd[507]: Serving stretch-installer/pxelinux.cfg/boot.txt to 10.64.20.43:49154
Sep  6 19:24:55 install1002 atftpd[507]: Serving stretch-installer/debian-installer/amd64/linux to 10.64.20.43:49155
Sep  6 19:24:56 install1002 atftpd[507]: Serving stretch-installer/debian-installer/amd64/initrd.gz to 10.64.20.43:49156

That shows it hitting the dhcp and then gets and loads installer, and then the followup request is the installer hitting and asking during the installer itself.

And output of the serial console:

boot:
Loading debian-installer/amd64/linux... ok
Loading debian-installer/amd64/initrd.gz...

The installer then fully loads successfully in the serial console, and halts at the H740P controller detection. This shows the installer loading fine in labs1-b-vlan just fine.

Next comment will outline the steps taken to load the installer over http in the labs1-b-vlan.

RobH added a comment.Sep 6 2018, 7:39 PM

cloudvirt1024.eqiad.wmnet is in the labs-hosts1-b-eqiad vlan/subnet with the IP address of 10.64.20.43. loading stretch over http fails

The only change from the test prior is the swap from loading the PXE image over TFTP to over HTTP. That change is done by removing the TFTP overrides in the dhcp file:

robh@asw2-b-eqiad> show interfaces descriptions | grep cloudvirt1024 
ge-8/0/22       up    up   cloudvirt1024 eth0
ge-8/0/24       up    up   cloudvirt1024 eth1

{master:2}
robh@asw2-b-eqiad> edit 
Entering configuration mode

{master:2}[edit]
robh@asw2-b-eqiad# show interfaces ge-8/0/22 | display inheritance 
description "cloudvirt1024 eth0";
##
## '9192' was expanded from interface-range 'vlan-cloud-hosts1-b-eqiad'
##
mtu 9192;
##
## '0' was expanded from interface-range 'vlan-cloud-hosts1-b-eqiad'
##
unit 0 {
    ##
    ## 'ethernet-switching' was expanded from interface-range 'vlan-cloud-hosts1-b-eqiad'
    ##
    family ethernet-switching {
        ##
        ## 'access' was expanded from interface-range 'vlan-cloud-hosts1-b-eqiad'
        ##
        interface-mode access;
        ##
        ## 'vlan' was expanded from interface-range 'vlan-cloud-hosts1-b-eqiad'
        ##
        vlan {
            ##
            ## 'cloud-hosts1-b-eqiad' was expanded from interface-range 'vlan-cloud-hosts1-b-eqiad'
            ##
            members cloud-hosts1-b-eqiad;
        }
    }
}

The PXE file has been modified to remove the tftp pxe override:

Excerpt of the dhcp file:

host cloudvirt1024 {
    hardware ethernet D0:94:66:61:15:75;
    fixed-address cloudvirt1024.eqiad.wmnet;
}

Then the defaults for how to load the kernel over PXE from the dhcpd.conf file:

option root-path "/tftpboot/";
option pxelinux.pathprefix "http://apt.wikimedia.org/tftpboot/stretch-installer/";
filename "lpxelinux.0";

That changes the stretch load over to http rather than tftp. When it is changed to that, we get a very similar output on the install host logs, but totally different result on the actual host serial redirection:

Output of the dhcp logs:

Sep  6 19:36:36 install1002 dhcpd: DHCPDISCOVER from d0:94:66:61:15:75 via 10.64.20.2
Sep  6 19:36:36 install1002 dhcpd: DHCPOFFER on 10.64.20.43 to d0:94:66:61:15:75 via 10.64.20.2
Sep  6 19:36:36 install1002 dhcpd: DHCPDISCOVER from d0:94:66:61:15:75 via 10.64.20.3
Sep  6 19:36:36 install1002 dhcpd: DHCPOFFER on 10.64.20.43 to d0:94:66:61:15:75 via 10.64.20.3
Sep  6 19:36:40 install1002 dhcpd: DHCPREQUEST for 10.64.20.43 (208.80.154.22) from d0:94:66:61:15:75 via 10.64.20.3
Sep  6 19:36:40 install1002 dhcpd: DHCPACK on 10.64.20.43 to d0:94:66:61:15:75 via 10.64.20.3
Sep  6 19:36:40 install1002 dhcpd: DHCPREQUEST for 10.64.20.43 (208.80.154.22) from d0:94:66:61:15:75 via 10.64.20.2
Sep  6 19:36:40 install1002 dhcpd: DHCPACK on 10.64.20.43 to d0:94:66:61:15:75 via 10.64.20.2
Sep  6 19:36:40 install1002 atftpd[507]: Serving lpxelinux.0 to 10.64.20.43:2070
Sep  6 19:36:40 install1002 atftpd[507]: Serving lpxelinux.0 to 10.64.20.43:2071

Please note this differs, as after trying to serve lpxelinux.0, it never moves on to the next step of loading:

Sep  6 19:24:52 install1002 atftpd[507]: Serving stretch-installer/ldlinux.c32 to 10.64.20.43:49152

Output of the serial console shows it hitting dhcp, but it never moves onto the load: section that the other attempt did. It never receives the ldlinux.c32 file.

The ONLY difference between the two attempts above was changing the load from tftp to http.

The next test will move the same host to the private1-b-eqiad vlan to show http works for this host in that vlan.

gerritbot added a comment.Sep 6 2018, 7:50 PM

Change 458563 merged by RobH:
[operations/dns@master] adding cloudvirt1024b dns entry

https://gerrit.wikimedia.org/r/458563

RobH added a comment.Sep 6 2018, 8:00 PM

cloudvirt1024b.eqiad.wmnet is in the private1-b-eqiad vlan/subnet with the IP address of 10.64.16.27. loading stretch over http is successful

So on this test, we're changing the vlan of the network port over, and changing the dhcp file to change the hostname to the b version. This just eliminates the need to push a dns change and then invalidate the old dns.

The updated inheritance on the switch to show its in the private1-b-vlan:

robh@asw2-b-eqiad# show interfaces ge-8/0/22 | display inheritance    
description "cloudvirt1024 eth0";
##
## '9192' was expanded from interface-range 'vlan-private1-b-eqiad'
##
mtu 9192;
##
## '0' was expanded from interface-range 'vlan-private1-b-eqiad'
##
unit 0 {
    ##
    ## 'ethernet-switching' was expanded from interface-range 'vlan-private1-b-eqiad'
    ##
    family ethernet-switching {
        ##
        ## 'access' was expanded from interface-range 'vlan-private1-b-eqiad'
        ##
        interface-mode access;
        ##
        ## 'vlan' was expanded from interface-range 'vlan-private1-b-eqiad'
        ##
        vlan {
            ##
            ## 'private1-b-eqiad' was expanded from interface-range 'vlan-private1-b-eqiad'
            ##
            members private1-b-eqiad;
        }
    }
}

The only change to PXE was to set the dns to cloudvirt1024b rather than cloudvirt1024. It still loads over http using the dhcp server defaults:

host cloudvirt1024 {
    hardware ethernet D0:94:66:61:15:75;
    fixed-address cloudvirt1024b.eqiad.wmnet;
}

Then the defaults for how to load the kernel over PXE from the dhcpd.conf file:

option root-path "/tftpboot/";
option pxelinux.pathprefix "http://apt.wikimedia.org/tftpboot/stretch-installer/";
filename "lpxelinux.0";

Output of the dhcp logs:

Sep  6 19:56:26 install1002 dhcpd: DHCPDISCOVER from d0:94:66:61:15:75 via 10.64.16.2
Sep  6 19:56:26 install1002 dhcpd: DHCPOFFER on 10.64.16.27 to d0:94:66:61:15:75 via 10.64.16.2
Sep  6 19:56:26 install1002 dhcpd: DHCPDISCOVER from d0:94:66:61:15:75 via 10.64.16.3
Sep  6 19:56:26 install1002 dhcpd: DHCPOFFER on 10.64.16.27 to d0:94:66:61:15:75 via 10.64.16.3
Sep  6 19:56:30 install1002 dhcpd: DHCPREQUEST for 10.64.16.27 (208.80.154.22) from d0:94:66:61:15:75 via 10.64.16.3
Sep  6 19:56:30 install1002 dhcpd: DHCPACK on 10.64.16.27 to d0:94:66:61:15:75 via 10.64.16.3
Sep  6 19:56:30 install1002 dhcpd: DHCPREQUEST for 10.64.16.27 (208.80.154.22) from d0:94:66:61:15:75 via 10.64.16.2
Sep  6 19:56:30 install1002 dhcpd: DHCPACK on 10.64.16.27 to d0:94:66:61:15:75 via 10.64.16.2
Sep  6 19:56:30 install1002 atftpd[507]: Serving lpxelinux.0 to 10.64.16.27:2070
Sep  6 19:56:30 install1002 atftpd[507]: Serving lpxelinux.0 to 10.64.16.27:2071

Please note the http load differs, as after trying to serve lpxelinux.0, it never moves on to the next step of loading:

Sep 6 19:24:52 install1002 atftpd[507]: Serving stretch-installer/ldlinux.c32 to 10.64.20.43:49152

However, it does load the isntaller on the host system.

Output of the serial console:

boot:
Loading debian-installer/amd64/linux... ok
Loading debian-installer/amd64/initrd.gz...

So I think this demonstrates it is not an issue with anything to do with the hardware involved. It seems to be an oddity of our cloud-hosts1-b-eqiad and how it seems to not allow http loading over from the install server, where private1-b-eqiad does.

Stashbot added a subscriber: Stashbot.Sep 7 2018, 4:42 PM

Mentioned in SAL (#wikimedia-operations) [2018-09-07T16:42:26Z] <XioNoX> explicitely permit install1002/2002:80 in filter labs-in4 on cr1/2-eqiad - T190424

ayounsi closed this task as Resolved.Sep 7 2018, 4:45 PM

Thanks, this has been useful, especially running a packet capture on the working vs. non working flows.

so default TTL for IP in Linux is 64
There is an old/weird firewall filter term denying all packets with a TTL != 64 out of that vlan
PXE boot sends that http query with a TTL of 254
so that's why it was working fine with the curl tests from hosts in the same vlan.

I added a term to permit that flow explicitly. Should be working fine now. Let me know if not.

ayounsi claimed this task.Sep 7 2018, 4:45 PM
Stashbot mentioned this in T221532: rack/setup/install db2[103-120].codfw.wmnet (18 hosts).May 7 2019, 12:53 AM

Mentioned in SAL (#wikimedia-operations) [2019-05-07T00:53:16Z] <mutante> install2002 - disabling puppet, live hacking DHCP config for db2103 to not serve installer via http to debug install issue for T221532 which seems like T190424#4548003

faidon mentioned this in rOHPU5e730593e20c: Modify labs-hosts1-vlans for the installer.Oct 3 2019, 11:11 AM
Maintenance_bot removed a project: Patch-For-Review.Oct 3 2019, 12:10 PM
RobH removed a subscriber: RobH.Oct 3 2019, 3:55 PM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL