Page MenuHomePhabricator
Create Task
Maniphest T217346

relocate/reimage cloudvirt1012 with 10G interfaces
Closed, ResolvedPublic
Actions

Description

This host is currently empty, so this is a good time to move it.

cloudvirt1012:

  • - put system offline in all checks for maint window
  • - relocate to 10G rack and update netbox
  • - update switch configuration for new primary 10G
  • - enable PXE for 10G primary interface.
  • - attach/cable secondary 10G port for instance traffic, update switch config.
  • - remove old switch config for 1G ports
  • - @Cmjohnson to apply HP SPP as @RobH was unable to update ilom/bios remotely.
  • - PXE boot and reimage system
  • - reintroduce system into service cluster

Related Objects
Search...

Event Timeline

Andrew created this task.Feb 28 2019, 5:09 PM
Restricted Application edited projects, added SRE; removed Patch-For-Review. · View Herald TranscriptFeb 28 2019, 5:09 PM
Andrew mentioned this in T216195: Move cloudvirt hosts to 10Gb ethernet.Feb 28 2019, 5:09 PM
Andrew added a project: DC-Ops.Feb 28 2019, 5:17 PM

Steps:

  • Move host to a rack with 10G -- B2, B4 or B7 I believe
  • Enable the 10G nic in the bios
  • Move/install cables
  • Update switch config
  • Re-image
jbond triaged this task as Medium priority.Mar 4 2019, 7:48 PM
Andrew added a parent task: T216195: Move cloudvirt hosts to 10Gb ethernet.Apr 4 2019, 5:41 PM
RobH assigned this task to • Cmjohnson.Apr 4 2019, 6:27 PM
RobH updated the task description. (Show Details)
RobH renamed this task from Move cloudvirt1012 to a 10G rack and connect 10g nics to relocate/reimage cloudvirt1012 with 10G interfaces.Apr 4 2019, 6:31 PM
RobH updated the task description. (Show Details)Apr 4 2019, 6:35 PM
RobH subscribed.
RobH added a comment.Apr 4 2019, 6:55 PM

Ok, I've attempted to update the firmware of the ilom multiple times, all to no avail.

The method I've attempted to use for cloudvirt1012 worked just fine in in cloudvirt1015. Both of these are HP DL360 Gen9 hosts, and should update identically.

On cloudvirt1015 it just updated fine, continuing to where it would reboot the ilom interface and require relogin. Then I would update the bios file the same way, reboot system, and it would come back online in less than 8 minutes.

I'm not sure what is up here, Chris will have to run the HP SPP.

• Cmjohnson reassigned this task from • Cmjohnson to RobH.Apr 5 2019, 4:04 PM
• Cmjohnson updated the task description. (Show Details)
• Cmjohnson subscribed.

@RobH connected the second cable, updated switch cfg with cloud-virt-instance-trunk. Ran the SPP

Andrew claimed this task.Apr 5 2019, 7:53 PM
Andrew added a comment.Apr 5 2019, 9:18 PM

I reimaged and built a canary VM -- the hosted VM cannot access any external networks. I haven't investigated this more deeply yet, but the first thing would be to confirm that that port is configured the same as the second port on cloudvirt1015 (since that one works as expected).

Andrew updated the task description. (Show Details)Apr 5 2019, 9:18 PM
RobH reassigned this task from Andrew to • Cmjohnson.Apr 5 2019, 10:53 PM

So, per @Andrew's request I've investigated the switch stack software for the secondary 'instance' connections for both cloudvirt1009 via T216324 and cloudvirt1012 via T217346.

This update applies to both, since both have the exact same symptoms and settings, as far as I can tell from the software/remote side.

@Andrew complained that the instance interface (the second 10G interface) on these two hosts (cloudvirt1009 and cloudvirt1012) wasn't coming up, while identically setup cloudvirt1015 worked fine.

I went ahead and logged into the switch stack, and will paste the output below. The summary is this, they all appear to have the exact same setup. They are in the exact same vlans/groups and all should work.

Output of switch commands to check they are all identically setup:

18 $> ssh asw2-b-eqiad.mgmt.eqiad.wmnet
--- JUNOS 14.1X53-D46.7 built 2017-11-23 22:06:48 UTC
{master:2}
robh@asw2-b-eqiad> show interfaces descriptions | grep cloudvirt1015 
xe-2/0/20       up    up   cloudvirt1015-eth0
xe-2/0/21       up    up   cloudvirt1015-eth1

{master:2}
robh@asw2-b-eqiad> show interfaces descriptions | grep cloudvirt1009    
xe-2/0/15       up    up   cloudvirt1009
xe-8/0/27                  cloudvirt1009 eth1

{master:2}
robh@asw2-b-eqiad> show interfaces descriptions | grep cloudvirt1012    
xe-2/0/16       up    up   cloudvirt1012
xe-8/0/28                  cloudvirt1012 eth1

{master:2}
robh@asw2-b-eqiad> edit   
Entering configuration mode
The configuration has been changed but not committed

{master:2}[edit]
robh@asw2-b-eqiad# show interfaces xe-2/0/21 | display inheritance 
description cloudvirt1015-eth1;
##
## '9192' was expanded from interface-range 'cloud-virt-instance-trunk'
##
mtu 9192;
##
## '0' was expanded from interface-range 'cloud-virt-instance-trunk'
##
unit 0 {
    ##
    ## 'ethernet-switching' was expanded from interface-range 'cloud-virt-instance-trunk'
    ##
    family ethernet-switching {
        ##
        ## 'trunk' was expanded from interface-range 'cloud-virt-instance-trunk'
        ##
        interface-mode trunk;
        ##
        ## 'vlan' was expanded from interface-range 'cloud-virt-instance-trunk'
        ##
        vlan {
            ##
            ## 'cloud-instances1-b-eqiad' was expanded from interface-range 'cloud-virt-instance-trunk'
            ## 'cloud-instances2-b-eqiad' was expanded from interface-range 'cloud-virt-instance-trunk'
            ##
            members [ cloud-instances1-b-eqiad cloud-instances2-b-eqiad ];
        }
    }
}

{master:2}[edit]
robh@asw2-b-eqiad# show interfaces xe-8/0/27 | display iner
                                                       ^
syntax error, expecting <command>.
robh@asw2-b-eqiad# show interfaces xe-8/0/27 | display inheritance 
description "cloudvirt1009 eth1";
##
## '9192' was expanded from interface-range 'cloud-virt-instance-trunk'
##
mtu 9192;
##
## '0' was expanded from interface-range 'cloud-virt-instance-trunk'
##
unit 0 {
    ##
    ## 'ethernet-switching' was expanded from interface-range 'cloud-virt-instance-trunk'
    ##
    family ethernet-switching {
        ##
        ## 'trunk' was expanded from interface-range 'cloud-virt-instance-trunk'
        ##
        interface-mode trunk;
        ##
        ## 'vlan' was expanded from interface-range 'cloud-virt-instance-trunk'
        ##
        vlan {
            ##
            ## 'cloud-instances1-b-eqiad' was expanded from interface-range 'cloud-virt-instance-trunk'
            ## 'cloud-instances2-b-eqiad' was expanded from interface-range 'cloud-virt-instance-trunk'
            ##
            members [ cloud-instances1-b-eqiad cloud-instances2-b-eqiad ];
        }
    }
}

{master:2}[edit]
robh@asw2-b-eqiad# show interfaces xe-8/0/28 | display inheritance    
description "cloudvirt1012 eth1";
##
## '9192' was expanded from interface-range 'cloud-virt-instance-trunk'
##
mtu 9192;
##
## '0' was expanded from interface-range 'cloud-virt-instance-trunk'
##
unit 0 {
    ##
    ## 'ethernet-switching' was expanded from interface-range 'cloud-virt-instance-trunk'
    ##
    family ethernet-switching {
        ##
        ## 'trunk' was expanded from interface-range 'cloud-virt-instance-trunk'
        ##
        interface-mode trunk;
        ##
        ## 'vlan' was expanded from interface-range 'cloud-virt-instance-trunk'
        ##
        vlan {
            ##
            ## 'cloud-instances1-b-eqiad' was expanded from interface-range 'cloud-virt-instance-trunk'
            ## 'cloud-instances2-b-eqiad' was expanded from interface-range 'cloud-virt-instance-trunk'
            ##
            members [ cloud-instances1-b-eqiad cloud-instances2-b-eqiad ];
        }
    }

I think this is a physical layer issue. We will need @Cmjohnson to confirm the physical cable for the following:

  • physically check and reseat the 10G DAC cable for cloudvirt1009 in xe-8/0/27.
    • you should see the link light come up, and show up in the switch software, please check this. If it doesn't come up, try another DAC cable.
  • physically check and reseat the 10G DAC cable for cloudvirt1012 in xe-8/0/28.
    • you should see the link light come up, and show up in the switch software, please check this. If it doesn't come up, try another DAC cable.
RobH mentioned this in T216324: relocate/reimage cloudvirt1009 with 10G interfaces.Apr 5 2019, 10:53 PM
• Cmjohnson added a comment.Apr 8 2019, 2:17 PM

@RobH do you mean xe-2/0/27 and 2/028 ....both have link lights and are new out of the box cables.

Andrew closed this task as Resolved.Apr 8 2019, 7:51 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL