Change Details

The new cage in codfw cannot support direct layer-2 adjacency to our LVS servers, meaning we can only support services behind LVS there which use IPIP encapsulation. **New codfw cage** The new switches for {T380240} were ordered without the VXLAN license for Juniper, and the cage will be a hybrid of Juniper/Nokia (due to needing the racks online sooner than we would be able to support a full Nokia setup). This rules our using VXLAN for layer-2 extension in the new cage. That decision was made after discussions at the SRE Summit which suggested that by the time the racks went live we would no longer have any services dependent on the layer-2 connectivity from LVS load-balancers. But I think we need to assess exactly where we are with that. **Current Status** I am aware that services deployed in Kubernetes PODs are not yet able to use IPIP. I believe that is being investigated under T352956, though I am not sure what stage we are at. Until it is ready we will avoid placing K8s worker nodes in the new racks. There are some non-K8s services not using IPIP too, which are detailed in T373020. It looks like most of them are managed by the search team? Until we get everything migrated I think we need to build a list of hostname-types that we can't rack in the new cage to avoid any problems. @Vgutierrez you've some neat scripts in that task, is there an easy way to turn them into a list of hostnames for that list we can give to DC-ops?

The new cage in codfw cannot support direct layer-2 adjacency to our LVS servers, meaning we can only support services behind LVS there which use IPIP encapsulation. (NOTE) I've edited this description based on feedback to list out the hosts we should avoid putting there for now. **Background** The new switches for {T380240} were ordered without the VXLAN license for Juniper, and the cage will be a hybrid of Juniper/Nokia (due to needing the racks online sooner than we would be able to support a full Nokia setup). This rules our using VXLAN for layer-2 extension in the new cage. That decision was made after discussions at the SRE Summit which suggested that by the time the racks went live we would no longer have any services dependent on the layer-2 connectivity from LVS load-balancers. But I think we need to assess exactly where we are with that. **Hostname List** The main hosts we need to avoid racking in the new cage for now are the Kubernetes hosts (work to move them to IPIP is tracked in T352956). Additionally the cirrussearch hosts should be avoided (see T373020). The full list of hostname prefixes we cannot rack in the new cage right away are: ``` aux-k8s-ctrl aux-k8s-worker cirrussearch dse-k8s-ctrl dse-k8s-worker elastic kubestage kubestagemaster ml-serve ml-serve-ctrl ml-staging ml-staging-ctrl wikikube-ctrl wikikube-worker ```

The new cage in codfw cannot support direct layer-2 adjacency to our LVS servers, meaning we can only support services behind LVS there which use IPIP encapsulation. **New codfw cage**(NOTE) I've edited this description based on feedback to list out the hosts we should avoid putting there for now. **Background** The new switches for {T380240} were ordered without the VXLAN license for Juniper, and the cage will be a hybrid of Juniper/Nokia (due to needing the racks online sooner than we would be able to support a full Nokia setup). This rules our using VXLAN for layer-2 extension in the new cage. That decision was made after discussions at the SRE Summit which suggested that by the time the racks went live we would no longer have any services dependent on the layer-2 connectivity from LVS load-balancers. But I think we need to assess exactly where we are with that. **Current Status** I am aware that services deployed in Kubernetes PODs are not yet able to use IPIP. I believe that is being investigated under T352956, though I am not sure what stage we are at. Until it is ready we will avoid placing K8s worker nodes in the new racks.Hostname List** There are some non-K8s services not using IPIP too, which are detailed in T373020. It looks like most of them are managed by the search team? main hosts we need to avoid racking in the new cage for now are the Kubernetes hosts (work to move them to IPIP is tracked in T352956). Until we get everything migrated I think we need to build a list of hostname-types that we can't rack in the new cage to avoid any problemsAdditionally the cirrussearch hosts should be avoided (see T373020). @Vgutierrez you've some neat scripts in that task, is there an easy way to turn them into a list of hostnames for that list we can give to DC-ops?The full list of hostname prefixes we cannot rack in the new cage right away are: ``` aux-k8s-ctrl aux-k8s-worker cirrussearch dse-k8s-ctrl dse-k8s-worker elastic kubestage kubestagemaster ml-serve ml-serve-ctrl ml-staging ml-staging-ctrl wikikube-ctrl wikikube-worker ```