Page MenuHomePhabricator
Create Task
Maniphest T352883

Test IP-renumbering on kubestage2002.codfw.wmnet
Closed, ResolvedPublic
Actions

Description

In order to test the IP renumbering process on kubernetes nodes, we will test it on a physical host in the codfw staging cluster, kubestage2002.codfw.wmnet

The steps are :

  • Drain the node (@Clement_Goubert)
  • Recable the node to the new switch (@Papaul)
  • Update netbox, and push config to set up switch vlans and bgp (@cmooney)
  • Manually change the IP on the host (@cmooney)
  • Identify the additional steps needed to adjust BGP and other elements to make it work (@Clement_Goubert)

The nodes are not behind LVS, so the tests should be able to proceed while the LVS migration to the new VLAN is in progress.

Related Objects
Search...

Event Timeline

Clement_Goubert renamed this task from Test IP-renumbering on a kubestage2002.codfw.wmnet to Test IP-renumbering on kubestage2002.codfw.wmnet.Dec 6 2023, 4:37 PM
Clement_Goubert created this task.
Clement_Goubert updated the task description. (Show Details)
Clement_Goubert claimed this task.Dec 6 2023, 4:40 PM
Clement_Goubert triaged this task as Medium priority.
Clement_Goubert added a project: Prod-Kubernetes.Dec 6 2023, 5:02 PM
Stashbot added a comment.Jan 8 2024, 3:35 PM

Mentioned in SAL (#wikimedia-operations) [2024-01-08T15:35:27Z] <claime> Draining and cordoning kubestage2002.codfw.wmnet - T352883

Clement_Goubert changed the task status from Open to In Progress.Jan 8 2024, 3:37 PM
Clement_Goubert reassigned this task from Clement_Goubert to Papaul.

Host is now drained and cordoned. It is in codfw rack B8 and it can be recabled to the new switch. All yours @Papaul

Clement_Goubert moved this task from Backlog to In progress on the Prod-Kubernetes board.Jan 8 2024, 3:37 PM
Clement_Goubert moved this task from Incoming 🐫 to Doing 😎 on the serviceops board.
Papaul added a comment.Jan 8 2024, 4:23 PM

@Clement_Goubert thanks will work on it in a minute

cmooney added a comment.Jan 8 2024, 4:25 PM

@Papaul let me know what port is used on lsw1-b8-codfw once done and I will make the Netbox changes and assign new IPs for it etc.

Papaul added a comment.Jan 8 2024, 5:57 PM

@cmooney xe-0/0/26

Papaul updated the task description. (Show Details)Jan 8 2024, 5:58 PM
ops-monitoring-bot added a comment.Jan 9 2024, 11:37 AM

Icinga downtime and Alertmanager silence (ID=c70b0979-84e8-4fe7-8682-45d50615a587) set by cmooney@cumin1002 for 1:00:00 on 2 host(s) and their services with reason: Adding vlan to switch, precaution in case it triggers EVPN L3 bug.

lsw1-b8-codfw,lsw1-b8-codfw IPv6
cmooney added a comment.Jan 9 2024, 1:15 PM

Ok I have made the Netbox changes and pushed the resulting config to lsw1-b8-codfw now, and the port it up (note the port is still 1G on ge-0/0/26).

I've also updated the host's IP addresses manually to the newly assigned ones, 10.192.22.137 and 2620:0:860:112:10:192:22:137. DNS has also been updated. The host should probably be rebooted to ensure the modified /etc/network/interfaces file works ok on a boot. It's reachable via SSH again now.

I also found the old IPv4 address had a manual entry in /etc/hosts, and the old subnet was referenced in /etc/networks, so I updated those too. For record the old IPs were 10.192.16.137 and 2620:0:860:102:10:192:16:137.

@Clement_Goubert ping me on irc if I can assist more. The major change I'm aware of is to the BGP peering. The host should now only have a single BGP peering in each address fam, to the top-of-rack switch lsw1-b8-codfw (10.192.22.1 / 2620:0:860:112::1). BGP should establish once the host is set up for that, I will need to check any announced addresses are correctly propagated from the leaf through the network as this is the first one like this we have.

cmooney updated the task description. (Show Details)Jan 9 2024, 1:32 PM
akosiaris added a comment.Jan 9 2024, 2:11 PM
In T352883#9445709, @cmooney wrote:

Ok I have made the Netbox changes and pushed the resulting config to lsw1-b8-codfw now, and the port it up (note the port is still 1G on ge-0/0/26).

I've also updated the host's IP addresses manually to the newly assigned ones, 10.192.22.137 and 2620:0:860:112:10:192:22:137. DNS has also been updated. The host should probably be rebooted to ensure the modified /etc/network/interfaces file works ok on a boot. It's reachable via SSH again now.

Thanks!

I also found the old IPv4 address had a manual entry in /etc/hosts, and the old subnet was referenced in /etc/networks, so I updated those too. For record the old IPs were 10.192.16.137 and 2620:0:860:102:10:192:16:137.

Note that none of the 2 are manual, they are the result of a standard linux install. Thanks for catching those.

@Clement_Goubert ping me on irc if I can assist more. The major change I'm aware of is to the BGP peering. The host should now only have a single BGP peering in each address fam, to the top-of-rack switch lsw1-b8-codfw (10.192.22.1 / 2620:0:860:112::1). BGP should establish once the host is set up for that, I will need to check any announced addresses are correctly propagated from the leaf through the network as this is the first one like this we have.

This is the thing we need to get fixed, I see

$ sudo calicoctl node status
Calico process is running.

IPv4 BGP status
+----------------+---------------+-------+----------+---------+
|  PEER ADDRESS  |   PEER TYPE   | STATE |  SINCE   |  INFO   |
+----------------+---------------+-------+----------+---------+
| 208.80.153.192 | node specific | start | 12:53:09 | Connect |
| 208.80.153.193 | node specific | start | 12:53:09 | Connect |
+----------------+---------------+-------+----------+---------+

IPv6 BGP status
+--------------------+---------------+-------+----------+--------------------------------+
|    PEER ADDRESS    |   PEER TYPE   | STATE |  SINCE   |              INFO              |
+--------------------+---------------+-------+----------+--------------------------------+
| 2620:0:860:ffff::1 | node specific | start | 13:04:07 | Idle Socket: Connection closed |
| 2620:0:860:ffff::2 | node specific | start | 13:04:07 | Idle Socket: Connection closed |
+--------------------+---------------+-------+----------+--------------------------------+

so, it's still trying to talk to the core routers.

ayounsi added a comment.Jan 9 2024, 2:14 PM

This is the thing we need to get fixed, I see

Yep, that's T352893: Update puppet's topology.kubernetes.io/zone logic to take into account the new setup and its 2 CRs.

Clement_Goubert added a comment.Jan 9 2024, 2:34 PM
# kubectl describe nodes kubestage2002.codfw.wmnet | grep -A3 Addresses
Addresses:
  InternalIP:  10.192.22.137
  InternalIP:  2620:0:860:112:10:192:22:137
  Hostname:    kubestage2002.codfw.wmnet

On the kubernetes side, new addresses have propagated correctly.

akosiaris added a comment.Jan 9 2024, 2:34 PM
In T352883#9445907, @ayounsi wrote:

This is the thing we need to get fixed, I see

Yep, that's T352893: Update puppet's topology.kubernetes.io/zone logic to take into account the new setup and its 2 CRs.

Ah indeed. I 've replied in https://gerrit.wikimedia.org/r/c/operations/puppet/+/980927/.

cmooney edited parent tasks, added: T354869: Re-IP codfw private baremetal hosts to new per-rack vlans/subnets; removed: T327938: Codfw row A/B top-of-rack switch refresh.Jan 11 2024, 2:38 PM
Stashbot added a comment.Jan 18 2024, 3:52 PM

Mentioned in SAL (#wikimedia-operations) [2024-01-18T15:52:07Z] <claime> stopping puppet on P:kubernetes::node to deploy 980927 - T352883

Stashbot added a comment.Jan 18 2024, 3:52 PM

Mentioned in SAL (#wikimedia-operations) [2024-01-18T15:52:12Z] <claime> Running puppet on kubestage2002 - T352883

Clement_Goubert added a comment.Jan 18 2024, 3:53 PM
cgoubert@kubestage2002:~$ sudo calicoctl node status
Calico process is running.

IPv4 BGP status
+----------------+---------------+-------+------------+---------+
|  PEER ADDRESS  |   PEER TYPE   | STATE |   SINCE    |  INFO   |
+----------------+---------------+-------+------------+---------+
| 208.80.153.192 | node specific | start | 2024-01-09 | Connect |
| 208.80.153.193 | node specific | start | 2024-01-09 | Connect |
+----------------+---------------+-------+------------+---------+

IPv6 BGP status
+--------------------+---------------+-------+------------+--------------------------------+
|    PEER ADDRESS    |   PEER TYPE   | STATE |   SINCE    |              INFO              |
+--------------------+---------------+-------+------------+--------------------------------+
| 2620:0:860:ffff::1 | node specific | start | 2024-01-09 | Active Socket: Connection      |
|                    |               |       |            | closed                         |
| 2620:0:860:ffff::2 | node specific | start | 2024-01-09 | Active Socket: Connection      |
|                    |               |       |            | closed                         |
+--------------------+---------------+-------+------------+--------------------------------+

cgoubert@kubestage2002:~$ sudo facter -p lldp.parent
lsw1-b8-codfw.mgmt.codfw.wmnet
cgoubert@kubestage2002:~$ sudo run-puppet-agent -e "deploy 980927 - T352893"
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for kubestage2002.codfw.wmnet
Info: Applying configuration version '(d46bf19759) Clément Goubert - k8s topology labels: add row to rack transition'
[...]
Notice: /Stage[main]/K8s::Kubelet/File[/etc/default/kubelet]/content: 
--- /etc/default/kubelet        2024-01-09 14:28:59.928924163 +0000
+++ /tmp/puppet-file20240118-1085678-1qe5zh     2024-01-18 15:49:43.363653307 +0000
@@ -8,7 +8,7 @@
  --kubeconfig=/etc/kubernetes/kubelet.conf \
  --network-plugin=cni \
  --node-ip=10.192.22.137,2620:0:860:112:10:192:22:137 \
- --node-labels=node.kubernetes.io/disk-type=ssd,topology.kubernetes.io/region=codfw,topology.kubernetes.io/zone=row-b \
+ --node-labels=node.kubernetes.io/disk-type=ssd,topology.kubernetes.io/region=codfw,topology.kubernetes.io/zone=row-b8 \
  --pod-infra-container-image=docker-registry.discovery.wmnet/pause:3.6-1 \
  --register-schedulable=false \
  --system-reserved=cpu=4.1,memory=7.21Gi \

Notice: /Stage[main]/K8s::Kubelet/File[/etc/default/kubelet]/content: content changed '{sha256}f6a8bb8eff06e336fd86f37e26ec05fc927dcb3278f3be015fca162cd12de610' to '{sha256}bce3f41ffd6ce0d32e522e48723f70951de6c94b766b278139ba1f1adfcf9dca'
Info: /Stage[main]/K8s::Kubelet/File[/etc/default/kubelet]: Scheduling refresh of Service[kubelet]
Notice: /Stage[main]/K8s::Kubelet/Service[kubelet]: Triggered 'refresh' from 1 event
Notice: Applied catalog in 14.60 seconds

cgoubert@kubestage2002:~$ sudo calicoctl node status
Calico process is running.

IPv4 BGP status
+--------------+---------------+-------+----------+-------------+
| PEER ADDRESS |   PEER TYPE   | STATE |  SINCE   |    INFO     |
+--------------+---------------+-------+----------+-------------+
| 10.192.22.1  | node specific | up    | 15:49:55 | Established |
+--------------+---------------+-------+----------+-------------+

IPv6 BGP status
+-------------------+---------------+-------+----------+--------------------------------+
|   PEER ADDRESS    |   PEER TYPE   | STATE |  SINCE   |              INFO              |
+-------------------+---------------+-------+----------+--------------------------------+
| 2620:0:860:112::1 | node specific | start | 15:49:54 | Active Socket: Connection      |
|                   |               |       |          | closed                         |
+-------------------+---------------+-------+----------+--------------------------------+

Works on this particular staging node, will now deploy to the rest of staging to ensure no-op.

Stashbot added a comment.Jan 18 2024, 3:54 PM

Mentioned in SAL (#wikimedia-operations) [2024-01-18T15:54:10Z] <claime> Running puppet on A:wikikube-staging-worker - T352883

Stashbot added a comment.Jan 18 2024, 4:03 PM

Mentioned in SAL (#wikimedia-operations) [2024-01-18T16:03:33Z] <claime> Running puppet on 'P{P:kubernetes::node} and P{F:lldp.parent ~ lsw}' - T352883

Clement_Goubert added a comment.Jan 18 2024, 4:17 PM

No-op on these nodes, proceeding with the rest.

Stashbot added a comment.Jan 18 2024, 4:18 PM

Mentioned in SAL (#wikimedia-operations) [2024-01-18T16:18:27Z] <claime> Running puppet on 'P{P:kubernetes::node} and not P{F:lldp.parent ~ lsw}' - T352883

Clement_Goubert added a comment.Jan 18 2024, 4:19 PM
In T352883#9469622, @Clement_Goubert wrote:
IPv6 BGP status
+-------------------+---------------+-------+----------+--------------------------------+
|   PEER ADDRESS    |   PEER TYPE   | STATE |  SINCE   |              INFO              |
+-------------------+---------------+-------+----------+--------------------------------+
| 2620:0:860:112::1 | node specific | start | 15:49:54 | Active Socket: Connection      |
|                   |               |       |          | closed                         |
+-------------------+---------------+-------+----------+--------------------------------+

Works on this particular staging node, will now deploy to the rest of staging to ensure no-op.

Actually didn't work for ipv6, I had to delete the calico pod and let it recreate for the session to establish.

Clement_Goubert added a comment.Jan 18 2024, 4:41 PM

No-op on the rest of the infra.

Clement_Goubert mentioned this in T352893: Update puppet's topology.kubernetes.io/zone logic to take into account the new setup.Jan 18 2024, 4:42 PM
Clement_Goubert closed this task as Resolved.Jan 22 2024, 10:33 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits