Migrate Cloud VPS instances to VXLAN based networks
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	taavi
	May 13 2024, 10:20 AM

Description

This task is to track the migration for instances from the cloud-instances2 VLAN based network to a VXLAN based approach. The current plan is to finish T326373: Migrate Cloud VPS to Neutron Open vSwitch agent first and then work on this.

Details

Related Changes in Gerrit:

Subject	Repo	Branch	Lines +/-
WMCS: Remove cloud-instances2-b specific ranges from BGP policy	operations/homer/public	master	+0 -4
WMCS: fix typo in updated cloud-in policy #2	operations/homer/public	master	+2 -2
WMCS: fix typo in updated cloud-in policy	operations/homer/public	master	+2 -2
Cloud network: update policy to support /17 IPv4 aggregates	operations/homer/public	master	+11 -7

Customize query in gerrit

Related Objects
Search...

Status	Subtype	Assigned	Task
Open		None	T53494 Use Beta cluster as a true canary for code deployments (epic)
Declined		None	T87220 Minimize infrastructure differences between Beta Cluster and production
Open		None	T211677 Support IPv6 in beta
Open		None	T404466 Support IPv6-only VMs
Open		None	T392688 Enable IPv6 on Cloud VPS infrastructure services
Resolved		taavi	T379175 Enable IPv6 for the Cloud VPS web proxy
Open		None	T392509 Enable IPv6 for Toolforge services
Resolved		taavi	T392506 Enable IPv6 for tools.wmflabs.org / *.toolserver.org legacy redirector service
Resolved		taavi	T211575 Enable IPv6 on toolforge.org
Open		None	T220306 Add IPv6 monitoring
Open		None	T270694 CloudVPS: introduce tenant networks
Resolved		• aborrero	T364725 Migrate Cloud VPS instances to VXLAN based networks
Resolved		Andrew	T326373 Migrate Cloud VPS to Neutron Open vSwitch agent
Resolved		taavi	T358761 Deploy OVS test setup in codfw1dev
Resolved		Andrew	T364457 Migrate eqiad1 hypervisors to Neutron OVS agent
Resolved		taavi	T364458 Create new g4 flavors to support hypervisor migration from Linuxbridge to OVS Neutron agents
Resolved		taavi	T365696 Investigate how to run OpenTofu to manage Cloud VPS admin-only resources
Resolved		Andrew	T367723 Migrate WMCS managed projects to g4 flavors
Resolved	BUG REPORT	fnegri	T368725 [trove] cannot create mariadb instances
Resolved		Andrew	T367797 dwl reboot coordination request
Resolved		taavi	T364459 Migrate eqiad1 cloudnets to Neutron OVS agent
Resolved		Andrew	T368426 Migrate codfw1dev hypervisors to Neutron OVS agent
Resolved		• aborrero	T248881 CloudVPS: research VXLAN implementation for neutron
Resolved		• aborrero	T373869 Cloud VPS: design target vxlan setup
Resolved		• aborrero	T374020 openstack: instrument VXLAN-based flat network
Resolved		• aborrero	T374111 netbox: allocate CIDRs for openstack VXLAN-based flat networks
Resolved		• aborrero	T37947 Enable IPv6 on CloudVPS
Resolved		• aborrero	T245495 CloudVPS: IPv6 in codfw1dev
Resolved		• aborrero	T187929 Cloud IPv6 subnets
Resolved		• aborrero	T374712 netbox: create IPv6 entries for Cloud VPS
Resolved		• aborrero	T376462 dns: integrate PTR support for 2a02:ec80:a100::/48
Resolved		• aborrero	T374715 openstack: work out IPv6 and designate integration
Resolved		• aborrero	T377740 neutron: clarify why DNS extension is not enabled
Resolved		• aborrero	T378192 openstack: wmf sink: extend it to support IPv6
Resolved		• aborrero	T378995 openstack codfw1dev: API endpoints errors 2024-11-04
Resolved		• aborrero	T374714 openstack: clarify IPv6 firewalling
Resolved		• aborrero	T377339 horizon: enable IPv6 security group panels
Resolved		• aborrero	T374716 cloudgw: add support and enable IPv6
Resolved		cmooney	T374713 cloudsw: codfw: enable IPv6
Resolved		• aborrero	T376879 keepalived: it doesn't support mixing IPv4 and IPv6 VIPs on the same VRRP instance
Resolved		• aborrero	T375847 openstack: initial IPv6 support in neutron
Resolved		• aborrero	T377467 Decision Request - How to do the Cloud VPS VXLAN/IPv6 migration
Resolved		• aborrero	T379356 openstack: nova-fullstack: add support for IPv6
Resolved		• aborrero	T380208 openstack: codfw1dev: fullstack tests failing
Resolved		• aborrero	T380303 Openstack: many orphaned (or seemingly orphaned) VMs in codfw1dev
Resolved		• aborrero	T380054 Cloud VPS: prepare documentation on VXLAN/IPV6 migration
Resolved		• aborrero	T380081 horizon: enable the UI to select networks on VM creation panel
Resolved		• aborrero	T380174 CloudVPS: IPv6 in eqiad1
Resolved		• aborrero	T380703 versions.toolforge.org is down
Resolved		• aborrero	T380728 openstack: network problems when introducing new networks
Resolved		cmooney	T389958 WMCS Eqiad: Enable IPv6 in cloud vrf on switches
Resolved		• aborrero	T390877 HAProxyServiceUnavailable
Resolved		• aborrero	T391043 CephClusterInError #page Ceph cluster in eqiad is in error status
Resolved		• aborrero	T391040 CloudVirtDown Cloudvirt node cloudvirt1063 is down. #page
Resolved		• aborrero	T391039 CephSlowOps Ceph cluster in eqiad has 486 slow ops
Resolved		• aborrero	T391325 openstack: improve networktests for newer network setup
Resolved		fnegri	T391467 gitlab ci: validate secrets settings in pipeline for tofu integration
Resolved		cmooney	T380746 dns: add PTR support for 2a02:ec80:a000::
Resolved		taavi	T386689 Add new WMCS IPv6 ranges to MediaWiki configuration where required
Resolved		cmooney	T388379 ProbeDown
Resolved		• aborrero	T389942 openstack: rename lan-flat-cloudinstances2b to VLAN/legacy
Resolved		JAllemandou	T392468 Add new WMCS IP ranges to analytics
Resolved		taavi	T392570 metricsinfra: Support scraping v6-enabled instances
Resolved		taavi	T392611 VMs with ferm host-level firewall do not permit DHCPv6 responses
Duplicate		• aborrero	T374822 openstack: create some automation to migrate VMs from VLAN to VXLAN networks
Resolved		• aborrero	T374824 openstack: make vxlan/ipv6-dualstack network the default for new instances
Resolved		• aborrero	T374828 openstack: vxlan: verify nova proxy and floating IPs work with new VXLAN-based network
Resolved		• aborrero	T375596 openstack: verify DNS integration for new VXLAN-based network
Resolved		• aborrero	T375604 openstack: cleanup and prepare security groups
Declined		• aborrero	T377346 openstack: develop a script to migrate a VM instance from the old network setting (vlan) to the new (vxlan, IPv6)
Declined		None	T379154 openstack: vxlan: potential changes to cloudvirt MTU to enable jumbo frames

Event Timeline

taavi created this task.May 13 2024, 10:20 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 13 2024, 10:20 AM

taavi added a subtask: T326373: Migrate Cloud VPS to Neutron Open vSwitch agent.May 13 2024, 10:20 AM

taavi added a subtask: T358868: Use BGP to announce VM ranges from cloudnet to cloudgw.May 13 2024, 10:22 AM

taavi added a subtask: T248881: CloudVPS: research VXLAN implementation for neutron.May 13 2024, 10:26 AM

taavi moved this task from Unsorted to Network on the Cloud-VPS board.Jun 4 2024, 3:55 PM

fnegri changed the status of subtask T326373: Migrate Cloud VPS to Neutron Open vSwitch agent from Open to In Progress.Jun 28 2024, 12:51 PM

this task is in the list of focus areas and WMCS team goals for Q1 FY 2024/25

• aborrero moved this task from Backlog to Next on the User-aborrero board.Jul 15 2024, 3:12 PM

• aborrero added a parent task: T270694: CloudVPS: introduce tenant networks.

• aborrero added a subtask: T370037: Cloud VPS: extend tofu-infra coverage.Jul 18 2024, 3:09 PM

• joanna_borun edited projects, added cloud-services-team (FY2024/2025-Q1-Q2); removed cloud-services-team.Jul 22 2024, 11:51 AM

• aborrero moved this task from Next to Doing on the User-aborrero board.Sep 2 2024, 9:39 AM

reading: https://docs.openstack.org/neutron/2024.1/admin/deploy-ovs-ha-vrrp.html

• aborrero changed the status of subtask T373869: Cloud VPS: design target vxlan setup from Open to In Progress.Sep 3 2024, 12:34 PM

• aborrero closed subtask T373869: Cloud VPS: design target vxlan setup as Resolved.Sep 4 2024, 3:09 PM

• aborrero closed subtask T248881: CloudVPS: research VXLAN implementation for neutron as Resolved.

• aborrero changed the status of subtask T374020: openstack: instrument VXLAN-based flat network from Open to In Progress.Sep 4 2024, 3:52 PM

• aborrero added a subtask: T37947: Enable IPv6 on CloudVPS.Sep 13 2024, 12:40 PM

• aborrero mentioned this in T37947: Enable IPv6 on CloudVPS.

It has been mentioned by @cmooney that doing T37947: Enable IPv6 on CloudVPS while doing this VXLAN transition might be the perfect opportunity. I agree, and I will try to do it.

• aborrero mentioned this in T374713: cloudsw: codfw: enable IPv6.Sep 13 2024, 12:46 PM

• aborrero closed subtask T374020: openstack: instrument VXLAN-based flat network as Resolved.Sep 16 2024, 9:22 AM

• aborrero removed a subtask: T370037: Cloud VPS: extend tofu-infra coverage.

• aborrero changed the status of subtask T374824: openstack: make vxlan/ipv6-dualstack network the default for new instances from Open to In Progress.Sep 16 2024, 11:14 AM

• aborrero mentioned this in T374835: cloud: tofu-infra: support neutron security groups.Sep 16 2024, 12:38 PM

• aborrero changed the status of subtask T374824: openstack: make vxlan/ipv6-dualstack network the default for new instances from In Progress to Stalled.Sep 24 2024, 3:56 PM

• aborrero changed the status of subtask T374828: openstack: vxlan: verify nova proxy and floating IPs work with new VXLAN-based network from Open to In Progress.Sep 25 2024, 8:37 AM

• aborrero changed the status of subtask T375596: openstack: verify DNS integration for new VXLAN-based network from Open to In Progress.

• aborrero closed subtask T375596: openstack: verify DNS integration for new VXLAN-based network as Resolved.Sep 25 2024, 9:27 AM

• aborrero changed the status of subtask T375604: openstack: cleanup and prepare security groups from Open to In Progress.Sep 25 2024, 9:59 AM

• aborrero closed subtask T374828: openstack: vxlan: verify nova proxy and floating IPs work with new VXLAN-based network as Resolved.Sep 26 2024, 3:00 PM

• aborrero closed subtask T375604: openstack: cleanup and prepare security groups as Resolved.Sep 27 2024, 8:13 AM

• aborrero changed the status of subtask T377346: openstack: develop a script to migrate a VM instance from the old network setting (vlan) to the new (vxlan, IPv6) from Open to In Progress.Oct 16 2024, 3:12 PM

the script being developed in T377346: openstack: develop a script to migrate a VM instance from the old network setting (vlan) to the new (vxlan, IPv6) is proving to be challenging for a number of reasons.

I wonder if instead of force-migrating VMs to the new network, we could use a different strategy:

use the IPv6 as incentive for users do to the migration
make the new VXLAN/IPv6 network the default via horizon (but make them both available, which may be important, for Toolforge for example)
announce the availability of the new network to the community
for users to move to the new network, they need to re-create their VMs

• aborrero mentioned this in T377467: Decision Request - How to do the Cloud VPS VXLAN/IPv6 migration.Oct 17 2024, 2:51 PM

• aborrero changed the status of subtask T377346: openstack: develop a script to migrate a VM instance from the old network setting (vlan) to the new (vxlan, IPv6) from In Progress to Stalled.Oct 21 2024, 12:14 PM

blocked on T377467: Decision Request - How to do the Cloud VPS VXLAN/IPv6 migration

• aborrero closed subtask T377467: Decision Request - How to do the Cloud VPS VXLAN/IPv6 migration as Resolved.Nov 6 2024, 10:41 AM

• aborrero closed subtask T377346: openstack: develop a script to migrate a VM instance from the old network setting (vlan) to the new (vxlan, IPv6) as Declined.

• aborrero closed subtask T379154: openstack: vxlan: potential changes to cloudvirt MTU to enable jumbo frames as Declined.Nov 7 2024, 12:04 PM

Andrew closed subtask T326373: Migrate Cloud VPS to Neutron Open vSwitch agent as Resolved.Nov 26 2024, 2:28 AM

per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_VXLAN_IPv6_migration this migration is expected to start on 2025-01-06.

leaving ticket as stalled until then.

fnegri moved this task from Backlog to Done on the cloud-services-team (FY2024/2025-Q1-Q2) board.Jan 13 2025, 4:26 PM

fnegri moved this task from Done to Blocked on the cloud-services-team (FY2024/2025-Q1-Q2) board.

fnegri edited projects, added cloud-services-team (FY2024/2025-Q3-Q4); removed cloud-services-team (FY2024/2025-Q1-Q2).Jan 14 2025, 3:20 PM

fnegri moved this task from Backlog to Blocked on the cloud-services-team (FY2024/2025-Q3-Q4) board.

Change #1136980 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/homer/public@master] Cloud network: update policy to support /17 IPv4 aggregates

https://gerrit.wikimedia.org/r/1136980

gerritbot added a project: Patch-For-Review.Apr 16 2025, 11:05 AM

Change #1136980 merged by jenkins-bot:

[operations/homer/public@master] Cloud network: update policy to support /17 IPv4 aggregates

https://gerrit.wikimedia.org/r/1136980

Change #1136991 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/homer/public@master] WMCS: fix typo in updated cloud-in policy

https://gerrit.wikimedia.org/r/1136991

Change #1136991 merged by jenkins-bot:

[operations/homer/public@master] WMCS: fix typo in updated cloud-in policy

https://gerrit.wikimedia.org/r/1136991

Change #1136993 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/homer/public@master] WMCS: fix typo in updated cloud-in policy #2

https://gerrit.wikimedia.org/r/1136993

Change #1136993 merged by jenkins-bot:

[operations/homer/public@master] WMCS: fix typo in updated cloud-in policy #2

https://gerrit.wikimedia.org/r/1136993

Maintenance_bot removed a project: Patch-For-Review.Apr 16 2025, 12:30 PM

Change #1137002 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/homer/public@master] WMCS: Remove cloud-instances2-b specific ranges from BGP policy

https://gerrit.wikimedia.org/r/1137002

gerritbot added a project: Patch-For-Review.Apr 16 2025, 12:52 PM

Change #1137002 merged by jenkins-bot:

[operations/homer/public@master] WMCS: Remove cloud-instances2-b specific ranges from BGP policy

https://gerrit.wikimedia.org/r/1137002

Maintenance_bot removed a project: Patch-For-Review.Apr 16 2025, 1:30 PM

• aborrero closed subtask T37947: Enable IPv6 on CloudVPS as Resolved.Apr 24 2025, 8:54 AM

taavi removed a subtask: T358868: Use BGP to announce VM ranges from cloudnet to cloudgw.Apr 25 2025, 9:20 AM

taavi changed the task status from Stalled to Open.Apr 28 2025, 11:51 AM

• aborrero changed the status of subtask T374824: openstack: make vxlan/ipv6-dualstack network the default for new instances from Stalled to Open.May 3 2025, 4:21 PM

• aborrero changed the status of subtask T374824: openstack: make vxlan/ipv6-dualstack network the default for new instances from Open to In Progress.May 3 2025, 4:25 PM

• aborrero closed subtask T374824: openstack: make vxlan/ipv6-dualstack network the default for new instances as Resolved.May 3 2025, 5:15 PM

I think we can consider this ticket as resolved.

We won't be "migrating" VMs into the VXLAN setup, but rebuilding them into VXLAN over time.

fnegri moved this task from Blocked to Done on the cloud-services-team (FY2024/2025-Q3-Q4) board.Jul 9 2025, 2:58 PM

Migrate Cloud VPS instances to VXLAN based networksClosed, ResolvedPublicActions

Description

Details

Related ObjectsSearch...

Event Timeline

Migrate Cloud VPS instances to VXLAN based networks
Closed, ResolvedPublic
Actions

Related Objects
Search...