openstack: network problems when introducing new networks
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• aborrero
	Nov 25 2024, 11:46 AM

Description

We detected some network problems when introducing the new dualstack and ipv4-only networks.

This ticket is to track the work to identify and fix them.

Details

Related Changes in Gerrit:

Subject	Repo	Branch	Lines +/-
openstack: networktests: support IPv6 and IPv4-only networks	operations/puppet	production	+110 -0
cloudgw: use vlan1120/vlan2120 prefix for FQDN	operations/puppet	production	+8 -8
openstack: networktests: refresh for latest network changes	operations/puppet	production	+15 -15

Customize query in gerrit

Related Changes in GitLab:

Title	Reference	Author	Source Branch	Dest Branch
codfw1dev: flavors: fix project access for g4.cores1.ram1.disk4	repos/cloud/cloud-vps/tofu-infra!171	aborrero	arturo-322-codfw1dev-flavors-f	main
codfw1dev: create network tests instances	repos/cloud/cloud-vps/tofu-infra!170	aborrero	arturo-276-codfw1dev-create-ne	main
codfw1dev: add vxlan-ipv4-only.cloudinstances2b-gw.svc.codfw1dev.wikimedia.cloud FQDN	repos/cloud/cloud-vps/tofu-infra!169	aborrero	arturo-169-codfw1dev-add-vxlan	main
codfw1dev: tools-codfw1dev: manage default security group	repos/cloud/cloud-vps/tofu-infra!140	aborrero	arturo-118-codfw1dev-tools-cod	main

Customize query in GitLab

Related Objects
Search...

Status	Assigned	Task
Open	None	T53494 Use Beta cluster as a true canary for code deployments (epic)
Declined	None	T87220 Minimize infrastructure differences between Beta Cluster and production
Open	None	T211677 Support IPv6 in beta
Open	None	T404466 Support IPv6-only VMs
Open	None	T392688 Enable IPv6 on Cloud VPS infrastructure services
Resolved	taavi	T379175 Enable IPv6 for the Cloud VPS web proxy
Open	None	T270694 CloudVPS: introduce tenant networks
Resolved	• aborrero	T364725 Migrate Cloud VPS instances to VXLAN based networks
Open	None	T392509 Enable IPv6 for Toolforge services
Resolved	taavi	T392506 Enable IPv6 for tools.wmflabs.org / *.toolserver.org legacy redirector service
Resolved	taavi	T211575 Enable IPv6 on toolforge.org
Open	None	T220306 Add IPv6 monitoring
Resolved	• aborrero	T37947 Enable IPv6 on CloudVPS
Resolved	• aborrero	T380174 CloudVPS: IPv6 in eqiad1
Resolved	• aborrero	T380728 openstack: network problems when introducing new networks
Resolved	cmooney	T389958 WMCS Eqiad: Enable IPv6 in cloud vrf on switches
Resolved	• aborrero	T390877 HAProxyServiceUnavailable
Resolved	• aborrero	T391043 CephClusterInError #page Ceph cluster in eqiad is in error status
Resolved	• aborrero	T391040 CloudVirtDown Cloudvirt node cloudvirt1063 is down. #page
Resolved	• aborrero	T391039 CephSlowOps Ceph cluster in eqiad has 486 slow ops
Resolved	• aborrero	T391325 openstack: improve networktests for newer network setup
Resolved	fnegri	T391467 gitlab ci: validate secrets settings in pipeline for tofu integration

Event Timeline

• aborrero created this task.Nov 25 2024, 11:46 AM

• aborrero changed the task status from Open to In Progress.

• aborrero triaged this task as Medium priority.

• aborrero moved this task from Backlog to Doing on the User-aborrero board.

• aborrero renamed this task from openstack: network problems when introducing new dualstack and ipv4-only networks to openstack: network problems when introducing new networks.Nov 25 2024, 11:49 AM

aborrero opened https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/140

codfw1dev: tools-codfw1dev: manage default security group

aborrero merged https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/140

codfw1dev: tools-codfw1dev: manage default security group

Maintenance_bot removed a project: Patch-For-Review.Nov 25 2024, 12:30 PM

Change #1097370 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] openstack: networktests: refresh for latest network changes

https://gerrit.wikimedia.org/r/1097370

Change #1097370 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] openstack: networktests: refresh for latest network changes

https://gerrit.wikimedia.org/r/1097370

Maintenance_bot removed a project: Patch-For-Review.Nov 25 2024, 1:31 PM

I detected a few inconsistencies in the network testing scripts, I will fix them.

Among others, I will use the vlanX120.cloudgwYYYY.<deploy>.wikimediacloud.org scheme for the IP addresses, like we do for vlan1107/vlan2107.

Change #1097380 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloudgw: use vlan1120/vlan2120 prefix for FQDN

https://gerrit.wikimedia.org/r/1097380

gerritbot added a project: Patch-For-Review.Nov 25 2024, 1:44 PM

Change #1097380 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloudgw: use vlan1120/vlan2120 prefix for FQDN

https://gerrit.wikimedia.org/r/1097380

Maintenance_bot removed a project: Patch-For-Review.Nov 25 2024, 2:31 PM

Change #1097440 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] openstack: networktests: support IPv6 and IPv4-only networks

https://gerrit.wikimedia.org/r/1097440

gerritbot added a project: Patch-For-Review.Nov 25 2024, 4:59 PM

today @cmooney reported this was maybe caused by some inconsistency on the edge routing configuration for cloudsw devices.

Dzahn unsubscribed.Jan 29 2025, 6:52 PM

• aborrero moved this task from Doing to Next on the User-aborrero board.Feb 4 2025, 4:44 PM

taavi moved this task from Unsorted to Network on the Cloud-VPS board.Feb 8 2025, 9:37 AM

• aborrero mentioned this in T380174: CloudVPS: IPv6 in eqiad1.Mar 4 2025, 10:35 AM

• aborrero changed the task status from In Progress to Open.Mar 4 2025, 11:25 AM

Geertivp subscribed.Mar 4 2025, 11:05 PM

• aborrero moved this task from Next to Doing on the User-aborrero board.Mar 19 2025, 1:18 PM

I'd forgot about this task, apologies.

The reason the problems occurred last time with this is the cloud switches had not yet been configured with IPv6 addressing on the various host-facing vlans, nor on their interconnects or link to the WMF core routers. Which basically meant IPv6 routing in and out of the cloud network was not functioning (indeed not even configured), and the cloud systems were firing traffic into a black hole.

We're working to add the IPv6 interfaces, routing protocols, public BGP announcements for assigned ranges etc now, after which we should be able to re-attempt enabling IPv6 on the host side.

The only caveat to that is I don't think any IPv4 networks should have been affected by the lack of v6 configuration on the physical infra, so perhaps there is some other issue here. Either way we should ensure the v6 infra is fully configured before we re-attempt.

taavi added a subtask: T389958: WMCS Eqiad: Enable IPv6 in cloud vrf on switches.Mar 26 2025, 9:26 AM

• aborrero moved this task from Doing to Blocked/waiting on the User-aborrero board.Apr 4 2025, 9:43 AM

Change #1097440 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] openstack: networktests: support IPv6 and IPv4-only networks

https://gerrit.wikimedia.org/r/1097440

aborrero opened https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/169

codfw1dev: add vxlan-ipv4-only.cloudinstances2b-gw.svc.codfw1dev.wikimedia.cloud FQDN

aborrero merged https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/169

codfw1dev: add vxlan-ipv4-only.cloudinstances2b-gw.svc.codfw1dev.wikimedia.cloud FQDN

Maintenance_bot removed a project: Patch-For-Review.Apr 4 2025, 3:30 PM

aborrero opened https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/170

codfw1dev: create network tests instances

aborrero merged https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/170

codfw1dev: create network tests instances

aborrero opened https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/171

codfw1dev: flavors: fix project access for g4.cores1.ram1.disk4

aborrero merged https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/171

codfw1dev: flavors: fix project access for g4.cores1.ram1.disk4

Maintenance_bot removed a project: Patch-For-Review.Apr 7 2025, 11:31 AM

cmooney closed subtask T389958: WMCS Eqiad: Enable IPv6 in cloud vrf on switches as Resolved.Apr 7 2025, 12:51 PM

Mentioned in SAL (#wikimedia-cloud) [2025-04-07T15:30:30Z] <arturo> create a bunch of VMs by hand, like networktests-vlan-legacy-floating T380728

Mentioned in SAL (#wikimedia-cloud) [2025-04-07T15:30:48Z] <arturo> [codfw1dev] testlabs create a bunch of VMs by hand, like networktests-vlan-legacy-floating T380728