Page MenuHomePhabricator

openstack: improve networktests for newer network setup
Closed, ResolvedPublic

Description

With the introduction of IPv6, network tests need to be more robust.

Among other things:

  • verify IPv6 dualstack, IPv4 only and VLAN legacy network connectivy, with and without floating IP
  • provision infrastructure on purpose, instead of reusing other random VMs which may go away at any time, breaking the tests
  • introduce opentofu for this infrastructure provisioning

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
aborrero changed the task status from Open to In Progress.Apr 8 2025, 9:06 AM
aborrero triaged this task as Medium priority.
aborrero moved this task from Backlog to Doing on the User-aborrero board.

Mentioned in SAL (#wikimedia-cloud) [2025-04-10T09:05:22Z] <arturo> [codfw1dev] root@cloudcontrol2004-dev:~# wmcs-makedomain --project testlabs --domain testlabs.codfw1dev.wmcloud.org --orig-project cloudinfra-codfw1dev (T391325)

aborrero opened https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/177

codfw1dev: add AAAA record for cloudinstances2b-gw.svc.codfw1dev.wikimedia.cloud

aborrero merged https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/177

codfw1dev: add AAAA record for cloudinstances2b-gw.svc.codfw1dev.wikimedia.cloud

Change #1135689 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloudgw: codfw: route IPv4-only subnet

https://gerrit.wikimedia.org/r/1135689

Change #1135689 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloudgw: codfw: route IPv4-only subnet

https://gerrit.wikimedia.org/r/1135689

Change #1135699 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] network: add Cloud VPS codfw1dev VXLAN / IPv4-only CIDR

https://gerrit.wikimedia.org/r/1135699

Change #1135699 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] network: add Cloud VPS codfw1dev VXLAN / IPv4-only CIDR

https://gerrit.wikimedia.org/r/1135699

aborrero merged https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/178

cloudinfra-codfw1dev: allow puppet-agent to run from the VXLAN / IPv4 subnet

Change #1135715 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] network: data: fix entry prefix to include 'cloud-instances'

https://gerrit.wikimedia.org/r/1135715

Change #1135715 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] network: data: fix entry prefix to include 'cloud-instances'

https://gerrit.wikimedia.org/r/1135715

Change #1135942 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] openstack: networktests: refresh all tests in the new IPv6-enabled networks

https://gerrit.wikimedia.org/r/1135942

Change #1135942 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] openstack: networktests: refresh all tests in the new IPv6-enabled networks

https://gerrit.wikimedia.org/r/1135942

Change #1135944 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] openstack: codfw1dev: networktests: fix typo in envvars

https://gerrit.wikimedia.org/r/1135944

Change #1135944 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] openstack: codfw1dev: networktests: fix typo in envvars

https://gerrit.wikimedia.org/r/1135944

Change #1135947 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] openstack: codfw1dev: fix typos in networktests

https://gerrit.wikimedia.org/r/1135947

Change #1135947 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] openstack: codfw1dev: fix typos in networktests

https://gerrit.wikimedia.org/r/1135947

Change #1135949 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] openstack: codfw1dev: networktests: refresh floating VM IP address

https://gerrit.wikimedia.org/r/1135949

Change #1135949 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] openstack: codfw1dev: networktests: refresh floating VM IP address

https://gerrit.wikimedia.org/r/1135949

Change #1136681 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] openstack: networktests: extend to check DNS, LDAP, internet, etc

https://gerrit.wikimedia.org/r/1136681

Change #1136681 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] openstack: networktests: extend to check DNS, LDAP, internet, etc

https://gerrit.wikimedia.org/r/1136681

Change #1136685 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] openstack: networktests: fix yaml typos

https://gerrit.wikimedia.org/r/1136685

Change #1136685 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] openstack: networktests: fix yaml typos

https://gerrit.wikimedia.org/r/1136685

Change #1136687 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] openstack: networktests: more YAML formatting

https://gerrit.wikimedia.org/r/1136687

Change #1136687 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] openstack: networktests: more YAML formatting

https://gerrit.wikimedia.org/r/1136687

aborrero claimed this task.

completed.

We have now 86 tests covering a variety of uses cases, with infra provisioned via https://gitlab.wikimedia.org/repos/cloud/cloud-vps/networktests-tofu-provisioning

aborrero@cloudcontrol2004-dev:~ 15m6s $ sudo -i cmd-checklist-runner --config-file /etc/networktests/networktests.yaml
[2025-04-15 11:45:03] INFO: --- cloudcontrol2004-dev Debian GNU/Linux 12 (bookworm) 6.1.0-32-amd64
[2025-04-15 11:45:03] INFO: ---
[2025-04-15 11:45:03] INFO: running: basic IPv6 ping to cloudgw addresses (raw addresses) from outside the cloud network
[2025-04-15 11:45:03] INFO: running: basic IPv6 ping to cloudgw addresses (DNS names) from outside the cloud network
[2025-04-15 11:45:03] INFO: running: basic IPv6 ping to neutron WAN from outside the cloud network
[2025-04-15 11:45:03] INFO: running: from VM VXLAN/Dualstack to neutron VXLAN/Dualstack IPv6 ping
[2025-04-15 11:45:05] INFO: running: from VM VXLAN/Dualstack to neutron VXLAN/Dualstack IPv4 ping
[2025-04-15 11:45:07] INFO: running: from VM VXLAN/Dualstack to VM VXLAN/IPv4-only IPv4 ping
[2025-04-15 11:45:10] INFO: running: from VM VXLAN/Dualstack to VM VXLAN/IPv4-only-fip IPv4 ping
[2025-04-15 11:45:12] INFO: running: from VM VXLAN/Dualstack to VM VLAN/legacy IPv4 ping
[2025-04-15 11:45:14] INFO: running: from VM VXLAN/Dualstack to VM VLAN/legacy-fip IPv4 ping
[2025-04-15 11:45:16] INFO: running: from VM VXLAN/IPv4-only to neutron VXLAN/IPv4-only IPv4 ping
[2025-04-15 11:45:18] INFO: running: from VM VXLAN/IPv4-only to VM VXLAN/Dualstack IPv4 ping
[2025-04-15 11:45:20] INFO: running: from VM VXLAN/IPv4-only to VM VXLAN/IPv4-only-fip IPv4 ping
[2025-04-15 11:45:22] INFO: running: from VM VXLAN/IPv4-only to VM VLAN/legacy IPv4 ping
[2025-04-15 11:45:24] INFO: running: from VM VXLAN/IPv4-only to VM VLAN/legacy-fip IPv4 ping
[2025-04-15 11:45:26] INFO: running: from VM VXLAN/IPv4-only-fip to neutron VXLAN/IPv4-only IPv4 ping
[2025-04-15 11:45:29] INFO: running: from VM VXLAN/IPv4-only-fip to VM VXLAN/Dualstack IPv4 ping
[2025-04-15 11:45:31] INFO: running: from VM VXLAN/IPv4-only-fip to VM VXLAN/IPv4-only IPv4 ping
[2025-04-15 11:45:33] INFO: running: from VM VXLAN/IPv4-only-fip to VM VLAN/legacy IPv4 ping
[2025-04-15 11:45:35] INFO: running: from VM VXLAN/IPv4-only-fip to VM VLAN/legacy-fip IPv4 ping
[2025-04-15 11:45:37] INFO: running: from VM VLAN/legacy to neutron VLAN/legacy IPv4 ping
[2025-04-15 11:45:39] INFO: running: from VM VLAN/legacy to VM VXLAN/Dualstack IPv4 ping
[2025-04-15 11:45:41] INFO: running: from VM VLAN/legacy to VM VXLAN/IPv4-only IPv4 ping
[2025-04-15 11:45:43] INFO: running: from VM VLAN/legacy to VM VXLAN/IPv4-only-fip IPv4 ping
[2025-04-15 11:45:44] INFO: running: from VM VLAN/legacy to VM VLAN/legacy-fip IPv4 ping
[2025-04-15 11:45:46] INFO: running: from VM VLAN/legacy-fip to neutron VLAN/legacy IPv4 ping
[2025-04-15 11:45:49] INFO: running: from VM VLAN/legacy-fip to VM VXLAN/Dualstack IPv4 ping
[2025-04-15 11:45:51] INFO: running: from VM VLAN/legacy-fip to VM VXLAN/IPv4-only IPv4 ping
[2025-04-15 11:45:53] INFO: running: from VM VLAN/legacy-fip to VM VXLAN/IPv4-only-fip IPv4 ping
[2025-04-15 11:45:55] INFO: running: from VM VLAN/legacy-fip to VM VLAN/legacy-fip IPv4 ping
[2025-04-15 11:45:57] INFO: running: from VXLAN/dualstack to cloudgw VIRT gateway IPv6 ping
[2025-04-15 11:45:59] INFO: running: from VXLAN/dualstack to cloudgw VIRT gateway IPv4 ping
[2025-04-15 11:46:01] INFO: running: from VXLAN/IPv4-only to cloudgw VIRT gateway IPv4 ping
[2025-04-15 11:46:03] INFO: running: from VXLAN/IPv4-only-fip to cloudgw VIRT gateway IPv4 ping
[2025-04-15 11:46:06] INFO: running: from VLAN/legacy to cloudgw VIRT gateway IPv4 ping
[2025-04-15 11:46:08] INFO: running: from VLAN/legacy-fip to cloudgw VIRT gateway IPv4 ping
[2025-04-15 11:46:10] INFO: running: from VXLAN/dualstack contacting the internet using IPv6
[2025-04-15 11:46:12] INFO: running: from VXLAN/dualstack contacting the internet using IPv4 uses NAT address
[2025-04-15 11:46:14] INFO: running: from VXLAN/IPv4-only contacting the internet uses NAT address
[2025-04-15 11:46:16] INFO: running: from VXLAN/IPv4-only-fip contacting the internet uses FIP
[2025-04-15 11:46:19] INFO: running: from VLAN/legacy contacting the internet uses NAT address
[2025-04-15 11:46:21] INFO: running: from VLAN/legacy-fip contacting the internet uses FIP
[2025-04-15 11:46:23] INFO: running: from VXLAN/dualstack contacting the wikis using IPv6
[2025-04-15 11:46:26] INFO: running: from VXLAN/dualstack contacting the wikis using IPv4
[2025-04-15 11:46:28] INFO: running: from VXLAN/IPv4-only contacting the wikis without NAT address
[2025-04-15 11:46:30] INFO: running: from VXLAN/IPv4-only-fip contacting the wikis with FIP address
[2025-04-15 11:46:32] INFO: running: from VLAN/legacy contacting the wikis wihtout NAT address
[2025-04-15 11:46:34] INFO: running: from VLAN/legacy-fip contacting the wikis with FIP address
[2025-04-15 11:46:37] INFO: running: from VXLAN/dualstack can contact auth DNS server, using IPv4
[2025-04-15 11:46:40] INFO: running: from VXLAN/IPv4-only can contact auth DNS server
[2025-04-15 11:46:42] INFO: running: from VXLAN/IPv4-only-fip can contact auth DNS server
[2025-04-15 11:46:44] INFO: running: from VLAN/legacy can contact auth DNS server
[2025-04-15 11:46:46] INFO: running: from VLAN/legacy-fip can contact auth DNS server
[2025-04-15 11:46:48] INFO: running: from VXLAN/dualstack can contact recursor DNS server, using IPv4
[2025-04-15 11:46:51] INFO: running: from VXLAN/IPv4-only can contact recursor DNS server
[2025-04-15 11:46:53] INFO: running: from VXLAN/IPv4-only-fip can contact recursor DNS server
[2025-04-15 11:46:55] INFO: running: from VLAN/legacy can contact recursor DNS server
[2025-04-15 11:46:56] INFO: running: from VLAN/legacy-fip can contact recursor DNS server
[2025-04-15 11:46:58] INFO: running: from VXLAN/dualstack can contact LDAP server, using IPv4
[2025-04-15 11:47:00] INFO: running: from VXLAN/IPv4-only can contact LDAP server
[2025-04-15 11:47:02] INFO: running: from VXLAN/IPv4-only-fip can contact LDAP server
[2025-04-15 11:47:04] INFO: running: from VLAN/legacy can contact LDAP server
[2025-04-15 11:47:06] INFO: running: from VLAN/legacy-fip can contact LDAP server
[2025-04-15 11:47:08] INFO: running: from VXLAN/dualstack can contact openstack APIs, using IPv4
[2025-04-15 11:47:10] INFO: running: from VXLAN/IPv4-only can contact openstack APIs
[2025-04-15 11:47:12] INFO: running: from VXLAN/IPv4-only-fip can contact openstack APIs
[2025-04-15 11:47:14] INFO: running: from VLAN/legacy can contact openstack APIs
[2025-04-15 11:47:16] INFO: running: from VLAN/legacy-fip can contact openstack APIs
[2025-04-15 11:47:18] INFO: running: basic ping to cloudgw addresses (raw addresses) from outside the cloud network
[2025-04-15 11:47:18] INFO: running: basic ping to cloudgw addresses (DNS names) from outside the cloud network
[2025-04-15 11:47:18] INFO: running: basic ping to neutron WAN from outside the cloud network
[2025-04-15 11:47:18] INFO: running: basic ping to neutron VIRT gateway from within the cloud virtual network, no floating IP
[2025-04-15 11:47:22] INFO: running: basic ping to neutron VIRT gateway from within the cloud virtual network, with floating IP
[2025-04-15 11:47:26] INFO: running: VM (no floating IP) contacting the internet gets NAT'd using routing_source_ip
[2025-04-15 11:47:28] INFO: running: VM (no floating IP) contacting an address covered by dmz_cidr doesn't get NAT'd
[2025-04-15 11:47:30] INFO: running: VM (using floating IP) isn't affected by either routing_source_ip or dmz_cidr
[2025-04-15 11:47:34] INFO: running: VM (no floating IP) can contact auth DNS server
[2025-04-15 11:47:35] INFO: running: VM (no floating IP) can contact recursor DNS server
[2025-04-15 11:47:37] INFO: running: VM (using floating IP) can contact auth DNS server
[2025-04-15 11:47:39] INFO: running: VM (using floating IP) can contact recursor DNS server
[2025-04-15 11:47:41] INFO: running: VM (using floating IP) can contact LDAP server
[2025-04-15 11:47:43] INFO: running: VM (not using floating IP) can contact LDAP server
[2025-04-15 11:47:45] INFO: running: VM (using floating IP) can contact openstack API
[2025-04-15 11:47:47] INFO: running: VM (no floating IP) can contact openstack API
[2025-04-15 11:47:49] INFO: running: puppetmasters can sync git tree
[2025-04-15 11:48:31] INFO: running: VM (using floating IP) can read dumps NFS
[2025-04-15 11:48:33] INFO: running: VM (no floating IP) can read dumps NFS
[2025-04-15 11:48:35] INFO: ---
[2025-04-15 11:48:35] INFO: --- passed tests: 86
[2025-04-15 11:48:35] INFO: --- failed tests: 0
[2025-04-15 11:48:35] INFO: --- total tests: 86

Change #1137785 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] openstack: networktests: enable IPv6 tests on eqiad1

https://gerrit.wikimedia.org/r/1137785

Change #1137785 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] openstack: networktests: enable IPv6 tests on eqiad1

https://gerrit.wikimedia.org/r/1137785