Page MenuHomePhabricator
Create Task
Maniphest T380174

CloudVPS: IPv6 in eqiad1
Closed, ResolvedPublic
Actions

Description

This ticket is to track the work to enable IPv6 in eqiad1.

Details

Related Changes in Gerrit:
Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenNoneT53494 Use Beta cluster as a true canary for code deployments (epic)
 DeclinedNoneT87220 Minimize infrastructure differences between Beta Cluster and production
 OpenNoneT211677 Support IPv6 in beta
 OpenNoneT404466 Support IPv6-only VMs
 OpenNoneT392688 Enable IPv6 on Cloud VPS infrastructure services
 ResolvedtaaviT379175 Enable IPv6 for the Cloud VPS web proxy
 OpenNoneT270694 CloudVPS: introduce tenant networks
 Resolved• aborreroT364725 Migrate Cloud VPS instances to VXLAN based networks
 OpenNoneT392509 Enable IPv6 for Toolforge services
 ResolvedtaaviT392506 Enable IPv6 for tools.wmflabs.org / *.toolserver.org legacy redirector service
 ResolvedtaaviT211575 Enable IPv6 on toolforge.org
 OpenNoneT220306 Add IPv6 monitoring
 Resolved• aborreroT37947 Enable IPv6 on CloudVPS
 Resolved• aborreroT380174 CloudVPS: IPv6 in eqiad1
 Resolved• aborreroT380703 versions.toolforge.org is down
 Resolved• aborreroT380728 openstack: network problems when introducing new networks
 ResolvedcmooneyT389958 WMCS Eqiad: Enable IPv6 in cloud vrf on switches
 Resolved• aborreroT390877 HAProxyServiceUnavailable
Resolved• aborreroT391043 CephClusterInError #page Ceph cluster in eqiad is in error status
 Resolved• aborreroT391040 CloudVirtDown Cloudvirt node cloudvirt1063 is down. #page
 Resolved• aborreroT391039 CephSlowOps Ceph cluster in eqiad has 486 slow ops
 Resolved• aborreroT391325 openstack: improve networktests for newer network setup
 ResolvedfnegriT391467 gitlab ci: validate secrets settings in pipeline for tofu integration
 ResolvedcmooneyT380746 dns: add PTR support for 2a02:ec80:a000::
 ResolvedtaaviT386689 Add new WMCS IPv6 ranges to MediaWiki configuration where required
 ResolvedcmooneyT388379 ProbeDown

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
• aborrero changed the status of subtask T380728: openstack: network problems when introducing new networks from Open to In Progress.Nov 25 2024, 11:47 AM
gerritbot added a comment.Nov 25 2024, 2:33 PM

Change #1097397 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/dns@master] Add reverse for new IPv6 range assigned by cloud services

https://gerrit.wikimedia.org/r/1097397

gerritbot added a project: Patch-For-Review.Nov 25 2024, 2:33 PM
gerritbot added a comment.Nov 25 2024, 2:38 PM

Change #1097397 merged by Cathal Mooney:

[operations/dns@master] Add reverse for new IPv6 range assigned by cloud services

https://gerrit.wikimedia.org/r/1097397

Maintenance_bot removed a project: Patch-For-Review.Nov 25 2024, 3:32 PM
• aborrero moved this task from Doing to Next on the User-aborrero board.Feb 4 2025, 4:44 PM
taavi moved this task from Unsorted to Network on the Cloud-VPS board.Feb 8 2025, 9:37 AM
Geertivp subscribed.Feb 14 2025, 4:12 PM
• aborrero added a comment.Mar 4 2025, 10:35 AM

Given we have a theory for why T380728: openstack: network problems when introducing new networks happened, we (@cmooney and me) are trying to target end of March for a rollout of this.

• aborrero changed the status of subtask T380728: openstack: network problems when introducing new networks from In Progress to Open.Mar 4 2025, 11:25 AM
• aborrero added a subtask: T388379: ProbeDown .Mar 10 2025, 2:10 PM
• aborrero moved this task from Next to Doing on the User-aborrero board.Mar 19 2025, 1:18 PM
taavi closed subtask T388379: ProbeDown as Resolved.Mar 20 2025, 9:11 AM
• aborrero moved this task from Doing to Blocked/waiting on the User-aborrero board.Apr 4 2025, 9:43 AM
• aborrero moved this task from Blocked/waiting to Doing on the User-aborrero board.Apr 15 2025, 12:51 PM
CodeReviewBot added a project: Patch-For-Review.Apr 15 2025, 1:03 PM

aborrero opened https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/197

eqiad1: refresh FQDN for neutron virtual router

gerritbot added a comment.Apr 15 2025, 1:08 PM

Change #1136719 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] openstack: networktests: refresh FQDN of the neutron virtual router

https://gerrit.wikimedia.org/r/1136719

CodeReviewBot added a comment.Apr 15 2025, 3:13 PM

aborrero merged https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/197

eqiad1: refresh FQDN for neutron virtual router

gerritbot added a comment.Apr 16 2025, 9:00 AM

Change #1136719 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] openstack: networktests: refresh FQDN of the neutron virtual router

https://gerrit.wikimedia.org/r/1136719

Maintenance_bot removed a project: Patch-For-Review.Apr 16 2025, 9:30 AM
CodeReviewBot added a project: Patch-For-Review.Apr 16 2025, 12:18 PM

aborrero opened https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/198

eqiad1: introduce VXLAN/IPv4-only settings

CodeReviewBot added a comment.Apr 16 2025, 12:26 PM

aborrero merged https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/198

eqiad1: introduce VXLAN/IPv4-only settings

Maintenance_bot removed a project: Patch-For-Review.Apr 16 2025, 12:31 PM
CodeReviewBot added a project: Patch-For-Review.Apr 21 2025, 10:58 AM

aborrero opened https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/204

eqiad1: enable VXLAN/dualstack network

• aborrero closed subtask T380728: openstack: network problems when introducing new networks as Resolved.Apr 21 2025, 11:12 AM
• aborrero added a comment.Apr 21 2025, 11:35 AM

merging https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/204 has been scheduled for 2025-04-23 @ 09:30 UTC.

gerritbot added a comment.Apr 21 2025, 3:46 PM

Change #1137793 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloudgw: enable IPv6

https://gerrit.wikimedia.org/r/1137793

• aborrero added a comment.Apr 21 2025, 3:52 PM

we need to allocate this in netbox:

  • 2a02:ec80:a000:fe04::1003:1 (cloudgw1003 virt leg)
  • 2a02:ec80:a000:fe03::1003:1 (cloudgw1003 wan leg)
  • 2a02:ec80:a000:fe04::1004:1 (cloudgw1004 virt leg)
  • 2a02:ec80:a000:fe03::1004:1 (cloudgw1004 wan leg)
  • 2a02:ec80:a100:fe04::1:1 (cloudgw VIP virt leg)
  • 2a02:ec80:a100:fe03::2 (cloudgw VIP wan leg)
gerritbot added a comment.Apr 23 2025, 9:00 AM

Change #1137793 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloudgw: enable IPv6

https://gerrit.wikimedia.org/r/1137793

ops-monitoring-bot added a comment.Apr 23 2025, 9:04 AM

Host rebooted by aborrero@cumin1002 with reason: enable IPv6

Stashbot added a comment.Apr 23 2025, 9:14 AM

Mentioned in SAL (#wikimedia-cloud) [2025-04-23T09:14:25Z] <arturo> enable IPv6 on cloudgw (T380174) -- includes server reboot

ops-monitoring-bot added a comment.Apr 23 2025, 9:18 AM

Host rebooted by aborrero@cumin1002 with reason: enable IPv6

gerritbot added a comment.Apr 23 2025, 9:24 AM

Change #1138297 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloudgw: fix IPv6 range on VIPs

https://gerrit.wikimedia.org/r/1138297

gerritbot added a comment.Apr 23 2025, 9:25 AM

Change #1138297 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloudgw: fix IPv6 range on VIPs

https://gerrit.wikimedia.org/r/1138297

gerritbot added a comment.Apr 23 2025, 9:44 AM

Change #1138304 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloudgw: update wan VIP

https://gerrit.wikimedia.org/r/1138304

gerritbot added a comment.Apr 23 2025, 9:47 AM

Change #1138304 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloudgw: update wan VIP

https://gerrit.wikimedia.org/r/1138304

gerritbot added a comment.Apr 23 2025, 9:58 AM

Change #1138306 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloudgw: use rack-specific default IPv6 routes

https://gerrit.wikimedia.org/r/1138306

gerritbot added a comment.Apr 23 2025, 10:04 AM

Change #1138306 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloudgw: use rack-specific default IPv6 routes

https://gerrit.wikimedia.org/r/1138306

ops-monitoring-bot added a comment.Apr 23 2025, 10:07 AM

Host rebooted by aborrero@cumin1002 with reason: enable IPv6

CodeReviewBot added a comment.Apr 23 2025, 10:14 AM

aborrero merged https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/204

eqiad1: enable VXLAN/dualstack network

Maintenance_bot removed a project: Patch-For-Review.Apr 23 2025, 10:30 AM
CodeReviewBot added a project: Patch-For-Review.Apr 23 2025, 11:22 AM

aborrero opened https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/208

eqiad1: network: add VXLAN/dualstack

CodeReviewBot added a comment.Apr 23 2025, 11:29 AM

aborrero merged https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/208

eqiad1: network: add VXLAN/dualstack

Maintenance_bot removed a project: Patch-For-Review.Apr 23 2025, 11:30 AM
CodeReviewBot added a project: Patch-For-Review.Apr 23 2025, 11:33 AM

aborrero opened https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/209

eqiad1: subnets: introduce vxlan-dualstack-ipv4

CodeReviewBot added a comment.Apr 23 2025, 11:42 AM

aborrero merged https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/209

eqiad1: subnets: introduce vxlan-dualstack-ipv4

CodeReviewBot added a comment.Apr 23 2025, 11:44 AM

aborrero opened https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/210

eqiad1: subnets: introduce vxlan-dualstack-ipv6

CodeReviewBot added a comment.Apr 23 2025, 11:45 AM

aborrero merged https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/210

eqiad1: subnets: introduce vxlan-dualstack-ipv6

CodeReviewBot added a comment.Apr 23 2025, 11:47 AM

aborrero opened https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/211

eqiad1: routers: introduce vxlan-dualstack-ipv6

CodeReviewBot added a comment.Apr 23 2025, 11:48 AM

aborrero merged https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/211

eqiad1: routers: introduce vxlan-dualstack-ipv6

CodeReviewBot added a comment.Apr 23 2025, 11:49 AM

aborrero opened https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/212

eqiad1: ports: introduce cloudinstances2b-gw-dualstack

CodeReviewBot added a comment.Apr 23 2025, 11:57 AM

aborrero merged https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/212

eqiad1: ports: introduce cloudinstances2b-gw-dualstack

CodeReviewBot added a comment.Apr 23 2025, 11:58 AM

aborrero opened https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/213

eqiad1: ports: introduce cloudinstances2b-gw-dualstack-v6

CodeReviewBot added a comment.Apr 23 2025, 11:59 AM

aborrero merged https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/213

eqiad1: ports: introduce cloudinstances2b-gw-dualstack-v6

CodeReviewBot added a comment.Apr 23 2025, 12:01 PM

aborrero opened https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/214

eqiad1: router_intercaces: introduce cloudinstances2b-gw-dualstack

CodeReviewBot added a comment.Apr 23 2025, 12:02 PM

aborrero merged https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/214

eqiad1: router_intercaces: introduce cloudinstances2b-gw-dualstack

CodeReviewBot added a comment.Apr 23 2025, 12:06 PM

aborrero opened https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/215

eqiad1: router_intercaces: introduce cloudinstances2b-gw-dualstack-v6

CodeReviewBot added a comment.Apr 23 2025, 12:07 PM

aborrero merged https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/215

eqiad1: router_intercaces: introduce cloudinstances2b-gw-dualstack-v6

CodeReviewBot added a comment.Apr 23 2025, 12:10 PM

aborrero opened https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/216

eqiad1: dns: add records for new neutron router addresses

CodeReviewBot added a comment.Apr 23 2025, 12:12 PM

aborrero merged https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/216

eqiad1: dns: add records for new neutron router addresses

CodeReviewBot added a comment.Apr 23 2025, 12:19 PM

aborrero opened https://gitlab.wikimedia.org/repos/cloud/cloud-vps/networktests-tofu-provisioning/-/merge_requests/16

networktests: create dualstack infra in both deployments

CodeReviewBot added a comment.Apr 23 2025, 12:22 PM

aborrero merged https://gitlab.wikimedia.org/repos/cloud/cloud-vps/networktests-tofu-provisioning/-/merge_requests/16

networktests: create dualstack infra in both deployments

• aborrero added a subscriber: taavi.Apr 23 2025, 12:25 PM

we discovered that when enabling https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/commit/1a85f66e47b615f16e9dc492d823c313c0bdf086 on the neutron router, it would magically reset the enable_snat bit on the openstack API. This resulted in the whole network not working.

Thanks to opentofu, this was detected in the diff of the next patch, and corrected with a simple tofu apply.

Thanks @taavi for spotting it.

gerritbot added a comment.Apr 23 2025, 12:28 PM

Change #1138336 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloudgw: enable NAT for additional VXLAN subnets

https://gerrit.wikimedia.org/r/1138336

gerritbot added a comment.Apr 23 2025, 12:30 PM

Change #1138336 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloudgw: enable NAT for additional VXLAN subnets

https://gerrit.wikimedia.org/r/1138336

Maintenance_bot removed a project: Patch-For-Review.Apr 23 2025, 12:31 PM
ops-monitoring-bot added a comment.Apr 23 2025, 12:35 PM

Host rebooted by aborrero@cumin1002 with reason: enable IPv6

gerritbot added a comment.Apr 23 2025, 12:38 PM

Change #1138339 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] network: Update new WMCS IPv6 ranges

https://gerrit.wikimedia.org/r/1138339

gerritbot added a project: Patch-For-Review.Apr 23 2025, 12:38 PM
gerritbot added a comment.Apr 23 2025, 12:46 PM

Change #1138342 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:dns: Update discovery-map for new WMCS addresses

https://gerrit.wikimedia.org/r/1138342

gerritbot added a comment.Apr 23 2025, 12:55 PM

Change #1138339 merged by Majavah:

[operations/puppet@production] network: Update new WMCS IPv6 ranges

https://gerrit.wikimedia.org/r/1138339

ops-monitoring-bot added a comment.Apr 23 2025, 1:05 PM

Host rebooted by aborrero@cumin1002 with reason: enable IPv6

gerritbot added a comment.Apr 23 2025, 1:21 PM

Change #1138353 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] openstack: eqiad1: networktests: fix typos

https://gerrit.wikimedia.org/r/1138353

gerritbot added a comment.Apr 23 2025, 1:26 PM

Change #1138353 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] openstack: eqiad1: networktests: fix typos

https://gerrit.wikimedia.org/r/1138353

gerritbot added a comment.Apr 23 2025, 1:37 PM

Change #1138356 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] hieradata: Set eqiad1 domain_id_internal_reverse_v6

https://gerrit.wikimedia.org/r/1138356

gerritbot added a comment.Apr 23 2025, 1:44 PM

Change #1138356 merged by Majavah:

[operations/puppet@production] hieradata: Set eqiad1 domain_id_internal_reverse_v6

https://gerrit.wikimedia.org/r/1138356

gerritbot added a comment.Apr 23 2025, 2:12 PM

Change #1138371 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] hieradata: Enable wmcs_nova_fixed_ptr in eqiad1

https://gerrit.wikimedia.org/r/1138371

gerritbot added a comment.Apr 23 2025, 2:16 PM

Change #1138371 merged by Majavah:

[operations/puppet@production] hieradata: Enable wmcs_nova_fixed_ptr in eqiad1

https://gerrit.wikimedia.org/r/1138371

Stashbot added a comment.Apr 23 2025, 2:27 PM

Mentioned in SAL (#wikimedia-cloud) [2025-04-23T14:27:11Z] <arturo> enabling IPv6 dualstack on neutron virtual router (T380174)

taavi closed subtask T380746: dns: add PTR support for 2a02:ec80:a000:: as Resolved.Apr 23 2025, 4:18 PM
• aborrero closed this task as Resolved.Apr 24 2025, 8:53 AM
• aborrero claimed this task.

IPv6 is up and running.

gerritbot added a comment.Apr 24 2025, 11:39 AM

Change #1138718 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] openstack: networktests: cleanup and simplify

https://gerrit.wikimedia.org/r/1138718

gerritbot added a comment.Apr 24 2025, 11:51 AM

Change #1138718 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] openstack: networktests: cleanup and simplify

https://gerrit.wikimedia.org/r/1138718

gerritbot added a comment.Apr 24 2025, 1:00 PM

Change #1138743 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloudgw: cleanup pre-IPv6 settings

https://gerrit.wikimedia.org/r/1138743

gerritbot added a comment.Apr 24 2025, 1:36 PM

Change #1138342 merged by Majavah:

[operations/puppet@production] P:dns: Update discovery-map for new WMCS addresses

https://gerrit.wikimedia.org/r/1138342

gerritbot added a comment.Apr 25 2025, 11:55 AM

Change #1138743 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloudgw: cleanup pre-IPv6 settings

https://gerrit.wikimedia.org/r/1138743

Maintenance_bot removed a project: Patch-For-Review.Apr 25 2025, 12:30 PM
jberkel unsubscribed.Apr 25 2025, 1:08 PM
taavi closed subtask T386689: Add new WMCS IPv6 ranges to MediaWiki configuration where required as Resolved.Apr 28 2025, 8:49 AM
Stang unsubscribed.May 2 2025, 4:33 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits