Page MenuHomePhabricator
Create Task
Maniphest T379175

Enable IPv6 for the Cloud VPS web proxy
Closed, ResolvedPublic
Actions

Description

Enable IPv6 support for the Cloud VPS web proxy. And since this requires re-creating the VMs anyway, upgrade them to bookworm.

TODO:

  • Prepare Puppet code, test in codfw1dev
  • Provision pair of new proxies in dualstack network
  • Provision IPv4/IPv6 VIPs for Keepalived usage in the new network
  • Add new proxies to cache_hosts in Hiera
  • Merge Puppet patches, start provisioning AAAA records for new proxies
  • Add AAAA records to initial testers (WMCS-managed infrastructure + others relatively high-traffic proxies with active maintainers)
  • Contact projects that get XFF data to update filters on their side (I think this mostly includes ACC and UTRS)
  • Backfill security group rules to existing projects (P75871)
  • Flip floating IPv4 to new cluster
  • Backfill AAAA records
    • wmflabs.org
    • wmcloud.org
    • Everything else
    • Special cases
  • Cleanup old proxy instances

Details

Related Changes in Gerrit:
Show related patches Customize query in gerrit
Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
eqiad1: project-proxy: Add IPv6 security group rulesrepos/cloud/cloud-vps/tofu-infra!225taavitaavi/proxymain
eqiad1: project-proxy: Reserve main proxy VIP portrepos/cloud/cloud-vps/tofu-infra!224taavitaavi/proxymain
proxy-codfw1dev: Permit IPv6 web proxy trafficrepos/cloud/cloud-vps/tofu-infra!130taavitaavi/proxy-ipv6-ingressmain
Customize query in GitLab

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenNoneT404466 Support IPv6-only VMs
 OpenNoneT392688 Enable IPv6 on Cloud VPS infrastructure services
 ResolvedtaaviT379175 Enable IPv6 for the Cloud VPS web proxy
 Resolved• aborreroT37947 Enable IPv6 on CloudVPS
 Resolved• aborreroT245495 CloudVPS: IPv6 in codfw1dev
 Resolved• aborreroT187929 Cloud IPv6 subnets
 Resolved• aborreroT374712 netbox: create IPv6 entries for Cloud VPS
 Resolved• aborreroT376462 dns: integrate PTR support for 2a02:ec80:a100::/48
 Resolved• aborreroT374715 openstack: work out IPv6 and designate integration
 Resolved• aborreroT377740 neutron: clarify why DNS extension is not enabled
 Resolved• aborreroT378192 openstack: wmf sink: extend it to support IPv6
 Resolved• aborreroT378995 openstack codfw1dev: API endpoints errors 2024-11-04
 Resolved• aborreroT374714 openstack: clarify IPv6 firewalling
 Resolved• aborreroT377339 horizon: enable IPv6 security group panels
 Resolved• aborreroT374716 cloudgw: add support and enable IPv6
 ResolvedcmooneyT374713 cloudsw: codfw: enable IPv6
 Resolved• aborreroT376879 keepalived: it doesn't support mixing IPv4 and IPv6 VIPs on the same VRRP instance
 Resolved• aborreroT375847 openstack: initial IPv6 support in neutron
 Resolved• aborreroT377467 Decision Request - How to do the Cloud VPS VXLAN/IPv6 migration
 Resolved• aborreroT379356 openstack: nova-fullstack: add support for IPv6
 Resolved• aborreroT380208 openstack: codfw1dev: fullstack tests failing
 Resolved• aborreroT380303 Openstack: many orphaned (or seemingly orphaned) VMs in codfw1dev
 Resolved• aborreroT380054 Cloud VPS: prepare documentation on VXLAN/IPV6 migration
 Resolved• aborreroT380081 horizon: enable the UI to select networks on VM creation panel
 Resolved• aborreroT380174 CloudVPS: IPv6 in eqiad1
 Resolved• aborreroT380703 versions.toolforge.org is down
 Resolved• aborreroT380728 openstack: network problems when introducing new networks
 ResolvedcmooneyT389958 WMCS Eqiad: Enable IPv6 in cloud vrf on switches
 Resolved• aborreroT390877 HAProxyServiceUnavailable
Resolved• aborreroT391043 CephClusterInError #page Ceph cluster in eqiad is in error status
 Resolved• aborreroT391040 CloudVirtDown Cloudvirt node cloudvirt1063 is down. #page
 Resolved• aborreroT391039 CephSlowOps Ceph cluster in eqiad has 486 slow ops
 Resolved• aborreroT391325 openstack: improve networktests for newer network setup
 ResolvedfnegriT391467 gitlab ci: validate secrets settings in pipeline for tofu integration
 ResolvedcmooneyT380746 dns: add PTR support for 2a02:ec80:a000::
 ResolvedtaaviT386689 Add new WMCS IPv6 ranges to MediaWiki configuration where required
 ResolvedcmooneyT388379 ProbeDown
Resolved• aborreroT389942 openstack: rename lan-flat-cloudinstances2b to VLAN/legacy
 ResolvedJAllemandouT392468 Add new WMCS IP ranges to analytics
 ResolvedtaaviT392570 metricsinfra: Support scraping v6-enabled instances
 ResolvedtaaviT392611 VMs with ferm host-level firewall do not permit DHCPv6 responses
 ResolvedtaaviT380057 Keepalived Puppet module: Support IPv6
 ResolvedMarosteguiT380073 wmf-mariadb client package for bookworm
 ResolvedBUG REPORTtaaviT393024 lua entry thread aborted: runtime error: /etc/nginx/lua/domainproxy.lua:32: bad request

Event Timeline

taavi created this task.Nov 6 2024, 4:56 PM
Restricted Application added a project: cloud-services-team. · View Herald TranscriptNov 6 2024, 4:56 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
taavi added a subtask: T37947: Enable IPv6 on CloudVPS.Nov 6 2024, 4:57 PM
gerritbot added a comment.Nov 7 2024, 4:44 PM

Change #1088338 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] dynamicproxy: Provision AAAA records

https://gerrit.wikimedia.org/r/1088338

gerritbot added a project: Patch-For-Review.Nov 7 2024, 4:44 PM

Change #1088339 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] dynamicproxy: Canocalize IP addresses before comparing

https://gerrit.wikimedia.org/r/1088339

taavi added a project: IPv6.Nov 10 2024, 7:33 AM
fnegri triaged this task as Medium priority.Nov 11 2024, 11:45 AM
gerritbot added a comment.Nov 15 2024, 2:14 PM

Change #1091733 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] keepalived::failover: Support IPv6

https://gerrit.wikimedia.org/r/1091733

taavi updated the task description. (Show Details)Nov 15 2024, 5:29 PM
gerritbot added a comment.Nov 15 2024, 6:39 PM

Change #1091796 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] dynamicproxy: Fix Lua path on bookworm

https://gerrit.wikimedia.org/r/1091796

gerritbot added a comment.Nov 15 2024, 6:42 PM

Change #1091796 merged by Majavah:

[operations/puppet@production] dynamicproxy: Fix Lua path on bookworm

https://gerrit.wikimedia.org/r/1091796

gerritbot added a comment.Nov 15 2024, 6:47 PM

Change #1091798 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] hieradata: Update codfw1dev cache hosts

https://gerrit.wikimedia.org/r/1091798

CodeReviewBot added a comment.Nov 15 2024, 7:07 PM

taavi opened https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/130

proxy-codfw1dev: Permit IPv6 web proxy traffic

CodeReviewBot added a comment.Nov 15 2024, 7:09 PM

taavi merged https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/130

proxy-codfw1dev: Permit IPv6 web proxy traffic

gerritbot added a comment.Nov 15 2024, 7:12 PM

Change #1091802 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] dynamicproxy: Listen on IPv6

https://gerrit.wikimedia.org/r/1091802

taavi claimed this task.Nov 15 2024, 7:38 PM
gerritbot added a comment.Nov 15 2024, 8:47 PM

Change #1091798 merged by Majavah:

[operations/puppet@production] hieradata: Update codfw1dev cache hosts

https://gerrit.wikimedia.org/r/1091798

gerritbot added a comment.Nov 16 2024, 7:54 AM

Change #1091848 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] dynamicproxy: Run Redis update in app context

https://gerrit.wikimedia.org/r/1091848

gerritbot added a comment.Nov 16 2024, 8:16 AM

Change #1091849 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] dynamicproxy: Bind Redis on IPv6

https://gerrit.wikimedia.org/r/1091849

taavi closed subtask T380057: Keepalived Puppet module: Support IPv6 as Resolved.Nov 22 2024, 4:26 PM
gerritbot added a comment.Nov 26 2024, 3:34 PM

Change #1091849 merged by Majavah:

[operations/puppet@production] dynamicproxy: Bind Redis on IPv6

https://gerrit.wikimedia.org/r/1091849

gerritbot added a comment.Nov 27 2024, 3:56 PM

Change #1091802 merged by Majavah:

[operations/puppet@production] dynamicproxy: Listen on IPv6

https://gerrit.wikimedia.org/r/1091802

gerritbot added a comment.Nov 27 2024, 4:04 PM

Change #1091848 merged by Majavah:

[operations/puppet@production] dynamicproxy: Run Redis update in app context

https://gerrit.wikimedia.org/r/1091848

gerritbot added a comment.Nov 27 2024, 4:16 PM

Change #1088339 merged by Majavah:

[operations/puppet@production] dynamicproxy: Canocalize IP addresses before comparing

https://gerrit.wikimedia.org/r/1088339

Marostegui closed subtask T380073: wmf-mariadb client package for bookworm as Resolved.Dec 10 2024, 3:24 PM
taavi added a comment.Apr 23 2025, 9:19 AM

One thing to keep in mind here is that many security groups reference 172.16.0.0/21 directly. I've updated the docs now but there are still old projects with old rules.

• aborrero closed subtask T37947: Enable IPv6 on CloudVPS as Resolved.Apr 24 2025, 8:54 AM
stwalkerster subscribed.Apr 24 2025, 9:11 AM
taavi updated the task description. (Show Details)Apr 24 2025, 1:05 PM
taavi updated the task description. (Show Details)Apr 24 2025, 1:44 PM
taavi added a parent task: T392688: Enable IPv6 on Cloud VPS infrastructure services.Apr 25 2025, 1:52 PM
XtexChooser subscribed.Apr 26 2025, 12:42 AM
CodeReviewBot added a comment.Apr 29 2025, 10:51 AM

taavi opened https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224

eqiad1: project-proxy: Reserve main proxy VIP port

Stashbot added a comment.Apr 29 2025, 10:54 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-04-29T10:54:33Z] <taavi@cloudcumin1001> START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'proxy' (T379175)

Stashbot added a comment.Apr 29 2025, 11:26 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-04-29T11:26:08Z] <taavi@cloudcumin1001> START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'proxy' (T379175)

Stashbot added a comment.Apr 29 2025, 11:26 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-04-29T11:26:27Z] <taavi@cloudcumin1001> END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'proxy' (T379175)

Stashbot added a comment.Apr 29 2025, 11:28 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-04-29T11:28:36Z] <taavi@cloudcumin1001> START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'proxy' (T379175)

gerritbot added a comment.Apr 29 2025, 11:33 AM

Change #1139818 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] hieradata: Add new eqiad1 proxies

https://gerrit.wikimedia.org/r/1139818

CodeReviewBot added a comment.Apr 29 2025, 11:47 AM

taavi merged https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224

eqiad1: project-proxy: Reserve main proxy VIP port

gerritbot added a comment.Apr 29 2025, 11:56 AM

Change #1139821 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:wmcs: novaproxy: Add separate keepalived_peers variable

https://gerrit.wikimedia.org/r/1139821

gerritbot added a comment.Apr 29 2025, 12:18 PM

Change #1139818 merged by Majavah:

[operations/puppet@production] hieradata: Add new eqiad1 proxies

https://gerrit.wikimedia.org/r/1139818

gerritbot added a comment.Apr 29 2025, 12:18 PM

Change #1139821 merged by Majavah:

[operations/puppet@production] P:wmcs: novaproxy: Add separate keepalived_peers variable

https://gerrit.wikimedia.org/r/1139821

CodeReviewBot added a comment.Apr 29 2025, 12:47 PM

taavi opened https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/225

eqiad1: project-proxy: Add IPv6 security group rules

taavi updated the task description. (Show Details)Apr 29 2025, 12:48 PM
CodeReviewBot added a comment.Apr 29 2025, 1:00 PM

taavi merged https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/225

eqiad1: project-proxy: Add IPv6 security group rules

gerritbot added a comment.Apr 29 2025, 1:29 PM

Change #1139857 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] keepalived: Fix IPv6 support

https://gerrit.wikimedia.org/r/1139857

gerritbot added a comment.Apr 29 2025, 1:38 PM

Change #1139857 merged by Majavah:

[operations/puppet@production] keepalived: Fix IPv6 support

https://gerrit.wikimedia.org/r/1139857

gerritbot added a comment.Apr 29 2025, 2:54 PM

Change #1139877 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] keepalived: failover: Select unicast source v6 more reliably

https://gerrit.wikimedia.org/r/1139877

gerritbot added a comment.Apr 29 2025, 4:32 PM

Change #1139877 merged by Majavah:

[operations/puppet@production] keepalived: failover: Select unicast source v6 more reliably

https://gerrit.wikimedia.org/r/1139877

Stashbot added a comment.Apr 29 2025, 4:43 PM

Mentioned in SAL (#wikimedia-cloud) [2025-04-29T16:43:14Z] <taavi> add AAAA record to codesearch.wmcloud.org. as a first test for T379175

gerritbot added a comment.Apr 30 2025, 8:14 AM

Change #1140125 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] dynamicproxy: Use lua-resty-redis from Debian package

https://gerrit.wikimedia.org/r/1140125

gerritbot added a comment.Apr 30 2025, 8:27 AM

Change #1140125 merged by Majavah:

[operations/puppet@production] dynamicproxy: Use lua-resty-redis from Debian package

https://gerrit.wikimedia.org/r/1140125

Stashbot added a comment.Apr 30 2025, 8:44 AM

Mentioned in SAL (#wikimedia-cloud) [2025-04-30T08:44:00Z] <taavi> add AAAA record to *.wmflabs.org (for redirects for proxies that have migrated to wmcloud.org) T379175

gerritbot added a comment.Apr 30 2025, 8:50 AM

Change #1088338 merged by Majavah:

[operations/puppet@production] dynamicproxy: Provision AAAA records

https://gerrit.wikimedia.org/r/1088338

taavi updated the task description. (Show Details)Apr 30 2025, 8:57 AM
taavi added a subtask: T393024: lua entry thread aborted: runtime error: /etc/nginx/lua/domainproxy.lua:32: bad request.Apr 30 2025, 3:50 PM
taavi mentioned this in T393024: lua entry thread aborted: runtime error: /etc/nginx/lua/domainproxy.lua:32: bad request.Apr 30 2025, 3:54 PM
taavi updated the task description. (Show Details)May 6 2025, 11:35 AM
taavi updated the task description. (Show Details)May 6 2025, 4:47 PM
• Pastykros subscribed.May 6 2025, 7:36 PM
This comment was removed by taavi.
taavi closed subtask T393024: lua entry thread aborted: runtime error: /etc/nginx/lua/domainproxy.lua:32: bad request as Resolved.May 7 2025, 8:02 AM
taavi updated the task description. (Show Details)May 7 2025, 9:27 AM
Stashbot added a comment.May 7 2025, 12:48 PM

Mentioned in SAL (#wikimedia-cloud) [2025-05-07T12:48:37Z] <taavi> updating all security group rules referencing old 172.16.0.0/21 subnet to reference new ip space instead (T379175)

taavi updated the task description. (Show Details)May 7 2025, 2:03 PM
taavi added a comment.May 7 2025, 5:27 PM

Noting for the records that the above-mentioned security group update resulted in https://wikitech.wikimedia.org/wiki/Incidents/2025-05-07_cloud-vps_security_groups_deleted.

I'm hoping to flip the floating IP (+ the Redis primary) to the new cluster tomorrow, and then look at backfilling AAAA records.

taavi updated the task description. (Show Details)May 8 2025, 8:09 AM
Stashbot added a comment.May 8 2025, 8:23 AM

Mentioned in SAL (#wikimedia-cloud) [2025-05-08T08:22:57Z] <taavi> migrating 185.15.56.49 floating IP to the new proxy cluster T379175

taavi updated the task description. (Show Details)May 8 2025, 8:27 AM
Stashbot added a comment.May 8 2025, 8:36 AM

Mentioned in SAL (#wikimedia-cloud) [2025-05-08T08:36:15Z] <taavi> flip redis/API primary status to proxy-5 T379175

Stashbot added a comment.May 8 2025, 10:53 AM

Mentioned in SAL (#wikimedia-cloud) [2025-05-08T10:53:43Z] <taavi> backfilling AAAA records for web proxies using wmflabs.org T379175

taavi updated the task description. (Show Details)May 8 2025, 10:56 AM
taavi updated the task description. (Show Details)
Stashbot added a comment.May 8 2025, 11:32 AM

Mentioned in SAL (#wikimedia-cloud) [2025-05-08T11:31:59Z] <taavi> backfilling AAAA records for web proxies using wmcloud.org T379175

taavi updated the task description. (Show Details)May 8 2025, 11:32 AM
taavi updated the task description. (Show Details)May 8 2025, 1:03 PM
Stashbot added a comment.Jun 10 2025, 9:11 AM

Mentioned in SAL (#wikimedia-cloud) [2025-06-10T09:11:51Z] <taavi> delete old proxy-03,04 instances T379175

gerritbot added a comment.Jun 10 2025, 9:15 AM

Change #1155149 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] hieradata: Remove old Cloud VPS proxies

https://gerrit.wikimedia.org/r/1155149

gerritbot added a comment.Jun 10 2025, 9:20 AM

Change #1155149 merged by Majavah:

[operations/puppet@production] hieradata: Remove old Cloud VPS proxies

https://gerrit.wikimedia.org/r/1155149

taavi added a comment.Jun 10 2025, 12:17 PM

Closing, as T379283 meant I could fix the NFS edge cases and the remaining one (UTRS) is tracked elsewhere.

taavi closed this task as Resolved.Jun 13 2025, 3:22 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits