To make T379282: IPv6 for cloud-realm services happen we need a possibility to add IPv6 connectivity to the cloud-private networks. I think we can slowly add actual addresses and DNS records for hosts as we need them for some service or refresh hosts instead of doing everything at once.
Description
Description
Details
Details
- Other Assignee
- cmooney
Related Changes in Gerrit:
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | taavi | T396448 Support IPv6 for Cloud VPS DNS services | |||
| Resolved | taavi | T396450 Support IPv6 for OpenStack API endpoints | |||
| Resolved | taavi | T396451 Support IPv6 for Wiki Replicas cloudlb endpoints | |||
| Resolved | taavi | T379282 IPv6 for cloud-realm services | |||
| Resolved | taavi | T379283 IPv6 support in cloud-private |
Event Timeline
Comment Actions
Hi @cmooney (and cc @aborrero) - For this and the parent task I need v6 subnets for the following:
- per-rack cloud-private subnets for hosts (v4 uses 172.20.x.0/24)
- public and private service ranges (v4 uses 172.20.255.0/24 and 185.15.56.160/28 in eqiad, and similar ones in codfw)
I skimmed through T187929: Cloud IPv6 subnets but couldn't see these mentioned specifically. Did you plan something for these already or do we need to figure them out here?
Comment Actions
Change #1088341 had a related patch set uploaded (by Majavah; author: Majavah):
[operations/puppet@production] P:wmcs::cloud_private_subnet: Add IPv6 support
Comment Actions
Change #1088341 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] P:wmcs::cloud_private_subnet: Add IPv6 support
Comment Actions
My feeling is that https://netbox.wikimedia.org/ipam/prefixes/1089/ 2a02:ec80:a100:200::/56 which is wmcs cloud-private codfw (non-openstack) was allocated for this.
But I'm a bit lost in the sea of the new prefixes, and a confirmation by @cmooney should be needed here.
For eqiad, I don't think we have allocated that just yet.
Comment Actions
Apologies for the delay guys. Yes @aborrero that /56 was allocated for this purpose. I would suggest we allocate the per-rack subnets as follows, attached to the vlans and site in the same way as the v4 subnets:
| Vlan | Vlan name | IPv4 Subnet | New IPv6 Subnet |
|---|---|---|---|
| 1151 | cloud-private-c8-eqiad | 172.20.1.0/24 | 2a02:ec80:a000:0201::/64 |
| 1152 | cloud-private-d5-eqiad | 172.20.2.0/24 | 2a02:ec80:a000:0202::/64 |
| 1153 | cloud-private-e4-eqiad | 172.20.3.0/24 | 2a02:ec80:a000:0203::/64 |
| 1154 | cloud-private-f4-eqiad | 172.20.4.0/24 | 2a02:ec80:a000:0204::/64 |
And for codfw:
| Vlan | Vlan name | IPv4 Subnet | New IPv6 Subnet |
|---|---|---|---|
| 2151 | cloud-private-b1-codfw | 172.20.5.0/24 | 2a02:ec80:a100:205::/64 |
The hosts will need routes for new aggregate ranges, via the '::1' IP on their local subnet (i.e. same as they route 172.20.0.0/16 via 172.20.x.1).
eqiad: 2a02:ec80:a000:200::/56 codfw: 2a02:ec80:a100:200::/56
While we don't currently have a path between the networks in either site it's probably no harm to add both routes to all hosts, and if we do that in future (via GRE or IPsec or otherwise - indeed plain routing here may be an option) we won't have to revisit.
We may also eventually need to allocate some public IPv6 addressing, for similar use to 185.15.56.160/28 in v4, but I think we can probably deal with that later?
Are you ok to add these or should I? I will need to allocate an IP for the switch on each vlan anyway, and then look at updating the routing config on the cloud switches to enable BGP for IPv6.
Comment Actions
Thanks for working on this @cmooney. I agree with the allocations.
I would appreciate if you can add the netbox objects yourself. Otherwise I will try to do it myself next week.
Comment Actions
The allocations SGTM too, thanks!
I'd like to get that done sooner than later :-) although that probably should be discussed in T379282: IPv6 for cloud-realm services if we're doing that separately.
Comment Actions
Change #1109732 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):
[operations/dns@master] DNS Template include statements for new WMCS IPv6 ranges
Comment Actions
Change #1109732 merged by Cathal Mooney:
[operations/dns@master] DNS Template include statements for new WMCS IPv6 ranges
Comment Actions
Change #1109983 had a related patch set uploaded (by Majavah; author: Majavah):
[operations/puppet@production] hieradata: Add cloud-private v6 supernets
Comment Actions
Change #1109983 merged by Majavah:
[operations/puppet@production] hieradata: Add cloud-private v6 supernets
Comment Actions
Change #1126035 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):
[operations/homer/public@master] Add cloud IPv6 ranges to Capirca IP block definitions
Comment Actions
Change #1126035 merged by jenkins-bot:
[operations/homer/public@master] Add cloud IPv6 ranges to Capirca IP block definitions
Comment Actions
Change #1127526 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):
[operations/homer/public@master] Cloud-in: Add specific term allowing ICMPv6 from cloud-transports
Comment Actions
Change #1127526 merged by jenkins-bot:
[operations/homer/public@master] Cloud-in: Add specific term allowing ICMPv6 from cloud-transports
Comment Actions
Change #1130143 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):
[operations/dns@master] Add include statement for WMCS Eqiad reverse IPv6 snippet
Comment Actions
Change #1130143 merged by Cathal Mooney:
[operations/dns@master] Add include statement for WMCS Eqiad reverse IPv6 snippet
Comment Actions
Change #1130159 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):
[operations/dns@master] Add reverse zone for 172.31.0.0/16
Comment Actions
Change #1130159 merged by Cathal Mooney:
[operations/dns@master] Add reverse zone for 172.31.0.0/16
Comment Actions
@aborrero I made some progress on this today, but we're not quite there.
Unfortunately when I enabled OSPF on the cloud switches it caused some problems - see https://phabricator.wikimedia.org/T389672#10663716
So I'll need to go back and try to piece all that together before we can proceed here.
Comment Actions
Change #1134694 had a related patch set uploaded (by Majavah; author: Majavah):
[operations/puppet@production] network: Add v6 cloud-private addresses
Comment Actions
Change #1134694 merged by Majavah:
[operations/puppet@production] network: Add v6 cloud-private addresses
Comment Actions
Change #1136995 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):
[operations/homer/public@master] WMCS: Remove static routes for cloudsw2-d5-eqiad loopbacks
Comment Actions
Change #1136995 merged by jenkins-bot:
[operations/homer/public@master] WMCS: Remove static routes for cloudsw2-d5-eqiad loopbacks
Comment Actions
Change #1136996 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):
[operations/homer/public@master] WMCS: Change ASN for cloudsw1-e4/f4
Comment Actions
Change #1136996 merged by jenkins-bot:
[operations/homer/public@master] WMCS: Change ASN for cloudsw1-e4/f4
Comment Actions
Change #1145093 had a related patch set uploaded (by Majavah; author: Majavah):
[operations/puppet@production] P:openstack: keystone: Update ACLs for cloud-private v6
Comment Actions
Change #1145094 had a related patch set uploaded (by Majavah; author: Majavah):
[operations/puppet@production] P:openstack: rabbitmq: Add cloud-private v6 nets to firewall
Comment Actions
Change #1145093 merged by Majavah:
[operations/puppet@production] P:openstack: keystone: Update ACLs for cloud-private v6
Comment Actions
Change #1145094 merged by Majavah:
[operations/puppet@production] P:openstack: rabbitmq: Add cloud-private v6 nets to firewall
Comment Actions
Mentioned in SAL (#wikimedia-cloud) [2025-06-10T10:16:23Z] <taavi> add cloud-private addresses to eqiad hosts T379283