openstack: wmf sink: extend it to support IPv6
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• aborrero
	Oct 25 2024, 3:19 PM

Description

We need to extend wmf sink to support IPv6.

Details

Related Changes in Gerrit:

Subject	Repo	Branch	Lines +/-
openstack: designate: Remove nova_fixed_multi code	operations/puppet	production	+7 -709
openstack: designate: deploy wmcs_nova_fixed_ptr	operations/puppet	production	+91 -16
openstack: designate: nova_fixed_multi: base: refactor record creation routine	operations/puppet	production	+44 -58

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Open	None	T53494 Use Beta cluster as a true canary for code deployments (epic)
Declined	None	T87220 Minimize infrastructure differences between Beta Cluster and production
Open	None	T211677 Support IPv6 in beta
Open	None	T404466 Support IPv6-only VMs
Open	None	T392688 Enable IPv6 on Cloud VPS infrastructure services
Resolved	taavi	T379175 Enable IPv6 for the Cloud VPS web proxy
Open	None	T270694 CloudVPS: introduce tenant networks
Resolved	• aborrero	T364725 Migrate Cloud VPS instances to VXLAN based networks
Open	None	T392509 Enable IPv6 for Toolforge services
Resolved	taavi	T392506 Enable IPv6 for tools.wmflabs.org / *.toolserver.org legacy redirector service
Resolved	taavi	T211575 Enable IPv6 on toolforge.org
Resolved	None	T209460 CloudVPS: network architecture
Resolved	None	T244727 CloudVPS: networking improvements
Open	None	T220306 Add IPv6 monitoring
Resolved	• aborrero	T37947 Enable IPv6 on CloudVPS
Resolved	• aborrero	T245495 CloudVPS: IPv6 in codfw1dev
Resolved	• aborrero	T187929 Cloud IPv6 subnets
Resolved	• aborrero	T374712 netbox: create IPv6 entries for Cloud VPS
Resolved	• aborrero	T376462 dns: integrate PTR support for 2a02:ec80:a100::/48
Resolved	• aborrero	T374715 openstack: work out IPv6 and designate integration
Resolved	• aborrero	T378192 openstack: wmf sink: extend it to support IPv6
Resolved	• aborrero	T378995 openstack codfw1dev: API endpoints errors 2024-11-04

Event Timeline

• aborrero created this task.Oct 25 2024, 3:19 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 25 2024, 3:19 PM

• aborrero triaged this task as Medium priority.Oct 25 2024, 3:23 PM

• aborrero updated the task description. (Show Details)

• aborrero removed projects: Infrastructure-Foundations, SRE, netops.

Change #1083820 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] openstack: designate: nova_fixed_multi: base: refactor record creation routine

https://gerrit.wikimedia.org/r/1083820

gerritbot added a project: Patch-For-Review.Oct 28 2024, 2:08 PM

• aborrero moved this task from Backlog to Doing on the User-aborrero board.Oct 29 2024, 9:20 AM

the native upstream designate notification handler code does a good chunk of the things we do in our custom code.

In particular:

wmf nova_fixed_multi -> base -> _create:
- create A records in a given zone
- create PTR v4 record in a given zone

designate nova_fixed -> base -> _create:
- create A records in a given zone
- create AAAA records in a given zone

The only missing bit for what we do in the upstream designate handler is PTR creation.

We may benefit from adopting the upstream code for A/AAAA record creation, and reducing the scope of our own custom handler to PTR creation.

The actions:

drop wmf nova_fixed_multi (and base)
adopt upstream designate nova_fixed handler
create a new handler to work with PTR records

Change #1083820 abandoned by Arturo Borrero Gonzalez:

[operations/puppet@production] openstack: designate: nova_fixed_multi: base: refactor record creation routine

Reason:

taking a different route

https://gerrit.wikimedia.org/r/1083820

Maintenance_bot removed a project: Patch-For-Review.Oct 29 2024, 11:30 AM

I created this repository to host the new code: https://gitlab.wikimedia.org/repos/cloud/cloud-vps/designate-sink-nova-fixed-ptr

I'm now testing the code.

I disabled puppet on cloudlb2001-dev to force nova and designate request to go to cloudcontrol2004-dev
I disabled puppet on cloudcontrol2004-dev, edited /etc/designate/designate.conf with content:

[service:sink]
# List of notification handlers to enable, configuration of these needs to
# correspond to a [handler:my_driver] section below or else in the config
# Can be one or more of : nova_fixed, neutron_floatingip
enabled_notification_handlers = nova_fixed, wmf_sink, wmcs_nova_fixed_ptr 

[handler:nova_fixed]
zone_id = 4c754100-1790-4858-a583-9de93c9e8b3d

[handler:wmcs_nova_fixed_ptr]
# handler-specific configuration
domain_name = eqiad1.wikimedia.cloud 
ptr_v4_zone_id = 187fdc06-c5d2-46e1-ab71-97d34dd067ce
ptr_v6_zone_id = 7a2210a4-3a55-48d8-9713-0256d7d9bc1b

installed https://gitlab.wikimedia.org/repos/cloud/cloud-vps/designate-sink-wmcs-nova-fixed-ptr manually with

git clone https://gitlab.wikimedia.org/repos/cloud/cloud-vps/designate-sink-wmcs-nova-fixed-ptr
cd designate-sink-wmcs-nova-fixed-ptr
sudo cp -r wmcs_nova_fixed_ptr /usr/local/lib/python3.11/dist-packages/

restarted designate and nova

Mentioned in SAL (#wikimedia-cloud) [2024-11-04T13:52:29Z] <arturo> [codfw1dev] live-hack cloudlb2001-dev and cloudcontrol2004-dev for T378192

• aborrero mentioned this in T378995: openstack codfw1dev: API endpoints errors 2024-11-04.Nov 4 2024, 4:31 PM

• aborrero added a subtask: T378995: openstack codfw1dev: API endpoints errors 2024-11-04.Nov 4 2024, 4:33 PM

I can't make the service work as expected. It seems designate-sink refuses to call the handlers, with this config:

[service:sink]
enabled_notification_handlers = nova, wmf_sink, wmcs_nova_fixed_ptr

[handler:nova_fixed]
notification_topics = monitor
notification_topics = notifications
control_exchange = nova

[handler:wmcs_nova_fixed_ptr]
domain_name = eqiad1.wikimedia.cloud
ptr_v4_zone_id = 187fdc06-c5d2-46e1-ab71-97d34dd067ce
ptr_v6_zone_id = 7a2210a4-3a55-48d8-9713-0256d7d9bc1b
notification_topics = monitor
notification_topics = notifications
control_exchange = nova

I also tried with enabled_notification_handlers = nova_fixed, wmf_sink, wmcs_nova_fixed_ptr

Mentioned in SAL (#wikimedia-cloud) [2024-11-05T10:24:09Z] <arturo> [codfw1dev] disable puppet and make changes for testing T378192

Some good progress today. I was able to get designate-sink to run the new code.

The config in /etc/designate/designate.conf:

[service:sink]
enabled_notification_handlers = nova_fixed, wmcs_nova_fixed_ptr, wmf_sink

[handler:nova_fixed]
notification_topics = monitor
notification_topics = notifications
control_exchange = nova
zone_id = 4c754100-1790-4858-a583-9de93c9e8b3d
formatv4 = '%(hostname)s.%(project)s.%(zone)s'
formatv6 = '%(hostname)s.%(project)s.%(zone)s'

[handler:wmcs_nova_fixed_ptr]
domain_name = codfw1dev.wikimedia.cloud
ptr_v4_zone_id = 187fdc06-c5d2-46e1-ab71-97d34dd067ce
ptr_v6_zone_id = 7a2210a4-3a55-48d8-9713-0256d7d9bc1b
notification_topics = monitor
notification_topics = notifications
control_exchange = nova

To deploy the code, I did the following:

user@cloudcontrol:~$ cat /usr/lib/python3/dist-packages/wmcs_nova_fixed_ptr.egg-info/entry_points.txt
[designate.notification.handler]
wmcs_nova_fixed_ptr = wmcs_nova_fixed_ptr.wmcs_nova_fixed_ptr:WMCSNovaFixedPtrHandler

Then:

user@cloudcontrol:~$ git clone https://gitlab.wikimedia.org/repos/cloud/cloud-vps/designate-sink-wmcs-nova-fixed-ptr
user@cloudcontrol:~$ cd designate-sink-wmcs-nova-fixed-ptr
user@cloudcontrol:~$ sudo cp -r wmcs_nova_fixed_ptr /usr/local/lib/python3.11/dist-packages/

Then restart designate-sink:

user@cloudcontrol:~$ systemctl restart designate-sink

seems to be working as expected:

aborrero@bastion-codfw1dev-04:~$ host fullstackd-20241105162427.admin-monitoring.codfw1dev.wikimedia.cloud
fullstackd-20241105162427.admin-monitoring.codfw1dev.wikimedia.cloud has address 172.16.129.162
fullstackd-20241105162427.admin-monitoring.codfw1dev.wikimedia.cloud has IPv6 address 2a02:ec80:a100:1::2a1
aborrero@bastion-codfw1dev-04:~$ host 172.16.129.162
162.129.16.172.in-addr.arpa domain name pointer fullstackd-20241105162427.admin-monitoring.codfw1dev.wikimedia.cloud.
aborrero@bastion-codfw1dev-04:~$ host 2a02:ec80:a100:1::2a1
1.a.2.0.0.0.0.0.0.0.0.0.0.0.0.0.1.0.0.0.0.0.1.a.0.8.c.e.2.0.a.2.ip6.arpa domain name pointer fullstackd-20241105162427.admin-monitoring.codfw1dev.wikimedia.cloud.

Change #1087521 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] openstack: designate: deploy and enable wmcs-nova-fixed-ptr

https://gerrit.wikimedia.org/r/1087521

gerritbot added a project: Patch-For-Review.Nov 5 2024, 4:51 PM

Change #1087521 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] openstack: designate: deploy wmcs_nova_fixed_ptr

https://gerrit.wikimedia.org/r/1087521

code was deployed and enabled in codfw1dev. Tomorrow I'll check everything is working as expected before declaring victory.

Maintenance_bot removed a project: Patch-For-Review.Nov 7 2024, 4:31 PM

I saw several failures today. I was unable to create VMs by hand, because nova failures when scheduling VMs.

Also, some failures related to nova-fullstack T379356: openstack: nova-fullstack: add support for IPv6

At the end, I was unable to confirm the new plugin is working as expeted.

ok, after restarting a few more openstack things:

aborrero@bastion-codfw1dev-04:~$ host please-work.cloudinfra-codfw1dev.codfw1dev.wikimedia.cloud
please-work.cloudinfra-codfw1dev.codfw1dev.wikimedia.cloud has address 172.16.129.95
please-work.cloudinfra-codfw1dev.codfw1dev.wikimedia.cloud has IPv6 address 2a02:ec80:a100:1::353
aborrero@bastion-codfw1dev-04:~$ host 172.16.129.95
95.129.16.172.in-addr.arpa domain name pointer please-work.cloudinfra-codfw1dev.codfw1dev.wikimedia.cloud.
aborrero@bastion-codfw1dev-04:~$ host 2a02:ec80:a100:1::353
3.5.3.0.0.0.0.0.0.0.0.0.0.0.0.0.1.0.0.0.0.0.1.a.0.8.c.e.2.0.a.2.ip6.arpa domain name pointer please-work.cloudinfra-codfw1dev.codfw1dev.wikimedia.cloud.