Page MenuHomePhabricator
Create Task
Maniphest T338125

cloudvirt: connect them to cloud-private
Closed, ResolvedPublic
Actions

Description

The cloudvirt hypervisors, with nova-compute and friends, need to talk to rabbitmq and other services over the cloud-private subnet.

This task is to track the work to connect cloudvirts to cloud-private.

The 3 of them are racked in the right rack, connected to the right switch and ready to go, barring configuring the new VLAN 2151 and adding the puppet profile.

Details

SubjectRepoBranchLines +/-
openstack: rabbitmq: add rule for cloud-privateoperations/puppetproduction+8 -0
Revert "Revert "openstack: rabbitmq: simplify cloud-private-subnet firewalling support""operations/puppetproduction+20 -6
openstack: rabbitmq: simplify cloud-private-subnet firewalling supportoperations/puppetproduction+20 -6
cloudvirt: codfw1dev: add cloud_private_subnetoperations/puppetproduction+1 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

aborrero triaged this task as High priority.Jun 5 2023, 9:51 AM
aborrero created this task.
aborrero moved this task from Backlog to Doing on the User-aborrero board.
gerritbot added a comment.Jun 5 2023, 9:55 AM

Change 927131 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloudvirt: codfw1dev: add cloud_private_subnet

https://gerrit.wikimedia.org/r/927131

gerritbot added a project: Patch-For-Review.Jun 5 2023, 9:55 AM
aborrero updated the task description. (Show Details)Jun 5 2023, 9:59 AM
aborrero updated the task description. (Show Details)
gerritbot added a comment.Jun 5 2023, 11:00 AM

Change 927140 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] openstack: rabbitmq: simplify cloud-private-subnet firewalling support

https://gerrit.wikimedia.org/r/927140

aborrero mentioned this in T336963: cloudcontrol2001-dev can't reach cloud-vps public IPs.Jun 5 2023, 12:37 PM
cmooney added a comment.Jun 6 2023, 9:05 AM

@aborrero FYI I've updated the config on cloudsw to trunk vlan2151 to the cloudvirts now.

gerritbot added a comment.Jun 6 2023, 10:17 AM

Change 927131 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloudvirt: codfw1dev: add cloud_private_subnet

https://gerrit.wikimedia.org/r/927131

aborrero added a comment.Jun 6 2023, 10:29 AM

running the netbox import script failed:

image.png (1×1 px, 224 KB)

aborrero added a comment.Jun 6 2023, 10:37 AM

But it worked just fine with the other 2 servers:

image.png (1×1 px, 194 KB)

aborrero added a comment.Jun 6 2023, 11:47 AM
In T338125#8905613, @aborrero wrote:

running the netbox import script failed:

image.png (1×1 px, 224 KB)

Fixed. There were 2 IP addresses on netbox for that host.

aborrero updated the task description. (Show Details)Jun 6 2023, 11:48 AM

The network is now setup correctly:

aborrero@cumin2002:~ $ sudo cumin cloudvirt2* 'ip route get 185.15.57.24 ; ping -c1 185.15.57.24'
3 hosts will be targeted:
cloudvirt[2001-2003]-dev.codfw.wmnet
OK to proceed on 3 hosts? Enter the number of affected hosts to confirm or "q" to quit: 3
===== NODE GROUP =====                                                                                                                        
(1) cloudvirt2001-dev.codfw.wmnet                                                                                                             
----- OUTPUT of 'ip route get 185...-c1 185.15.57.24' -----                                                                                   
185.15.57.24 via 172.20.5.1 dev vlan2151 src 172.20.5.11 uid 0                                                                                
    cache                                                                                                                                     
PING 185.15.57.24 (185.15.57.24) 56(84) bytes of data.
64 bytes from 185.15.57.24: icmp_seq=1 ttl=63 time=0.214 ms

--- 185.15.57.24 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.214/0.214/0.214/0.000 ms
===== NODE GROUP =====                                                                                                                        
(1) cloudvirt2002-dev.codfw.wmnet                                                                                                             
----- OUTPUT of 'ip route get 185...-c1 185.15.57.24' -----                                                                                   
185.15.57.24 via 172.20.5.1 dev vlan2151 src 172.20.5.12 uid 0                                                                                
    cache                                                                                                                                     
PING 185.15.57.24 (185.15.57.24) 56(84) bytes of data.
64 bytes from 185.15.57.24: icmp_seq=1 ttl=63 time=0.225 ms

--- 185.15.57.24 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.225/0.225/0.225/0.000 ms
===== NODE GROUP =====                                                                                                                        
(1) cloudvirt2003-dev.codfw.wmnet                                                                                                             
----- OUTPUT of 'ip route get 185...-c1 185.15.57.24' -----                                                                                   
185.15.57.24 via 172.20.5.1 dev vlan2151 src 172.20.5.13 uid 0                                                                                
    cache                                                                                                                                     
PING 185.15.57.24 (185.15.57.24) 56(84) bytes of data.
64 bytes from 185.15.57.24: icmp_seq=1 ttl=63 time=0.218 ms

--- 185.15.57.24 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.218/0.218/0.218/0.000 ms
================                                                                                                                              
PASS |████████████████████████████████████████████████████████████████████████████████████████████████| 100% (3/3) [00:00<00:00,  4.28hosts/s]
FAIL |                                                                                                        |   0% (0/3) [00:00<?, ?hosts/s]
100.0% (3/3) success ratio (>= 100.0% threshold) for command: 'ip route get 185...-c1 185.15.57.24'.
100.0% (3/3) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.

All that remains is to merge this:

https://gerrit.wikimedia.org/r/c/operations/puppet/+/927140/

gerritbot added a comment.Jun 6 2023, 12:03 PM

Change 927140 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] openstack: rabbitmq: simplify cloud-private-subnet firewalling support

https://gerrit.wikimedia.org/r/927140

Maintenance_bot removed a project: Patch-For-Review.Jun 6 2023, 12:11 PM
aborrero closed this task as Resolved.Jun 6 2023, 12:45 PM

Basic connectivity seems to be working.

Closing this task and tracking the remaining bits on the parent task.

gerritbot added a comment.Jun 7 2023, 9:40 AM

Change 927699 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] Revert "Revert "openstack: rabbitmq: simplify cloud-private-subnet firewalling support""

https://gerrit.wikimedia.org/r/927699

gerritbot added a project: Patch-For-Review.Jun 7 2023, 9:40 AM
gerritbot added a comment.Jun 7 2023, 11:39 AM

Change 928009 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] openstack: rabbitmq: add rule for cloud-private

https://gerrit.wikimedia.org/r/928009

gerritbot added a comment.Jun 7 2023, 11:45 AM

Change 927699 abandoned by Arturo Borrero Gonzalez:

[operations/puppet@production] Revert "Revert "openstack: rabbitmq: simplify cloud-private-subnet firewalling support""

Reason:

Merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/928009/ instead

https://gerrit.wikimedia.org/r/927699

gerritbot added a comment.Jun 7 2023, 11:53 AM

Change 928009 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] openstack: rabbitmq: add rule for cloud-private

https://gerrit.wikimedia.org/r/928009

Maintenance_bot removed a project: Patch-For-Review.Jun 7 2023, 12:11 PM
aborrero mentioned this in T338314: cloudnet: connect them to cloud-private.Jun 7 2023, 12:13 PM
aborrero mentioned this in T346651: cloudvirt: eqiad1: connect them to cloud-private.Sep 18 2023, 3:25 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits