Page MenuHomePhabricator
Create Task
Maniphest T361537

connectivity from cloudbackup200[34] and eqiad ceph
Closed, ResolvedPublic
Actions

Description

In order to support offsite backups, these codfw cloudbackup hosts spend a lot of time talking to services in eqiad. They can talk to openstack in eqiad, but cannot talk to wmcs ceph.

Expected behavior (as present on cloudbackup200[12]):

andrew@cloudbackup2001:~$ hostname -i
10.192.0.130

andrew@cloudbackup2001:~$ telnet cloudcephmon1001.eqiad.wmnet 3300
Trying 2620:0:861:118:10:64:20:67...
Trying 10.64.20.67...
Connected to cloudcephmon1001.eqiad.wmnet.
Escape character is '^]'.
ceph v2

Actual behavior (on replacement hosts cloudbackup200[34]):

andrew@cloudbackup2003:~$ hostname -i
10.192.21.7

andrew@cloudbackup2003:~$ telnet cloudcephmon1001.eqiad.wmnet 3300
Trying 2620:0:861:118:10:64:20:67...
<times out>

This looks like a firewall issue, but does not seem to be; both originating IPs are present in ferm on the target:

root@cloudcephmon1003:/etc/ferm/conf.d# grep -r 10.192.48.34 10_ceph_mon_peers_v2 
&R_SERVICE(tcp, 3300, (10.192.0.130 10.192.20.0/24 10.192.21.7 10.192.32.186 10.192.48.34 10.64.148.0/24 10.64.148.14 10.64.148.15 10.64.148.16 10.64.148.2 10.64.148.3 10.64.148.4 10.64.148.5 10.64.148.6 10.64.149.0/24 10.64.149.14 10.64.149.15 10.64.149.16 10.64.149.2 10.64.149.3 10.64.149.4 10.64.149.5 10.64.149.6 10.64.150.0/24 10.64.151.0/24 10.64.20.0/24 10.64.20.13 10.64.20.14 10.64.20.15 10.64.20.16 10.64.20.17 10.64.20.18 10.64.20.19 10.64.20.20 10.64.20.21 10.64.20.57 10.64.20.58 10.64.20.59 10.64.20.60 10.64.20.62 10.64.20.63 10.64.20.64 10.64.20.65 10.64.20.66 10.64.20.67 10.64.20.68 10.64.20.69 2620:0:860:101:10:192:0:130 2620:0:860:103:10:192:32:186 2620:0:860:104:10:192:48:34 2620:0:860:111:10:192:21:7));

root@cloudcephmon1003:/etc/ferm/conf.d# grep -r 10.192.0.130 10_ceph_mon_peers_v2 
&R_SERVICE(tcp, 3300, (10.192.0.130 10.192.20.0/24 10.192.21.7 10.192.32.186 10.192.48.34 10.64.148.0/24 10.64.148.14 10.64.148.15 10.64.148.16 10.64.148.2 10.64.148.3 10.64.148.4 10.64.148.5 10.64.148.6 10.64.149.0/24 10.64.149.14 10.64.149.15 10.64.149.16 10.64.149.2 10.64.149.3 10.64.149.4 10.64.149.5 10.64.149.6 10.64.150.0/24 10.64.151.0/24 10.64.20.0/24 10.64.20.13 10.64.20.14 10.64.20.15 10.64.20.16 10.64.20.17 10.64.20.18 10.64.20.19 10.64.20.20 10.64.20.21 10.64.20.57 10.64.20.58 10.64.20.59 10.64.20.60 10.64.20.62 10.64.20.63 10.64.20.64 10.64.20.65 10.64.20.66 10.64.20.67 10.64.20.68 10.64.20.69 2620:0:860:101:10:192:0:130 2620:0:860:103:10:192:32:186 2620:0:860:104:10:192:48:34 2620:0:860:111:10:192:21:7));

Related Objects
Search...

Event Timeline

Andrew created this task.Apr 2 2024, 3:12 AM
Restricted Application edited projects, added: cloud-services-team; removed: Patch-For-Review, cloud-services-team (Hardware). · View Herald TranscriptApr 2 2024, 3:12 AM
Andrew updated the task description. (Show Details)Apr 2 2024, 3:15 AM
Andrew added a subscriber: cmooney.

@cmooney do you recall if we have special secret routing set up someplace to make this work for the old cloudbackup hosts?

taavi claimed this task.Apr 2 2024, 7:38 AM
taavi subscribed.

I ran the Capirca netbox script and that updated the firewall policy on cr*-eqiad:

[edit firewall family inet filter labs-in4 term cloudbackup from destination-address]
         10.192.0.130/32 { ... }
+        /* cloudbackup2003 */
+        10.192.21.7/32;
         10.192.32.186/32 { ... }
[edit firewall family inet filter labs-in4 term cloudbackup from destination-address]
         10.192.32.186/32 { ... }
+        /* cloudbackup2004 */
+        10.192.48.34/32;
         172.20.1.6/32 { ... }
[edit firewall family inet6 filter labs-in6 term cloudbackup from destination-address]
         2620:0:860:103:10:192:32:186/128 { ... }
+        /* cloudbackup2004 */
+        2620:0:860:104:10:192:48:34/128;
+        /* cloudbackup2003 */
+        2620:0:860:111:10:192:21:7/128;
         2620:0:861:102:10:64:16:58/127 { ... }

And now it works (over IPv4 only for some reason :():

taavi@cloudbackup2003 ~ $ telnet cloudcephmon1001.eqiad.wmnet 3300
Trying 2620:0:861:118:10:64:20:67...
Connection failed: Connection refused
Trying 10.64.20.67...
Connected to cloudcephmon1001.eqiad.wmnet.
Escape character is '^]'.
ceph v2
taavi closed this task as Resolved.Apr 2 2024, 7:38 AM
Stashbot added a comment.Apr 2 2024, 7:39 AM

Mentioned in SAL (#wikimedia-operations) [2024-04-02T07:39:25Z] <taavi> update firewall policy on cr-eqiad, cr-codfw T361537

taavi mentioned this in T361549: Automatically run Capirca Netbox script regularly.Apr 2 2024, 7:40 AM
gerritbot added a comment.Apr 2 2024, 9:43 PM

Change #1016446 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] cinder backups: move schedule config from a template into hiera

https://gerrit.wikimedia.org/r/1016446

gerritbot added a project: Patch-For-Review.Apr 2 2024, 9:43 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits