Page MenuHomePhabricator
Create Task
Maniphest T128590

Cassandra uses default ip address for outbound packets while bootstrapping
Closed, DeclinedPublic
Actions

Description

spotted this while bootstrapping restbase1010, it looks like outbound packets are sourced with the main host ip and not the ones configured in cassandra.yaml, in this case it wasn't working because of firewall rules on destination hosts

14:14:03.847886 IP 10.64.0.112.35406 > 10.64.0.220.7000: Flags [S], seq 2750243604, win 29200, options [mss 1460,sackOK,TS val 40440048 ecr 0,nop,wscale 9], length 0
14:14:03.847898 IP 10.64.0.112.36730 > 10.64.0.221.7000: Flags [S], seq 823540075, win 29200, options [mss 1460,sackOK,TS val 40440048 ecr 0,nop,wscale 9], length 0
14:14:03.848099 IP 10.64.0.112.47684 > 10.64.48.99.7000: Flags [S], seq 2253905698, win 29200, options [mss 1460,sackOK,TS val 40440048 ecr 0,nop,wscale 9], length 0
14:14:03.848101 IP 10.64.0.112.37021 > 10.64.48.100.7000: Flags [S], seq 3996782521, win 29200, options [mss 1460,sackOK,TS val 40440048 ecr 0,nop,wscale 9], length 0
14:14:03.848103 IP 10.64.0.112.52035 > 10.64.48.120.7000: Flags [S], seq 3150107822, win 29200, options [mss 1460,sackOK,TS val 40440048 ecr 0,nop,wscale 9], length 0
14:14:03.848105 IP 10.64.0.112.58029 > 10.64.32.187.7000: Flags [S], seq 1891138042, win 29200, options [mss 1460,sackOK,TS val 40440048 ecr 0,nop,wscale 9], length 0
14:14:03.848299 IP 10.64.0.112.57468 > 10.64.32.195.7000: Flags [S], seq 2077856937, win 29200, options [mss 1460,sackOK,TS val 40440048 ecr 0,nop,wscale 9], length 0
14:14:03.848302 IP 10.64.0.112.45408 > 10.64.0.231.7000: Flags [S], seq 3375292860, win 29200, options [mss 1460,sackOK,TS val 40440048 ecr 0,nop,wscale 9], length 0
14:14:03.848486 IP 10.64.0.112.46393 > 10.64.32.159.7000: Flags [S], seq 72980954, win 29200, options [mss 1460,sackOK,TS val 40440048 ecr 0,nop,wscale 9], length 0
14:14:03.848488 IP 10.64.0.112.45264 > 10.64.48.130.7000: Flags [S], seq 2112860170, win 29200, options [mss 1460,sackOK,TS val 40440048 ecr 0,nop,wscale 9], length 0
14:14:03.848490 IP 10.64.0.112.47592 > 10.64.0.230.7000: Flags [S], seq 1749977256, win 29200, options [mss 1460,sackOK,TS val 40440048 ecr 0,nop,wscale 9], length 0
14:14:03.912097 IP 10.64.0.112.38070 > 10.64.32.192.7000: Flags [S], seq 1521230055, win 29200, options [mss 1460,sackOK,TS val 40440064 ecr 0,nop,wscale 9], length 0
restbase1010:~$ host restbase1010.eqiad.wmnet
restbase1010.eqiad.wmnet has address 10.64.0.112
restbase1010:~$ host restbase1010-a.eqiad.wmnet
restbase1010-a.eqiad.wmnet has address 10.64.0.114

Related Objects

Event Timeline

fgiunchedi created this task.Mar 2 2016, 2:23 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 2 2016, 2:23 PM
fgiunchedi triaged this task as Low priority.Apr 27 2016, 1:13 PM
Eevans renamed this task from cassandra uses default ip address for outbound packets while bootstrapping to Cassandra uses default ip address for outbound packets while bootstrapping.Apr 29 2016, 8:15 PM
Eevans added a project: Cassandra.
• GWicke added a project: Services.Oct 12 2016, 11:21 PM
• mobrovac edited projects, added Services (watching); removed Services.Dec 22 2016, 3:45 PM
Eevans subscribed.EditedJan 20 2017, 4:24 PM

Weird:

$ sudo tcpdump -ni eth0 src host restbase1010-a.eqiad.wmnet and proto TCP and src portrange 7000-7001
...
16:18:22.947178 IP 10.64.0.116.7000 > 10.64.0.230.38846: Flags [.], ack 19728, win 625, options [nop,nop,TS val 1876382839 ecr 1877066570], length 0
16:18:22.948080 IP 10.64.0.116.7000 > 10.64.32.207.38955: Flags [.], ack 70296, win 1893, options [nop,nop,TS val 1876382839 ecr 1875566858], length 0
16:18:22.950021 IP 10.64.0.116.7001 > 10.192.48.70.56434: Flags [.], ack 1563, win 2301, options [nop,nop,TS val 1876382839 ecr 1418950764], length 0
16:18:22.950267 IP 10.64.0.116.7000 > 10.64.32.207.38955: Flags [.], ack 70367, win 1893, options [nop,nop,TS val 1876382839 ecr 1875566858], length 0
16:18:22.952780 IP 10.64.0.116.7000 > 10.64.32.203.36409: Flags [.], ack 127333, win 1268, options [nop,nop,TS val 1876382840 ecr 1875848084], length 0
16:18:22.953011 IP 10.64.0.116.7001 > 10.192.48.69.36462: Flags [.], ack 11881, win 731, options [nop,nop,TS val 1876382840 ecr 1418950765], length 0
16:18:22.953112 IP 10.64.0.116.7000 > 10.64.0.230.38846: Flags [.], ack 19828, win 625, options [nop,nop,TS val 1876382840 ecr 1877066571], length 0
16:18:22.955494 IP 10.64.0.116.7000 > 10.64.0.230.38846: Flags [.], ack 20320, win 625, options [nop,nop,TS val 1876382841 ecr 1877066572], length 0
16:18:22.955561 IP 10.64.0.116.7000 > 10.64.32.207.38955: Flags [.], ack 70438, win 1893, options [nop,nop,TS val 1876382841 ecr 1875566860], length 0
1549 packets captured
1819 packets received by filter
0 packets dropped by kernel
$

restbase1011-a.eqiad.wmnet is sourcing traffic from 7000 and 7001.

$ sudo tcpdump -ni eth0 src host restbase1010.eqiad.wmnet and proto TCP and src portrange 7000-7001
...
16:20:56.009256 IP 10.64.0.112.7001 > 10.192.48.49.53258: Flags [R.], seq 0, ack 2774926345, win 0, length 0
16:20:56.146691 IP 10.64.0.112.7001 > 10.192.48.49.42450: Flags [R.], seq 0, ack 2846120781, win 0, length 0
16:20:56.285205 IP 10.64.0.112.7001 > 10.192.48.49.55610: Flags [R.], seq 0, ack 1463595064, win 0, length 0
16:20:56.422580 IP 10.64.0.112.7001 > 10.192.48.49.46609: Flags [R.], seq 0, ack 2854990770, win 0, length 0
16:20:56.563799 IP 10.64.0.112.7001 > 10.192.48.49.49357: Flags [R.], seq 0, ack 1645837451, win 0, length 0
16:20:56.700951 IP 10.64.0.112.7001 > 10.192.48.49.42508: Flags [R.], seq 0, ack 2990377545, win 0, length 0
16:20:56.838376 IP 10.64.0.112.7001 > 10.192.48.49.36284: Flags [R.], seq 0, ack 1089322175, win 0, length 0
16:20:56.975936 IP 10.64.0.112.7001 > 10.192.48.49.38562: Flags [R.], seq 0, ack 3556649634, win 0, length 0
16:20:57.113562 IP 10.64.0.112.7001 > 10.192.48.49.35763: Flags [R.], seq 0, ack 3299051790, win 0, length 0
16:20:57.251023 IP 10.64.0.112.7001 > 10.192.48.49.49958: Flags [R.], seq 0, ack 3362137409, win 0, length 0
619 packets captured
629 packets received by filter
0 packets dropped by kernel
$

restbase1010.eqiad.wmnet (no Cassandra instance bound), is sourcing some packets from 7001 (only).

And port 7001 is only bound to the 3 aliases:

$ sudo netstat -anpl | grep -i listen | grep 7001
tcp        0      0 10.64.0.116:7001        0.0.0.0:*               LISTEN      22600/java      
tcp        0      0 10.64.0.115:7001        0.0.0.0:*               LISTEN      18917/java      
tcp        0      0 10.64.0.114:7001        0.0.0.0:*               LISTEN      14972/java      
unix  2      [ ACC ]     STREAM     LISTENING     277567001 22405/java          /tmp/.java_pid22405.tmp
$
Eevans added a comment.EditedJan 20 2017, 4:35 PM
In T128590#2956184, @Eevans wrote:

[ ... ]

$ sudo tcpdump -ni eth0 src host restbase1010.eqiad.wmnet and proto TCP and src portrange 7000-7001
...
16:20:56.009256 IP 10.64.0.112.7001 > 10.192.48.49.53258: Flags [R.], seq 0, ack 2774926345, win 0, length 0
16:20:56.146691 IP 10.64.0.112.7001 > 10.192.48.49.42450: Flags [R.], seq 0, ack 2846120781, win 0, length 0
16:20:56.285205 IP 10.64.0.112.7001 > 10.192.48.49.55610: Flags [R.], seq 0, ack 1463595064, win 0, length 0
16:20:56.422580 IP 10.64.0.112.7001 > 10.192.48.49.46609: Flags [R.], seq 0, ack 2854990770, win 0, length 0
16:20:56.563799 IP 10.64.0.112.7001 > 10.192.48.49.49357: Flags [R.], seq 0, ack 1645837451, win 0, length 0
16:20:56.700951 IP 10.64.0.112.7001 > 10.192.48.49.42508: Flags [R.], seq 0, ack 2990377545, win 0, length 0
16:20:56.838376 IP 10.64.0.112.7001 > 10.192.48.49.36284: Flags [R.], seq 0, ack 1089322175, win 0, length 0
16:20:56.975936 IP 10.64.0.112.7001 > 10.192.48.49.38562: Flags [R.], seq 0, ack 3556649634, win 0, length 0
16:20:57.113562 IP 10.64.0.112.7001 > 10.192.48.49.35763: Flags [R.], seq 0, ack 3299051790, win 0, length 0
16:20:57.251023 IP 10.64.0.112.7001 > 10.192.48.49.49958: Flags [R.], seq 0, ack 3362137409, win 0, length 0
619 packets captured
629 packets received by filter
0 packets dropped by kernel
$

On second look, these are entirely resets (and for more hosts than 10.192.48.49, as show here), and so must be the result of having the main IP listed as a seed everywhere else.

Either what @fgiunchedi saw in March of last year is limited to bootstrapping, or it has since resolved itself.

Eevans edited projects, added User-Eevans; removed RESTBase-Cassandra.Jul 3 2018, 10:28 PM

@fgiunchedi, is this still a thing?

Eevans moved this task from Backlog to Blocked on the User-Eevans board.Jul 3 2018, 10:28 PM
fgiunchedi added a comment.Jul 4 2018, 2:11 PM
In T128590#4396369, @Eevans wrote:

@fgiunchedi, is this still a thing?

Good question, I think we'll have to try on the next bootstrap and sniff the traffic. So far it hasn't been an issue in practice because we're including the main ip address in ferm rules.

• mobrovac changed the task status from Open to Stalled.Jul 4 2018, 3:20 PM
• mobrovac added a project: Platform Team Legacy (Watching / External).Dec 20 2018, 12:04 PM
• Phabricator_maintenance moved this task from Backlog to Acknowledged on the SRE board.Jan 26 2019, 8:33 PM
Aklapper added a comment.May 21 2020, 12:48 PM
In T128590#4397640, @fgiunchedi wrote:
In T128590#4396369, @Eevans wrote:

is this still a thing?

Good question, I think we'll have to try on the next bootstrap and sniff the traffic. So far it hasn't been an issue in practice because we're including the main ip address in ferm rules.

Two years later: Does anyone know if this is still an issue? If yes, this task should be open. If not, this task should be declined or resolved. Thanks!

Aklapper added a comment.Nov 4 2020, 10:44 PM

@fgiunchedi, @Eevans: Ping - anyone knows if this is still an issue? If yes, this task should be open. If not, this task should be declined or resolved. Thanks!

Eevans closed this task as Declined.Jun 7 2021, 8:02 PM

Boldly closing declined, please reopen (at a commiserate priority) if this is something we should see to.

Eevans mentioned this in T327954: session storage: dissonant cluster status after reboot (was: 'cannot achieve consistency level' errors).Mar 14 2023, 12:25 AM
Eevans mentioned this in T334751: Cassandra instances use host IP interface in outgoing messages.Apr 14 2023, 3:42 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL