Page MenuHomePhabricator

Collect Google IPs pinging the load balancers
Closed, ResolvedPublic

Description

A large amount of our ICMP echo volume comes from IPs owned by Google. Such traffic caused the packet loss issues mentioned in T163312 and now fixed.

We need to collect information about the Google (Compute Engine?) IPs involved and send it to abuse@google.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript
ema triaged this task as Medium priority.May 18 2017, 7:14 AM
ema moved this task from Backlog to LoadBalancer on the Traffic board.

I've collected 60s of ICMP traffic from GCE on the load balancers and sent a report through https://support.google.com/code/contact/cloud_platform_report?hl=en. I've also added a column to https://grafana.wikimedia.org/dashboard/db/load-balancers plotting incoming ICMP requests per second.

I don't think anything has changed since on Google's end. Do we try harder or just accept it?

I don't think anything has changed since on Google's end. Do we try harder or just accept it?

I guess we should try harder. :)

The full list of IPs/ASNs was generated with:

# on lvs1001
sudo timeout 60 tcpdump -n -e icmp[icmptype] == 8 > lvs1001.wikimedia.org-icmptype_8.log
# off to another system with geoip-bin installed
awk '{print $10}' lvs1001.wikimedia.org-icmptype_8.log | sort | uniq -c | sort -n | while read n ip; do
  printf "$n $ip "; geoiplookup $ip | awk '/ASNum/ { print $4 }'
done

Then I did select the IPs belonging to ASNs owned by Google (according to whois) and sent the report mentioning those.

I've copied the full data files to lvs1001.wikimedia.org:~ema/google-icmp-ips in case someone wants to take a look, including the data relevant to other LVSs.

fgiunchedi claimed this task.
fgiunchedi subscribed.

I mentioned this task and problem to a friend working in SRE networking, we're now receiving about one tenth of the icmp traffic inbound on lvses.

2018-03-02-114207_238x987_scrot.png (987×238 px, 47 KB)