Page MenuHomePhabricator

varnish filtering: should we automatically update public_cloud_nets
Open, MediumPublic

Description

Currently we have a hiera key abuse_networks['public_cloud_nets'] which is used in activity used in varnish to provide some rate limiting. As IP allocations for theses big cloud providers change some what frequently i wonder if we should put something in place to automate refreshing this data. The current data suggests it was " generated on 2019-12-30 "

Event Timeline

jbond triaged this task as Medium priority.Dec 17 2020, 2:48 PM
jbond created this task.

AWS allow to subscribe to the modification of the list fwiw, see https://docs.aws.amazon.com/general/latest/gr/aws-ip-ranges.html#subscribe-notifications
Google cloud compute is a bit less structured, see https://cloud.google.com/vpc/docs/vpc#manually_created_subnet_ip_ranges (under the Restricted ranges paragraph, link to ipranges/goog.txt.

Volans renamed this task from varnihs filtering: should we automaticly update public_cloud_nets to varnish filtering: should we automatically update public_cloud_nets .Dec 17 2020, 2:57 PM

2 other options:

  • Define a list of ASNs and get the matching prefixes from BGP (or API like RIPE stats)
  • Define a list of ASNs and get the matching prefixes from MaxMind DBs

I like the 2nd as we already have the tooling around it, and it doesn't require regularly fetching data from URLs that could change/break.

A downside, for example with Google is that it will most likely include crawlers IPs

A downside, for example with Google is that it will most likely include crawlers IPs

I'm also worried about cases where the ASN IP space includes things like all their MXes, or their corporate workstation IP space as well. This is true of multiple cloud providers.

We might have to implement a few different scrape approaches...