Currently we have a hiera key abuse_networks['public_cloud_nets'] which is used in activity used in varnish to provide some rate limiting. As IP allocations for theses big cloud providers change some what frequently i wonder if we should put something in place to automate refreshing this data. The current data suggests it was " generated on 2019-12-30 "
AWS allow to subscribe to the modification of the list fwiw, see https://docs.aws.amazon.com/general/latest/gr/aws-ip-ranges.html#subscribe-notifications
Google cloud compute is a bit less structured, see https://cloud.google.com/vpc/docs/vpc#manually_created_subnet_ip_ranges (under the Restricted ranges paragraph, link to ipranges/goog.txt.
2 other options:
- Define a list of ASNs and get the matching prefixes from BGP (or API like RIPE stats)
- Define a list of ASNs and get the matching prefixes from MaxMind DBs
I like the 2nd as we already have the tooling around it, and it doesn't require regularly fetching data from URLs that could change/break.
A downside, for example with Google is that it will most likely include crawlers IPs
I'm also worried about cases where the ASN IP space includes things like all their MXes, or their corporate workstation IP space as well. This is true of multiple cloud providers.
We might have to implement a few different scrape approaches...