Page MenuHomePhabricator

Consider renumbering Labs to separate address spaces
Closed, ResolvedPublic

Description

Recent events have proved that our IP numbering scheme/overlap between production and Labs is confusing or non-obvious especially during casual changes that fly under the radar.

I think at the next major networking work in Labs (Neutron?), we should perhaps consider a new numbering plan for Labs, to make things easier for everyone.

For example, we could consider using:

  • The 172.16.0.0/12 space (still RFC 1918) for private addresses, instead of 10/8.
  • An entirely separate IPv4 subnet for floating IPs outside of our regular 208.80.152.0/22 space (e.g. one from either our RIPE or ulsfo space, or perhaps even get/pay for more space specifically for this use case). I think something like this was originally attempted with 208.80.155.0/24, but we ended up placing a lot more there (production, frack etc.) which defeated the point in the end.
  • Assuming that we'll also do IPv6 along with this change, a piece of IP space outside of our regular 2620:0:860::/46 — perhaps also something from our RIPE /32, or an expansion of our /46 from ARIN.

Event Timeline

faidon raised the priority of this task from to Low.
faidon updated the task description. (Show Details)
faidon added projects: SRE, netops, Cloud-Services.
faidon added subscribers: faidon, mark.

We agreed on all of the above during the Barcelona offsite. We've preliminary agreed to attempt implementing them in tandem with the Neutron migration, which would require a separate address space anyway.

The 172.16.0.0/12 space (still RFC 1918) for private addresses, instead of 10/8.

Right now our allocation for instances in the main deployment in eqiad is a /21 and we have around 725 instances out of that huge space. For now I'm thinking we should drop a /21 out of the 172 address space for consistency, and the next few iterations here and then IPv6 will bring more sanity.

Current instance allocations:

templates/10.in-addr.arpa:; 10.68.0.0/24 - labs-instances1-a-eqiad
templates/10.in-addr.arpa:; 10.68.16.0/21 - labs-instances1-b-eqiad
templates/10.in-addr.arpa:; 10.68.32.0/24 - labs-instances1-c-eqiad
templates/10.in-addr.arpa:; 10.68.48.0/24 - labs-instances1-d-eqiad
templates/10.in-addr.arpa:; 10.196.0.0/24 - labs-instances1-a-codfw
templates/10.in-addr.arpa:; 10.196.16.0/21 - labs-instances1-b-codfw
templates/10.in-addr.arpa:; 10.196.32.0/24 - labs-instances1-c-codfw
templates/10.in-addr.arpa:; 10.196.48.0/24 - labs-instances1-d-codfw

I think this is now done with Neutron, and while the old space remains for now, the migration is underway, so this task can be closed. @ayounsi, @aborrero, @chasemp?

faidon claimed this task.

Perfect! As far as I can see, there a few pending tasks, but are or should probably be covered in other tasks.

Specifically:

  • Deprecation of the old space, which is equivalent at this point to deprecation of nova-network. This is already planned and the migration to Neutron is underway.
  • For Neutron, a static route has been set up in core routers for 172.16.0.0/21 which is not something we should continue doing (as it defeats the point of separation!) and would need to be fixed.
  • Neutron's cloud-instance-transport1-b-eqiad uses 10/8 space, and that should be renumbered at some point as well. Probably migrate to public /31s or something like that.
  • IPv6 hasn't been set up in Neutron yet, but nova-network doesn't have IPv6 space anyway :)

With all that said, the core premise of this task is resolved I think. @ayounsi, can you make sure we have tasks for the other actionables here? Thanks!

Here are the tasks for the record:

  • Deprecation of the old space, which is equivalent at this point to deprecation of nova-network. This is already planned and the migration to Neutron is underway.

T193496 (in some way)

  • For Neutron, a static route has been set up in core routers for 172.16.0.0/21 which is not something we should continue doing (as it defeats the point of separation!) and would need to be fixed.

T174596#4681328

  • Neutron's cloud-instance-transport1-b-eqiad uses 10/8 space, and that should be renumbered at some point as well. Probably migrate to public /31s or something like that.

T207663

  • IPv6 hasn't been set up in Neutron yet, but nova-network doesn't have IPv6 space anyway :)

T187929