@chasemp Can you provide an ETA for returning the /25?

In T193496#4192431, @ayounsi wrote:

@chasemp Can you provide an ETA for returning the /25?

It's educated guess work but towards the end of calendar year 2018 is realistic. Oct/November would be my best estimate.

The /25 -> /24 renumbering seems fairly straightforward, but given a) IPv4's depletion (we effectively cannot get more IPv4 space from any of the RIRs), b) the Neutron redesign and c) Cloud Services' growth and needs like T122406's, I think it's worthwhile to look at it a bit more broadly in order to make sure we avoid e.g. depletion or fragmentation of our IP space. Perhaps for instance we need to be looking at a larger assignment :)

So first of:

• My understanding from the description is that the intention is for this to be used for floating IPs. Do you foresee any other needs in terms of public IPv4 space besides those?
• Relatedly, do we have any kind of historical growth figures and/or estimates for future growth, in the short-term or mid-term, besides those extra 20 IPs for Toolforge that you mentioned?
• Is this going to be routed just to eqiad, or does the upcoming addition of the second zone in the next FY means that this is also going to be partially routed to codfw? I'm asking because routing a single /24 to two DCs is possible, but it would cause an impact to availability in case of a split brain between our data centers that may or may not be acceptable?
• How do we intend to subnet this /24? Is this all going to be flat for floating IPs, or partitioned somehow per data center, row, other kind of availability zone, etc.?

Finally, I think it would be useful to look at this holistically and figure out the addressing plan for {eqiad, codfw} × {IPv4, IPv6} × {public, private} for at least as far as we can reasonably foresee. Do we have a sense for that yet?

In T193496#4210037, @faidon wrote:

The /25 -> /24 renumbering seems fairly straightforward, but given a) IPv4's depletion (we effectively cannot get more IPv4 space from any of the RIRs), b) the Neutron redesign and c) Cloud Services' growth and needs like T122406's, I think it's worthwhile to look at it a bit more broadly in order to make sure we avoid e.g. depletion or fragmentation of our IP space. Perhaps for instance we need to be looking at a larger assignment :)

So first of:

• My understanding from the description is that the intention is for this to be used for floating IPs. Do you foresee any other needs in terms of public IPv4 space besides those?
• Relatedly, do we have any kind of historical growth figures and/or estimates for future growth, in the short-term or mid-term, besides those extra 20 IPs for Toolforge that you mentioned?
• Is this going to be routed just to eqiad, or does the upcoming addition of the second zone in the next FY means that this is also going to be partially routed to codfw? I'm asking because routing a single /24 to two DCs is possible, but it would cause an impact to availability in case of a split brain between our data centers that may or may not be acceptable?
• How do we intend to subnet this /24? Is this all going to be flat for floating IPs, or partitioned somehow per data center, row, other kind of availability zone, etc.?

Finally, I think it would be useful to look at this holistically and figure out the addressing plan for {eqiad, codfw} × {IPv4, IPv6} × {public, private} for at least as far as we can reasonably foresee. Do we have a sense for that yet?

Thanks man.

(5 responses top-to-bottom)

My understanding from the description is that the intention is for this to be used for floating IPs. Do you foresee any other needs in terms of public IPv4 space besides those?

Yes, all for floating IP assignment (which mirrors the use for the existing /25). The intention/expectation is for this to carry things forward for 3+ years. Based on historic growth this seems realistic. When we get to IPv6 the models I have seen do not support any kind of NAT or IP overlapping and so it's assumed all addresses are public addresses. I have only really stepped through it as far as Newton though.

Relatedly, do we have any kind of historical growth figures and/or estimates for future growth, in the short-term or mid-term, besides those extra 20 IPs for Toolforge that you mentioned?

I do have some growth thinking but it's a lot of adhoc. I've gone through the last year and found 5 instances of floating IP allocation and 2 of those were Tools and Toolsbeta. It's fair to say we burn through about +/-10 a year, which is pretty small for a few reasons: we offer (and push) the general purpose nginx reverse proxy-as-a-service to conserve, we have shared jump bastions for all projects, we encourage Toolforge usage where it's not on the user, and we almost always try to find a way to do things that will not use a floating IP due to it being a very finite resource historically.

I can think of a few use cases coming so my estimate (based on having a decent quota/project request process for about 2 years) is we'll continue allocating 10-15 a year (outside of the 20 backlog). There are a few workarounds to exhaustion in that Toolforge instances can be rebuilt with more resourcing for tighter clustering to reduce floating pool IP usage, etc, but in general we've been very conservative here. The 20 in Toolforge missing now are an anomaly where for a few exec builds it was just forgotten and then we decided to bend our 1:1 rule for external IP (for auditing, rate-limiting, etc) for execs when building out the k8s workers. They were initially all webservices and so it wasn't as big of a deal but this is changing. So to correct that say 20 off the bat and then a very safe and probably liberal thought of 10-15/y. This is pretty rough but in the ballpark.

Is this going to be routed just to eqiad, or does the upcoming addition of the second zone in the next FY means that this is also going to be partially routed to codfw? I'm asking because routing a single /24 to two DCs is possible, but it would cause an impact to availability in case of a split brain between our data centers that may or may not be acceptable?

The way we have talked about it so far internally, regions would be separate address space allocations, and the story of extending individual projects across regions is very muddy and probably not something we can bite off soon. Probably regions would match our site model (i.e. one region for eqiad and one for codfw). The /24 here is expected to be for the existing eqiad deployment only, and the second region thought about for Q3 purchasing in 18-19 would be its own consideration. Is it possible to keep the existing /25 reserved for that potential use case and still allocate the /24 here? That would I believe resource all asks within 36months.

How do we intend to subnet this /24? Is this all going to be flat for floating IPs, or partitioned somehow per data center, row, other kind of availability zone, etc.?

I looked into this a bit more since you asked. At the moment the breakdown on floating IP assignment is:

53 Toolforge
50 Everything else

The intention is for this entire /24 block to be allocated to the eqiad deployment that exists now (or the Neutron version of it that things are being folded into). It may make sense to break it into two /25's with one for Toolforge and one for everything else. Both of these would be the same deployment in eqiad. The intention has been to scale out horizontally with Toolforge as the first use case. (i.e. Toolforge multi-row VXLAN overlay (not there yet), and hopefully Toolforge across regions for HA/scale/etc at some point.) Roughly the idea at the moment is region == site, availability zone == row and each region should be capable of multi-row compute but is still tied to single row for north-south (l3 traffic) making a totally second region attractive.

Finally, I think it would be useful to look at this holistically and figure out the addressing plan for {eqiad, codfw} × {IPv4, IPv6} × {public, private} for at least as far as we can reasonably foresee. Do we have a sense for that yet?

Getting this /24 for the existing /25 would cover IPv4 for the existing deployment for 3 years I expect. Then in addition there is the the planned second region. If we want to future-proof then thinking on another similar block assignment (or holding the /25 there now for that case) makes sense. I'm optimistic about the IPv6 potential here as it's a first class citizen in OpenStack starting around Newton but I'm unsure when/how we would be able to deploy as IPv6 only. @ayounsi and I have been working off of a google sheet where this is being laid out for existing but this does not include thinking on a new region at either site at the moment.

OK, so it looks 185.15.56.0/24 is proposed to be used immediately in eqiad, to replace 208.80.155.128/25 in the next ~6 months. Additionally, 185.15.57.0/24 is proposed to be reserved (but not assigned) to be used tentatively in Q3 FY18-19 in codfw, for a region 2 deployment. Both of these sound good to me and you can proceed :)

@ayounsi, we need to actually create objects for this in RIPE -- it's currently our LIR space, but not assigned to any entity.

It also seems there are some ideas around IPv6 space, but it seems that these can wait until the project progresses further, correct? I'd like to figure out our plan for 2a02:ec80::/32 before we commit to anything here.

The only remaining part that is unclear to me is how the two /24s will be subnet. I'd like for us to think about this through to make sure we don't fragment the IP space and require even more IP space down the line, which unfortunately will be hard to grab :( I think e.g. it may be a good idea to not allocate this space entirely and limit to some subnet, with the rest reserved for "future use". As an example of what I was thinking of: codfw currently has 208.80.153.128/27 as labtest, and it's not clear to me whether the new /24 allocation for region 2 will replace it. Perhaps we should e.g. reserve a /27 from each of eqiad/codfw's /24 for this sort of "labtest" setup?

https://apps.db.ripe.net/db-web-ui/#/lookup?source=ripe&key=185.15.56.0%2F24AS14907&type=route created.

IPv6 is tracked in T187929 and can indeed wait.

Maybe something like the following, but just a suggestion, I don't want to step on @chasemp's toes

185.15.56.0/25   - 1:1 replacement of 208.80.155.128/25
185.15.56.128/26 - reserved for groth beyond first /25
185.15.56.192/27 - reserved for future use
185.15.56.224/27 - reserved for labtest

In T193496#4228340, @ayounsi wrote:

https://apps.db.ripe.net/db-web-ui/#/lookup?source=ripe&key=185.15.56.0%2F24AS14907&type=route created.

185.15.56.0/25   - 1:1 replacement of 208.80.155.128/25
185.15.56.128/26 - reserved for groth beyond first /25
185.15.56.192/27 - reserved for future use
185.15.56.224/27 - reserved for labtest

Nice, I don't anticipate we will need a labtest out of this 185.15.56.0/24 block as the deployment in codfw is enough (ping @Andrew on my reasoning there).

So seems like for Eqiad:

185.15.56.0/25     - 1:1 replacement of 208.80.155.128/25
185.15.56.128/26   - reserved for growth beyond first /25
185.15.56.192/26   - reserved for future use

The cloud address allocation spreadsheet was mostly updated but I added a bit to clarify (I hope) re: labtest.

We do want to keep labtest and the 208.80.153.128/27 - Labtest being entirely adhoc right now seems better assigned out of the codfw /24 when that range becomes assigned and not only reserved. I updated the doc to be clear at least

Perhaps we should e.g. reserve a /27 from each of eqiad/codfw's /24 for this sort of "labtest" setup?

So yes, but we only need it from the codfw range. If we want to mark that allocated and not just reserved we can give back the 208.80.153.128/27 for labtest now:

In that case Codfw looks like:

185.15.57.0/25     - OpenStack Region2
185.15.57.128/26   - reserved for growth beyond first /25
185.15.57.224/27   - future use
185.15.57.192/27   - labtest

Based on the approvals above I'm moving forward with the idea that 185.15.56.0/24 is good to go with some details on breaking it down to be agreed on. Maybe what I've got here will suffice :)

In T193496#4420713, @chasemp wrote:

+    /* Cloud public prefix via labnet100[45] */
+    route 185.15.56.0/25 next-hop 10.64.22.4;

@ayounsi if https://phabricator.wikimedia.org/T193496#4417390 seems sane to you could you possibly advertise 185.15.56.0/24 to the world when you get a chance? Thanks

(@Andrew is working on the DNS portion for this in T199374)

Mentioned in SAL (#wikimedia-operations) [2018-07-12T21:12:04Z] <XioNoX> advertising 185.15.56.0/24 to the DFZ - T193496

For the record, with the migration away from and shutdown of the nova-network 'main' region, the 208.80.155.128/25 range is no longer in use. Andrew approved the commit to remove the relevant reverse DNS delegation records (whose commit message also includes a statement to this effect) at https://gerrit.wikimedia.org/r/#/c/operations/dns/+/505478/.
I'm not sure if anything else needs doing in this ticket or if it can be resolved?

Edit: it routed to labnet1001 which has just been decommed in T221818

Mentioned in SAL (#wikimedia-operations) [2019-05-03T00:10:05Z] <XioNoX> remove static route to 208.80.155.128/25 on cr1/2-eqiad - T193496

Change 507907 had a related patch set uploaded (by Alex Monk; owner: Alex Monk):
[operations/puppet@production] network: remove old labs public range

https://gerrit.wikimedia.org/r/507907

Change 507907 merged by Alexandros Kosiaris:
[operations/puppet@production] network: remove old labs public range

https://gerrit.wikimedia.org/r/507907

I was 6 months off on my estimate for this :)