First step I think would be to have a proper routing_source_ip address in codfw1dev.

Current status in eqiad1:

• routing_source_ip is 185.16.56.1
• belongs to CIDR 185.15.56.0/25 https://netbox.wikimedia.org/ipam/prefixes/2/
• FQDN nat.openstack.eqiad1.wikimediacloud.org
• core routers have routing to direct this CIDR to 208.80.155.92/29 which is cloudinstances2b-gw.openstack.eqiad1.wikimediacloud.org (the neutron virtual router in eqiad1)

Current status in codfw1dev:

• routing_source_ip is 172.16.129.254
• belongs to CIDR 172.16.129.0/24 https://netbox.wikimedia.org/ipam/prefixes/307/
• no FQDN

Thus, we would need to:

• allocate a public IPv4 CIDR (probably a /29 should be enough). This provides us with 8 addresses. We can use 1 for routing_source_ip and the other 7 for floating_ips we may need in this deployment.
• routing in core routers. The whole CIDR should be directed to 208.80.153.190/29, which is cloudinstances2b-gw.openstack.codfw1dev.wikimediacloud.org (the neutron virtual router in codfw1dev)
• introduce FQDN nat.openstack.codfw1dev.wikimediacloud.org for the first address in the CIDR.
• update neutron net/subnet objects in codf1dev, making sure we trim down the related IP pool to account for the routing_source_ip address.
• review and double-check core router filtering, although it should be already very similar to the one for eqiad1.

cc @ayounsi for actionable items on his side (allocating the CIDR and adding routing).

I'm all in to make codfw similar to eqiad. In that optic maybe we should assign a /25 as well. But I don't have a strong preference.

What's the timeline? Is it a Q2 goal?

If a /25 works for you, then it works for us too :-) It is very unlikely that we will use such amount of addresses though.

We don't have a strong timeline (this is not part of a Q goal, yet). Would like to see this moving forward but can wait until holidays season is over.

Mentioned in SAL (#wikimedia-operations) [2019-12-18T04:54:40Z] <XioNoX> add static routes for cloud's 185.15.57.0/29 on cr1/2-codfw - T239347

Mentioned in SAL (#wikimedia-operations) [2019-12-18T04:59:42Z] <XioNoX> advertise 185.15.57.0/24 from [co|eq]dfw - T239347

We previously reserved 185.15.57.0/24 for "cloud site 2".
I cut 185.15.57.0/29 out of it (it's easier to grow a prefix than shrink it :))

Added the following routes on cr1/2-codfw:
set routing-options static route 185.15.57.0/29 next-hop 208.80.153.190

As well as starting advertising 185.15.57.0/24 from codfw/eqdfw:
set routing-options aggregate route 185.15.57.0/24 policy BGP_aggregate_contributors
set policy-options prefix-list bgp-out 185.15.57.0/24

Another option would be to advertise 185.15.56.0/23 from both eqiad/codfw.

Change 558821 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/homer/public@master] Start advertising 185.15.57.0/24 from codfw/eqdfw

https://gerrit.wikimedia.org/r/558821

Change 558821 merged by Ayounsi:
[operations/homer/public@master] Start advertising 185.15.57.0/24 from codfw/eqdfw

https://gerrit.wikimedia.org/r/558821

Change 559036 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] openstack: codfw1dev: update routing_source_ip

https://gerrit.wikimedia.org/r/559036

Change 559036 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] openstack: codfw1dev: update routing_source_ip

https://gerrit.wikimedia.org/r/559036

Mentioned in SAL (#wikimedia-cloud) [2019-12-18T12:55:07Z] <arturo> [codfw1dev] created a new subnet neutron object to hold the new CIDR for floating IPs (cloud-codfw1dev-floating - 185.15.57.0/29) T239347

Change 559040 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/dns@master] wikimediacloud.org: introduce FQDN for routing_source_ip in codfw1dev

https://gerrit.wikimedia.org/r/559040

@ayounsi this can be cleaned up https://netbox.wikimedia.org/ipam/prefixes/307/ you do it or I dot?

Change 559040 merged by Arturo Borrero Gonzalez:
[operations/dns@master] wikimediacloud.org: introduce FQDN for routing_source_ip in codfw1dev

https://gerrit.wikimedia.org/r/559040

In T239347#5750944, @aborrero wrote:

@ayounsi this can be cleaned up https://netbox.wikimedia.org/ipam/prefixes/307/ you do it or I dot?

cleaned-up myself after IRC confirmation by @ayounsi

Change 559349 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] nova firstboot.sh: remove a no-longer-needed apt hack for codf1dev

https://gerrit.wikimedia.org/r/559349

Change 559349 merged by Andrew Bogott:
[operations/puppet@production] nova firstboot.sh: remove a no-longer-needed apt hack for codf1dev

https://gerrit.wikimedia.org/r/559349

It all looks good now. Thanks all!

Change 560206 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] nova firstboot: add a few setup steps to firstboot.sh

https://gerrit.wikimedia.org/r/560206