Proposed operation steps:

Operation announcement sent: https://lists.wikimedia.org/pipermail/cloud-announce/2019-October/000226.html

In T235846#5587754, @Krenair wrote:

Copying from the patch for visibility: Let's please not put anything in the admin project that doesn't strictly need to be there - IIRC it has some special meaning/purpose, novaobserver's rights don't apply there, and it may cause difficulty splitting up permissions later on if we wanted to.
We can just make an empty project (in terms of instances) to hold a DNS zone or two.

In T235863#5587266, @JHedden wrote:

In T235863#5587207, @aborrero wrote:

The patch seems simple enough that perhaps we can do the patching ourselves? What do you think?

--- a/designateclient/v2/client.py
+++ b/designateclient/v2/client.py
@@ -58,7 +58,7 @@ class DesignateAdapter(adapter.LegacyJsonAdapter):
         if self.all_projects:
             kwargs['headers'].setdefault(
                 'X-Auth-All-Projects',
-                self.all_projects
+                str(self.all_projects)
             )
         if self.edit_managed:

I think that's a good idea. What's our best approach to adding a patch to the python-designateclient=2.3.0-2 package? I was looking at using dpkg-source --commit but I'm not sure how we should increment the version.

In T235863#5586981, @Andrew wrote:

@aborrero can you reproduce this now? I just ran some tests (in the 'testlabs' project) and things seemed ok.

+1 for the admin project! Will try creating it now.

It is not clear to me which project should own the eqiad1.wikimedia.cloud subdomain. The admin project? The wmflabsdotorg project? a new one?
I remember there was a special case for domains with no project association, I guess that's the case of the current eqiad.wmflabs:

Much better now, thanks!

In CloudVPS every VM I could check have this puppet agent error now:

Any news on this one? CC @Andrew @RobH

Apparently with the current setup, nobody in SRE is paged (by SMS) because mgmt interfaces, but we WMCS do get paged:

cool thanks! I will work on the operation steps soon for you to review.

Drafting docs at https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Networking_and_ingress

Please @Bstorm and @JHedden confirm the scheduled operation window works for you both, thanks!

This is done.

Ok, I've been playing with the dynamicproxy nginx+lua components and I have a working setup. I disabled SSL/https in my tests until we handle T235252: Toolforge: SSL support for new domain toolforge.org.

In T235252#5567844, @Krenair wrote:

Yep.

toolforge:
  CN: toolforge.org
  SNI:
  - toolforge.org
  - '*.toolforge.org'
  - tools.wmflabs.org
  - '*.tools.wmflabs.org'
  authorized_regexes:
  - ^tools-proxy-[0-9]+\.tools\.eqiad\.wmflabs$
  challenge: dns-01

I believe the last person doing this kind of movement was @Andrew, so assigning to him to see if he can provide further info. I couldn't find any docs on wikitech.

ping @Krenair would you like to help me in this project? also T234617: Toolforge. introduce new domain toolforge.org

I think the patch https://gerrit.wikimedia.org/r/c/operations/puppet/+/508560 is mostly ready.

We had a team meeting on 2019-10-10 and we decided to try the custom admission controller to enforce correctness in ingress objects.

We had a meeting yesterday 2019-10-10 and we decided to try option 3 first, with fallback to option 2.

@Vgutierrez could you please advice on how to proceed with this?

Thanks! Done already:

We agreed on doing this in the WMCS team meeting. Assigning to @Andrew for final approval before doing the changes.

I'm proposing 2 changes:

More things to adjust:

I checked nova-fullstack this morning. Everything looks good. No leaks so far.

I can do the buster build. Perhaps we should consider uploading this package to Debian, it should be interesting for other people too.

Awesome! I think I get the idea. Perhaps we should continue conversation about this at T234231: Toolforge ingress: decide on how ingress configuration objects will be managed.
BTW I believe the FQDN scheme is <$toolname>.toolforge.org and not <$toolname>.tools.toolforge.org.

I've been playing with option 2, and here my tests:

Raising priority of this ticket, since the ceph project is part of our Q2 goals.

Raising priority of this ticket, since the ceph project is part of our Q2 goals.

All is done!

This should be done now:

I created a VM and tried both dmz_cidr and routing_source_ip:

Thanks @JHedden. In this case it helped me review every bit of the setup. I cleaned a lot of stuff and documented a bunch of missing bits about the network setup for codfw1dev in https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Neutron

I put the l3-agent in debug mode and it reports creating the rule correctly, no mention to the wrong rule in the logs:

I'm reviewing the neutron setup in codfw1dev following this checklist: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Deployment_sanity_checklist#General_networking_&_neutron
I noticed something weird (possibly a bug somewhere) when checking the dmz_cidr setting and how that translates to iptables rules.

This may be due to lack of credentials, thus a false positive. I didn't load the env vars before running the script.

For the record, Bryan mentioned another option: having dynamicproxy understand how to forward to the new k8s cluster.

This is related to T234218: Can't login into Wikimedia Space . I already replied there. Please restart the docker service.

I see some issues with docker in the discuss-space server: