Mentioned in SAL (#wikimedia-cloud) [2020-03-09T13:10:59Z] <arturo> created VM toolsbeta-legacy-redirector (T247236)

I saw a couple of patches by @bd808 :

• https://gerrit.wikimedia.org/r/c/operations/software/tools-webservice/+/578413 Add support for redirecting to toolforge.org
• https://gerrit.wikimedia.org/r/c/operations/puppet/+/578406 toolforge: support canonical redirects in urlproxy

The approach in those patches differs a bit from what I had in mind, so I guess this topic worth a discussion in the team.
Specifically, I have a couple of questions/concerns on how Bryan's approach would work:

• the approach is mostly coupled to the k8s ingress engine. We need ingress configuration to hande the legacy redirection. How do we control/enforce that tools developers effectively use this setting?
• how do we implement the D day in which we want to stop adding any more webservices to the legacy URL scheme?
• it involves adding some additional magic to the tools frontproxy, whereas my feeling is that we should do exactly the opposite, simplifying tools frontproxy.
• in summary, this feels a bit more complex for no added value? Or at least I fail to see which added value we get vs my approach.

My original idea was to introduce a new server doing only the redirection, external to the tools front proxy or the k8s cluster.

From my point of view, this approach has some benefits:

• compile a static list of tool webservices that need this redirection and maintain the file in puppet (or any other repo).
• we "snapshot" this static list on the day we officially drop support for adding new tools webservices to the legacy URL scheme.
• on D day, we point tools.wmflabs.org to the new redirection server, and leave tools frontproxy only serving toolforge.org.
• from this day on new tool webservices will get only the new domain. Since the list of legacy redirections is static, and we control it, we stop right away from adding more webservices to the old URL scheme.
• we add SSL support in the new legacy redirector for tools.wmflabs.org and we drop it in the tools front proxy for tools.wmflabs.org.
• pretty simple setup, we don't need any additional migration steps or any other magic or coding in tools-webservice or any other mechanism (front proxy, k8s ingress, etc).

In T247236#5955523, @aborrero wrote:

I saw a couple of patches by @bd808 :

• https://gerrit.wikimedia.org/r/c/operations/software/tools-webservice/+/578413 Add support for redirecting to toolforge.org
• https://gerrit.wikimedia.org/r/c/operations/puppet/+/578406 toolforge: support canonical redirects in urlproxy

The approach in those patches differs a bit from what I had in mind, so I guess this topic worth a discussion in the team.

My intent with these patches is to enable an opt-in period for forcing traffic to the toolforge.org host for a given tool. I have played with migrating a number of my tools and found that some of them (not many, but some) needed code or configuration changes in order to work as intended. For tools where a code change is needed, I would like to introduce this method of allowing the tool maintainer to make those changes and then force all visitors into the new toolforge.org domain. This will let people self-migrate to the new scheme during an announced testing + self-migration window. I have documented a manual method for forcing redirection on wikitech, but that method will not survive a webservice stop; webservice start hard restart. It also only works for Kubernetes hosted webservices. Adding the --canonical (open to discussion if this name makes sense) option to webservice makes the redirection less manual, easier to apply during a hard restart cycle, and available to those tools using the grid engine backend.

My original idea was to introduce a new server doing only the redirection, external to the tools front proxy or the k8s cluster.

I agree that this is the desired end state of the migration. This setup makes routing the tools.wmflabs.org URLs work very similarly to the routing that we have for the even older legacy toolserver.org URLs. It will let us stop new tools.wmflabs.org/$tool routes from being created which is also an important part of a complete migration.

So I guess I'm suggesting that we are both correct. My changes are intended to be temporary bridge between the legacy state of only supporting tools.wmflabs.org/$tool and the desired future stable state of preserving routing for the historic URLs while only allowing creation of new URLs under the $tool.toolforge.org scheme that @aborrero's proposal would handle.

Agreed on everything!

Next steps as discussed in our team meeting:

• merge everything
• deploy webservice, rebuild docker images
• test everything
• work on a News wikitech page to announce the new domain to our users, with a timeline and a description on how to test stuff
• we shouldn't enable the final setup (separating tools.wmflabs.org from the front proxy) until the transition period is over

Change 583593 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] toolforge: introduce role/profile for legacy URL redirector

https://gerrit.wikimedia.org/r/583593

Change 583593 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] toolforge: introduce role/profile for legacy URL redirector

https://gerrit.wikimedia.org/r/583593

Change 584575 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] toolforge: legacy_redirector: fix toolsbeta support

https://gerrit.wikimedia.org/r/584575

Change 584575 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] toolforge: legacy_redirector: fix toolsbeta support

https://gerrit.wikimedia.org/r/584575

Change 584582 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] toolforge: legacy redirector: fix/improve table of allowed tools

https://gerrit.wikimedia.org/r/584582

Change 584582 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] toolforge: legacy redirector: fix/improve table of allowed tools

https://gerrit.wikimedia.org/r/584582

Change 588380 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] toolforge: legacy URLs: use HTTP 307/308 for the redirects

https://gerrit.wikimedia.org/r/588380

Change 588380 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] toolforge: legacy URLs: use HTTP 307/308 for the redirects

https://gerrit.wikimedia.org/r/588380

Mentioned in SAL (#wikimedia-cloud) [2020-06-17T10:40:10Z] <arturo> created VM tools-legacy-redirector, with the corresponding puppet prefix (T247236, T234617)

Change 606155 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] toolforge: legacy-redirector: use specific certificate

https://gerrit.wikimedia.org/r/606155

Change 606155 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] toolforge: legacy-redirector: use specific certificate

https://gerrit.wikimedia.org/r/606155

Change 606163 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] toolforge: legacy-redirector: declare explicitly sslcert::dhparam

https://gerrit.wikimedia.org/r/606163

Change 606163 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] toolforge: legacy-redirector: declare explicitly sslcert::dhparam

https://gerrit.wikimedia.org/r/606163

Mentioned in SAL (#wikimedia-cloud) [2020-07-06T11:50:24Z] <arturo> associate floating IP address 185.15.56.60 to tools-legacy-redirector (T247236)

Mentioned in SAL (#wikimedia-cloud) [2020-07-06T11:54:01Z] <arturo> briefly point DNS tools.wmflabs.org A record to 185.15.56.60 (tools-legacy-redirector) and then switch back to 185.15.56.11 (tools-proxy-05). The legacy redirector does HTTP/307 (T247236)

In T247236#6281095, @aborrero wrote:

In T254857: Toolforge: domain migration: track down missing OAuth grants we are tracking known tools with missing OAuth grants for the new domain.
In T244473: Toolforge: both domains in parallel and OAuth it was suggested a maintenance script could be created to automatically refresh grants with the new URL.

Given the original deadline, plus the extended deadline has past, I propose the following:

• consider that tools that didn't migrate already are not under proactive maintenance by the community (but might be still useful)
• create such maintenance script (I guess by @Tgr, @bd808 or any other OAuth admin?) and have it ready to run.
• Enable the forced legacy redirection. Run the maintenance script. Proposed date/time: 2020-07-14 14:00Z
• collect feedback from the community on what do/doesn't work and figure out what to do with the (very few) cases a tool ended in a broken state.

Before the proposed date, I will run a few quick tests with the legacy redirector to make sure it will work as expected when finally introduced for real.

I talked with @bd808 and we decided to do this change tomorrow. Will send and announcement via email.

Mentioned in SAL (#wikimedia-cloud) [2020-07-07T09:59:39Z] <arturo> point DNS tools.wmflabs.org A record to 185.15.56.60 (tools-legacy-redirector) (T247236)