Page MenuHomePhabricator
Create Task
Maniphest T242719

https://tools.wmflabs.org/{toolname} no longer redirects to https://tools.wmflabs.org/{toolname}/ on new k8s cluster
Closed, ResolvedPublic
Actions

Description

On the old k8s cluster:

user@dev ~> curl -I 'https://tools.wmflabs.org/extreg-wos'
HTTP/2 301 
server: nginx/1.14.2
date: Tue, 14 Jan 2020 09:22:21 GMT
content-type: text/html; charset=utf-8
content-length: 281
location: https://tools.wmflabs.org/extreg-wos/
strict-transport-security: max-age=86400
x-clacks-overhead: GNU Terry Pratchett
content-security-policy-report-only: default-src 'self' 'unsafe-eval' 'unsafe-inline' blob: data: filesystem: mediastream: wikibooks.org *.wikibooks.org wikidata.org *.wikidata.org wikimedia.org *.wikimedia.org wikinews.org *.wikinews.org wikipedia.org *.wikipedia.org wikiquote.org *.wikiquote.org wikisource.org *.wikisource.org wikiversity.org *.wikiversity.org wikivoyage.org *.wikivoyage.org wiktionary.org *.wiktionary.org *.wmflabs.org wikimediafoundation.org mediawiki.org *.mediawiki.org wss://tools.wmflabs.org; report-uri https://tools.wmflabs.org/csp-report/collect;

On the new k8s cluster it serves the page rather than redirecting

user@dev ~> curl -I 'https://tools.wmflabs.org/extreg-wos'
HTTP/2 200 
server: nginx/1.14.2
date: Tue, 14 Jan 2020 09:24:00 GMT
content-type: text/html; charset=utf-8
content-length: 217628
vary: Accept-Encoding
strict-transport-security: max-age=86400
x-clacks-overhead: GNU Terry Pratchett
content-security-policy-report-only: default-src 'self' 'unsafe-eval' 'unsafe-inline' blob: data: filesystem: mediastream: wikibooks.org *.wikibooks.org wikidata.org *.wikidata.org wikimedia.org *.wikimedia.org wikinews.org *.wikinews.org wikipedia.org *.wikipedia.org wikiquote.org *.wikiquote.org wikisource.org *.wikisource.org wikiversity.org *.wikiversity.org wikivoyage.org *.wikivoyage.org wiktionary.org *.wiktionary.org *.wmflabs.org wikimediafoundation.org mediawiki.org *.mediawiki.org wss://tools.wmflabs.org; report-uri https://tools.wmflabs.org/csp-report/collect;

This is a problem because relative URLs (as can be seen on the extreg-wos tool, which I'm leaving using the new k8s cluster) are now broken.

Details

ProjectBranchLines +/-Subject
operations/software/tools-webservicemaster+4 -1kubernetes: ingress: introduce annotation to redirect the webapp root
Customize query in gerrit

Related Objects
Search...

Event Timeline

Legoktm created this task.Jan 14 2020, 9:25 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 14 2020, 9:25 AM
Legoktm merged a task: T242720: Regression in URL path handling on 2020 (?) kubernetes.Jan 14 2020, 9:50 AM
Legoktm added a subscriber: Magnus.
aborrero added a subscriber: aborrero.Jan 14 2020, 10:28 AM

I'm not sure I understand the curl output you pasted.

I've done some tests with the browser to try to understand the issue.

In both cases, the HTML code is the same:

<link rel="stylesheet" type="text/css" href="static/wos.css">

which seems correct.

Now I'm trying to understand where is this redirection supposed to happen in the first place.

aborrero claimed this task.Jan 14 2020, 10:28 AM
aborrero triaged this task as High priority.
aborrero moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.
aborrero added a comment.Jan 14 2020, 11:16 AM

What happens with other tools?

<link rel="stylesheet" type="text/css" href="https://tools.wmflabs.org/sge-status/assets/site.css">
<link rel="stylesheet" href="https://tools-static.wmflabs.org/cdnjs/ajax/libs/twitter-bootstrap/3.1.1/css/bootstrap.min.css">

This can be seen in the front proxy (dynamicproxy) logs:

tools.wmflabs.org x.x.x.x - - [14/Jan/2020:10:39:32 +0000] "GET /contact HTTP/2.0" 301 273 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0" (backend 192.168.76.212:8000)
tools.wmflabs.org x.x.x.x - - [14/Jan/2020:10:39:33 +0000] "GET /contact/ HTTP/2.0" 200 1234 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0" (backend 192.168.76.212:8000)

So far I've been not able to discover what is doing that first 301.

aborrero added a comment.Jan 14 2020, 11:39 AM

This tool is also running in the grid and gets the redirection:

tools.wmflabs.org x.x.x.x - - [14/Jan/2020:11:35:10 +0000] "GET /dow HTTP/2.0" 301 0 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0" (backend 172.16.1.89:59711)
tools.wmflabs.org x.x.x.x - - [14/Jan/2020:11:35:11 +0000] "GET /dow/ HTTP/2.0" 200 1002 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0" (backend 172.16.1.89:59711)

Again, this suggest the front proxy is the one doing this redirection, but I can't find where. Wait, what about the LUA code?

bd808 added a subscriber: bd808.Jan 14 2020, 4:17 PM

I think we could do this on the new cluster by adding an annotation to the Ingress object. I have not tried it yet, but something like nginx.ingress.kubernetes.io/configuration-snippet: rewrite ^(/TOOL_NAME)$ $1/ redirect; is suggested by https://github.com/kubernetes/ingress-nginx/issues/646.

I am not sure that the legacy cluster normally redirects /tool to /tool/. See https://tools.wmflabs.org/sge-status as an example. That is a grid engine backend rather than a Kubernetes backend however, so I guess it is possible that the legacy k8s cluster does somehow redirect.

bd808 added a comment.Jan 14 2020, 4:26 PM

The old redirect behavior comes from the admin tool when the front proxy redirects there for a 404:

if ( $notFoundHandler && $info['name'] !== false ) {
        // Route was for a known tool
        if ( $uri === "/{$info['name']}" ) {
                // Redirect bare /<toolname> to /<toolname>
                $this->redirect( "/{$info['name']}/", 301 );
        } else {
                // The tool's service must be down. Send a 503 response.
                $errorCode = '503';
        }
}

I apparently did not add this same logic to the fourohfour handler tool.

Bstorm added a subscriber: Bstorm.Jan 14 2020, 4:39 PM
In T242719#5802299, @bd808 wrote:

I apparently did not add this same logic to the fourohfour handler tool.

Would that work there? Seems like the annotation (which is another webservice thing, I imagine) might be the way to go on a quick reading.

bd808 added a comment.Jan 14 2020, 4:49 PM
In T242719#5802345, @Bstorm wrote:
In T242719#5802299, @bd808 wrote:

I apparently did not add this same logic to the fourohfour handler tool.

Would that work there? Seems like the annotation (which is another webservice thing, I imagine) might be the way to go on a quick reading.

I think it would be possible to add to the fourohfour tool. I would also like to play with an Ingress config solution. Going with a belt and suspenders approach of both might not be horrible actually. If we get it working in the Ingress in a reasonable way then the fourohfour logic should never fire, but would be there if we decide to change ingress systems at some point.

aborrero removed aborrero as the assignee of this task.Jan 14 2020, 4:52 PM
aborrero moved this task from Doing to Soon! on the cloud-services-team (Kanban) board.
bd808 assigned this task to aborrero.Jan 14 2020, 5:23 PM
bd808 moved this task from Soon! to Doing on the cloud-services-team (Kanban) board.
Lucas_Werkmeister_WMDE mentioned this in T242848: extreg-wos tool (Extension Registration Wall of Superpowers) broken (500 Internal Server Error).Jan 15 2020, 12:39 PM
bd808 added a comment.Jan 16 2020, 7:08 AM

I manually added the annotation I suggested in T242719#5802258 to the Ingress object for extreg-wos (/usr/bin/kubectl edit ingress extreg-wos) and it appears to be doing the redirect as hoped. This should be a simple addition to the webservice generated object.

Default ingress
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /extreg-wos/$2
  creationTimestamp: "2020-01-16T06:52:48Z"
  generation: 1
  labels:
    name: extreg-wos
    toolforge: tool
    tools.wmflabs.org/webservice: "true"
    tools.wmflabs.org/webservice-version: "1"
  name: extreg-wos
  namespace: tool-extreg-wos
  resourceVersion: "14574323"
  selfLink: /apis/extensions/v1beta1/namespaces/tool-extreg-wos/ingresses/extreg
-wos
  uid: f634d856-b986-4a03-9800-5d3c3e59084f
spec:
  rules:
  - host: tools.wmflabs.org
    http:
      paths:
      - backend:
          serviceName: extreg-wos
          servicePort: 8000
        path: /extreg-wos(/|$)(.*)
status:
  loadBalancer: {}
Edited ingress
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/configuration-snippet: |
      rewrite ^(/extreg-wos)$ $1/ redirect;
    nginx.ingress.kubernetes.io/rewrite-target: /extreg-wos/$2
  creationTimestamp: "2020-01-16T06:52:48Z"
  generation: 1
  labels:
    name: extreg-wos
    toolforge: tool
    tools.wmflabs.org/webservice: "true"
    tools.wmflabs.org/webservice-version: "1"
  name: extreg-wos
  namespace: tool-extreg-wos
  resourceVersion: "14576012"
  selfLink: /apis/extensions/v1beta1/namespaces/tool-extreg-wos/ingresses/extreg-wos
  uid: f634d856-b986-4a03-9800-5d3c3e59084f
spec:
  rules:
  - host: tools.wmflabs.org
    http:
      paths:
      - backend:
          serviceName: extreg-wos
          servicePort: 8000
        path: /extreg-wos(/|$)(.*)
status:
  loadBalancer: {}
gerritbot added a comment.Jan 16 2020, 11:39 AM

Change 565259 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/software/tools-webservice@master] kubernetes: ingress: introduce annotation to redirect the webapp root

https://gerrit.wikimedia.org/r/565259

gerritbot added a project: Patch-For-Review.Jan 16 2020, 11:39 AM
Stashbot added a comment.Jan 16 2020, 12:07 PM

Mentioned in SAL (#wikimedia-cloud) [2020-01-16T12:07:18Z] <arturo> live-hack tools-webservice in tools-sgebastion-04 to test https://gerrit.wikimedia.org/r/c/565259 (T242719)

aborrero added a comment.Jan 16 2020, 12:40 PM
In T242719#5808752, @gerritbot wrote:

Change 565259 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/software/tools-webservice@master] kubernetes: ingress: introduce annotation to redirect the webapp root

https://gerrit.wikimedia.org/r/565259

I tested this patch in the 'test' tool in toolsbeta:

aborrero@toolsbeta-sgebastion-04:~$ curl -IL toolsbeta.wmflabs.org/test
HTTP/1.1 302 Moved Temporarily
Server: nginx/1.14.2
Date: Thu, 16 Jan 2020 12:36:54 GMT
Content-Type: text/html
Content-Length: 151
Connection: keep-alive
Location: http://toolsbeta.wmflabs.org/test/
X-Clacks-Overhead: GNU Terry Pratchett
Content-Security-Policy-Report-Only: default-src 'self' 'unsafe-eval' 'unsafe-inline' blob: data: filesystem: mediastream: wikibooks.org *.wikibooks.org wikidata.org *.wikidata.org wikimedia.org *.wikimedia.org wikinews.org *.wikinews.org wikipedia.org *.wikipedia.org wikiquote.org *.wikiquote.org wikisource.org *.wikisource.org wikiversity.org *.wikiversity.org wikivoyage.org *.wikivoyage.org wiktionary.org *.wiktionary.org *.wmflabs.org wikimediafoundation.org mediawiki.org *.mediawiki.org wss://tools.wmflabs.org; report-uri https://tools.wmflabs.org/csp-report/collect;

HTTP/1.1 200 OK
Server: nginx/1.14.2
Date: Thu, 16 Jan 2020 12:36:54 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 29
Connection: keep-alive
X-Clacks-Overhead: GNU Terry Pratchett
Content-Security-Policy-Report-Only: default-src 'self' 'unsafe-eval' 'unsafe-inline' blob: data: filesystem: mediastream: wikibooks.org *.wikibooks.org wikidata.org *.wikidata.org wikimedia.org *.wikimedia.org wikinews.org *.wikinews.org wikipedia.org *.wikipedia.org wikiquote.org *.wikiquote.org wikisource.org *.wikisource.org wikiversity.org *.wikiversity.org wikivoyage.org *.wikivoyage.org wiktionary.org *.wiktionary.org *.wmflabs.org wikimediafoundation.org mediawiki.org *.mediawiki.org wss://tools.wmflabs.org; report-uri https://tools.wmflabs.org/csp-report/collect;

In server side:

172.16.0.138 - [172.16.0.138] - - [16/Jan/2020:12:39:06 +0000] "HEAD /test HTTP/1.1" 302 0 "-" "curl/7.52.1" 157 0.000 [tool-test-test-8000] [] - - - - a84484b186320c75d638224a72b888d9
192.168.23.192 - [192.168.23.192] - - [16/Jan/2020:12:39:06 +0000] "HEAD /test/ HTTP/1.1" 200 0 "-" "curl/7.52.1" 159 0.009 [tool-test-test-8000] [] 192.168.222.161:8000 0 0.008 200 5b5054a1f1246270179791e5337c59b9
gerritbot added a comment.Jan 16 2020, 6:51 PM

Change 565259 merged by Bstorm:
[operations/software/tools-webservice@master] kubernetes: ingress: introduce annotation to redirect the webapp root

https://gerrit.wikimedia.org/r/565259

Bstorm added a comment.Jan 17 2020, 3:14 PM

Note: because of a change in the way restarts work (they are lighter now and don't destroy ingresses), anyone looking to use the new ingress setting should webservice stop and then webservice start --backend kubernetes <whatever>

bd808 mentioned this in T214278: Quickstatements, "backend is overloaded".Jan 22 2020, 8:01 PM
aborrero closed this task as Resolved.Jan 23 2020, 1:02 PM

This seems fixed. Closing task now. Please reopen if required.

LucasWerkmeister added a subscriber: LucasWerkmeister.Feb 2 2020, 4:29 PM

https://tools.wmflabs.org/docker-registry doesn’t redirect to https://tools.wmflabs.org/docker-registry/, causing scripts to be loaded from the wrong location:

Loading failed for the <script> with source “https://tools.wmflabs.org/scripts/vendor.js”. docker-registry:16:1
Loading failed for the <script> with source “https://tools.wmflabs.org/scripts/docker-registry-ui-static.js”.

The result is a blank page. Does the tool need to be restarted?

Stashbot added a comment.Feb 3 2020, 2:19 AM

Mentioned in SAL (#wikimedia-cloud) [2020-02-03T02:19:36Z] <bd808> Hard restart of webservice to pick up redirect logic (T242719)

bd808 mentioned this in T246562: 'wdumps' tool does not contain the expected www/python/src/app.py entrypoint script.Mar 3 2020, 4:25 PM
zhuyifei1999 merged a task: T174498: Monumental tool failed to load bundle.min.js.Apr 8 2020, 9:35 AM
zhuyifei1999 added subscribers: Quiddity, simon04, Automatik and 4 others.
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL