Toolforge ingress: create a default landing page for unknown/default URLs
Closed, ResolvedPublic
Actions

Description

Once Toolforge k8s is working with ingress and we have the routing in place, we will need a default route for URL not matched by any ingress rule.

My proposal is to direct traffic to this URL to a pod controlled by us.
This pod can either:

have a static HTML web page with some error message or further instructions
redirect to wikitech or some other external landing page

The ingress definition for this could be something similar to this:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: toolforge-default-ingress-route
spec:
  rules:
  - host: toolforge.org
    http:
      paths:
      - backend:
          serviceName: toolforge-default-landing-page
          servicePort: 80

(exact setting TBD, depending on how the rest of the ingress is configured)

Foot note: I always loved the github 404 page:

Details

	Project	Branch	Lines +/-	Subject
	operations/puppet	production	+2 -0	toolforge: new k8s: specify default backend for nginx-ingress

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
		Restricted Task
Resolved	Bstorm	T246122 Upgrade the Toolforge Kubernetes cluster to v1.16
Resolved	Bstorm	T246553 "teg" tool needs a higher Services quota to migrate to 2020 Kubernetes cluster
Open	None	T246592 Establish processes for quota increases in Toolforge Kubernetes
		Restricted Task
Resolved	bd808	T232536 Toolforge Kubernetes internal API down, causing `webservice` and other tooling to fail
Resolved	Bstorm	T236565 "tools" Cloud VPS project jessie deprecation
Resolved	aborrero	T101651 Set up toolsbeta more fully to help make testing easier
Stalled	None	T166949 Homedir/UID info breaks after a while in Tools Kubernetes (can't read replica.my.cnf)
Open	Bstorm	T246059 Add admin account creation to maintain-kubeusers
Resolved	Bstorm	T154504 Make webservice backend default to kubernetes
Open	None	T245230 Investigate cpu/ram requests and limits for DaemonSets pods
Resolved	Bstorm	T214513 Deploy and migrate tools to a Kubernetes v1.15 or newer cluster
Resolved	aborrero	T215531 Deploy upgraded Kubernetes to toolsbeta
Resolved	aborrero	T228500 Toolforge: evaluate ingress mechanism
Open	None	T180262 Reduce byte-size of Toolforge 404 page
Resolved	bd808	T107697 Extend 'tool not found' 404 page
Resolved	aborrero	T234032 Toolforge ingress: create a default landing page for unknown/default URLs

Event Timeline

aborrero created this task.Sep 27 2019, 11:28 AM

aborrero removed aborrero as the assignee of this task.Sep 27 2019, 12:46 PM

aborrero triaged this task as High priority.

aborrero lowered the priority of this task from High to Medium.

aborrero moved this task from Inbox to Soon! on the cloud-services-team (Kanban) board.

aborrero mentioned this in T234037: Toolforge ingress: decide on final layout of north-south proxy setup.Oct 14 2019, 4:13 PM

aborrero mentioned this in T237443: toolsbeta: new k8s: deploy a front proxy (dynamicproxy).Nov 6 2019, 1:24 PM

@bd808 has created https://phabricator.wikimedia.org/source/tool-fourohfour/ which we could apply as a default route in the ingress.

@bd808: is that deployed in toolsbeta already? If not we can try to figure that out as well (the nonsense in ldap that is).

Perhaps if we can leave off the "host" field, it will work for both toolforge.org and wmflabs.org?

Found T107697: Extend 'tool not found' 404 page and T180262: Reduce byte-size of Toolforge 404 page
Just for reference and possible connection/closure.

In T234032#5649174, @Bstorm wrote:

@bd808: is that deployed in toolsbeta already? If not we can try to figure that out as well (the nonsense in ldap that is).

I haven't tried to put it into toolsbeta yet. I can take a look at that and maybe as part of the process figure out if there is a nicer way to deal with the tool account creation in that environment. Setting up a Striker deployment pointing to the toolsbeta project would be the ideal thing to do. We would need to put that Sriker deployment in a trusted network (meaning the public vlan functionally) for me to be comfortable with the LDAP auth and writing operations.

We could turn fourohfour into a "service" rather that a tool too if that seems like a nicer way to manage it. It is a fairly simple flask app with ldap3 & pyyaml dependencies.

bd808 mentioned this in T180262: Reduce byte-size of Toolforge 404 page.Nov 9 2019, 10:24 PM

bd808 added a subtask: T180262: Reduce byte-size of Toolforge 404 page.

bd808 removed a subtask: T180262: Reduce byte-size of Toolforge 404 page.

bd808 added a parent task: T180262: Reduce byte-size of Toolforge 404 page.

bd808 mentioned this in T107697: Extend 'tool not found' 404 page.Nov 10 2019, 4:52 AM

bd808 added a parent task: T107697: Extend 'tool not found' 404 page.

I made a script on cloudcontrol1003 that can be used to make all the LDAP things for a toolsbeta tool:

$ ssh cloudcontrol1003.wikimedia.org
$ cd /home/bd808/projects/toolsbeta
$ python3 toolsctl.py add fourohfour -m uid=bd808,ou=people,dc=wikimedia,dc=org -m uid=bstorm,ou=people,dc=wikimedia,dc=org -m uid=aborrero,ou=people,dc=wikimedia,dc=org
$ python3 toolsctl.py list
toolsbeta.fourohfour
    uid=bd808,ou=people,dc=wikimedia,dc=org
    uid=bstorm,ou=people,dc=wikimedia,dc=org
    uid=aborrero,ou=people,dc=wikimedia,dc=org
toolsbeta.toolschecker
    uid=scfc,ou=people,dc=wikimedia,dc=org
toolsbeta.test2
    uid=scfc,ou=people,dc=wikimedia,dc=org
    uid=toolsbeta.test,ou=people,ou=servicegroups,dc=wikimedia,dc=org
toolsbeta.test
    uid=andrew,ou=people,dc=wikimedia,dc=org
    uid=bstorm,ou=people,dc=wikimedia,dc=org
    uid=petrb,ou=people,dc=wikimedia,dc=org
toolsbeta.admin
    uid=aborrero,ou=people,dc=wikimedia,dc=org
    uid=andrew,ou=people,dc=wikimedia,dc=org
    uid=bd808,ou=people,dc=wikimedia,dc=org
    uid=bstorm,ou=people,dc=wikimedia,dc=org
    uid=chicocvenancio,ou=people,dc=wikimedia,dc=org
    uid=scfc,ou=people,dc=wikimedia,dc=org
    uid=valhallasw,ou=people,dc=wikimedia,dc=org
    uid=zhuyifei1999,ou=people,dc=wikimedia,dc=org

The script does not have a lot of input validation, so be gentle if you decide to use it to create more tools in toolsbeta.

@Bstorm I seem to have the fourohfour tool deployed in toolsbeta, but I'm not sure how to actually connect to it:

$ toolsbeta-sgebastion-04.toolsbeta.eqiad.wmflabs
$ become fourohfour
$ mkdir -p www/python
$ cd www/python/
$ git clone https://phabricator.wikimedia.org/source/tool-fourohfour.git src
$ /usr/bin/kubectl config current-context
toolforge
$ /usr/bin/kubectl get pods
No resources found.
$ webservice --backend=kubernetes python3.5 shell
$ cd www/python/
$ python3 -mvenv venv
$ venv/bin/pip3 install -U pip
...
Successfully installed pip-19.3.1
$ venv/bin/pip3 install -r src/requirements.txt
...
Successfully installed Jinja2-2.10.3 MarkupSafe-1.1.1 Werkzeug-0.16.0 cachelib-0.1 click-7.0 flask-1.1.1 fuzzywuzzy-0.17.0 itsdangerous-1.1.0 ldap3-2.6.1 pyasn1-0.4.7 python-levenshtein-0.12.0 pyyaml-5.1.2 redis-3.3.11
$ exit
$ webservice --backend=kubernetes python3.5 start
Starting webservice....
$ /usr/bin/kubectl get po
NAME                          READY   STATUS    RESTARTS   AGE
fourohfour-66bf569f4f-zkzhg   1/1     Running   0          8s
$ /usr/bin/kubectl get service
NAME         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
fourohfour   ClusterIP   10.100.6.148   <none>        8000/TCP   26s
$ /usr/bin/kubectl get ingress
NAME         HOSTS                   ADDRESS   PORTS   AGE
fourohfour   toolsbeta.wmflabs.org             80      34s
$ curl https://toolsbeta.wmflabs.org/fourohfour/
^C
$ curl http://10.100.6.148:8000/fourohfour/
^C

toolsbeta frontproxy (dynamicproxy) doesn't support SSL. Try this:

aborrero@toolsbeta-sgebastion-04:~ $ curl http://toolsbeta.wmflabs.org/fourohfour/
<!DOCTYPE HTML>
<html lang="en">
[...]
<h1>Not found</h1>
<p>The URL you have requested, <code>http://toolsbeta.wmflabs.org/</code>, doesn't seem to actually exist.</p>
[...]

note http:// vs https:// in your example above @bd808 .
The IP address 10.100.6.148 not sure what means, but that's not how you are supposed to interact with a tool webservice running in the k8s cluster.

In T234032#5649176, @Bstorm wrote:

Perhaps if we can leave off the "host" field, it will work for both toolforge.org and wmflabs.org?

I'm thinking that due to our ingress-admission-controller policy, the ingress object you are proposing may not be allowed.

We may need to deploy an ingress object by puppet and load it as part of the initial deployment. Then, with hiera (and a default) we let the yaml know which tool is handling the 404 (or default route).

I can work on this if you want.

On a second thought, and investigating a bit more, this may be as simple as using the nginx-ingress --default-backend-service flag:

[...]
    spec:
      containers:
      - args:
        - /nginx-ingress-controller
        - --http-port=8080
        - --configmap=$(POD_NAMESPACE)/nginx-configuration
        - --default-backend-service=tool-fourohfour/fourohfour
[...]

Then:

aborrero@toolsbeta-sgebastion-04:~ $ curl http://toolsbeta.wmflabs.org/random
<!DOCTYPE HTML>
[..]
<h1>Not found</h1>
<p>The URL you have requested, <code>http://toolsbeta.wmflabs.org/random</code>, doesn't seem to actually exist.</p>
[..]

I will write a patch.

Change 550347 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] toolforge: new k8s: specify default backend for nginx-ingress

https://gerrit.wikimedia.org/r/550347

gerritbot added a project: Patch-For-Review.Nov 11 2019, 6:31 PM

The logs in the ingress pod is clearly differentiated too:

172.16.0.151 - [172.16.0.151] - - [11/Nov/2019:18:40:48 +0000] "GET /random HTTP/1.1" 404 1975 "-" "curl/7.52.1" 160 0.963 [upstream-default-backend] [] 192.168.44.217:8000 1975 0.964 404 f3eda38da0ea2994376b7f41c520536b
192.168.44.192 - [192.168.44.192] - - [11/Nov/2019:18:40:57 +0000] "GET /fourohfour HTTP/1.1" 404 1969 "-" "curl/7.52.1" 168 0.655 [tool-fourohfour-fourohfour-8000] [] 192.168.44.217:8000 1969 0.656 404 615a146292e1b9df1c8ab1994422566a

Note the [upstream-default-backend] tag.

Change 550347 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] toolforge: new k8s: specify default backend for nginx-ingress

https://gerrit.wikimedia.org/r/550347

This seems solved. Thanks @bd808 and @Bstorm !! Closing task now, please reopen if required.

Maintenance_bot removed a project: Patch-For-Review.Nov 22 2019, 6:10 PM

Toolforge ingress: create a default landing page for unknown/default URLsClosed, ResolvedPublicActions

Description

Details

Related ObjectsSearch...

Event Timeline

Toolforge ingress: create a default landing page for unknown/default URLs
Closed, ResolvedPublic
Actions

Related Objects
Search...