Page MenuHomePhabricator

Update webservicemonitor to work without dynamicproxy
Closed, DeclinedPublic

Description

In https://gerrit.wikimedia.org/r/c/operations/software/tools-webservice/+/697096 webservices in the grid will be created as Kubernetes services and ingresses, with the eventual goal of removing dynamicproxy entirely. Once that is live and all webservices have been restarted to create Kubernetes objects we need to modify webservicemonitor.py to read the list of running web services from Kubernetes.

This can be done with creating a prometheus-style external kubernetes certificate for the monitor to access kubernetes directly or by having a special tool with extra privs query that and expose the information via HTTP, I prefer the latter as that will have its certificates renewed automatically.

Event Timeline

Can the API exposing the data you need for this be added to the k8s-status tool? I don't have any objections to another tool specifically for this either, but we have that one today which I think has all the rights you would need.

Can the API exposing the data you need for this be added to the k8s-status tool? I don't have any objections to another tool specifically for this either, but we have that one today which I think has all the rights you would need.

It would, but that given the scope of the tools are fairly different (while they both load it from kubernetes, one is about grid tools and one is about kubernetes tool) and k8s-status sometimes takes a while to load all of its data I'd like to have this tool on a different pool of WSGI workers.

T233372: Create a "novaobserver" equivalent for Toolforge Kubernetes cluster inspection is the magic that you need for a tool account to get the read-only "observer" ability that k8s-status has. There is a "TODO: document the observer role and how it gets granted to a tool." at https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Kubernetes/RBAC_and_PSP#Observer_role that maybe @Bstorm could help flesh out as part of this.

@Majavah you have enough privs to self-serve that process if you want. I think there's a ticket around you created for it?

The general idea is that it gives a non-default service account full cluster read (with a few extras like nodes). You just want to make sure your pods launch as that service account and not default to have the access.

@Majavah you have enough privs to self-serve that process if you want. I think there's a ticket around you created for it?

I didn't make a dedicated ticket for the observer access, I pointed people on -admin to this task if they had any objections last week. I'll self-service the access in a moment given there were none and this should be relatively uncontroversial.

Mentioned in SAL (#wikimedia-cloud) [2021-06-16T18:34:21Z] <majavah> enable kubernetes cluster monitor access for grid-webservices tool T284564

Yeah, especially since you have that access from your user account already, it isn't an issue. For the most part, there is little in the observer access that is a problem for anybody to have. It's just not granted normally just to keep things following the principle of least privilege and prevent excessive use of the APIs because most users don't actually *need* it.

Change 700883 had a related patch set uploaded (by Majavah; author: Majavah):

[cloud/toolforge/grid-webservices@master] Add grid-webservices application

https://gerrit.wikimedia.org/r/700883

Change 700883 merged by Majavah:

[cloud/toolforge/grid-webservices@master] Add grid-webservices application

https://gerrit.wikimedia.org/r/700883

Change 703189 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/software/tools-manifest@master] Read running tools from grid-webservices tool

https://gerrit.wikimedia.org/r/703189

nskaggs moved this task from Backlog to In Progress on the Toolforge board.