It is possible that this will be resolved with T317787

In T326164#8497724, @rook wrote:

It is possible that this will be resolved with T317787

Not really. :/, this is the new Jupyterhub rbac.

There are some bugs that have been fixed in jupyterhub 3.1.0, and maybe this is one of them.

In particular we see

[W 2023-01-04 12:44:25.736 JupyterHub auth:298] Service Server at /user/VRook_%28WMF%29/ requested scopes read:users:groups!user,read:users:name!user,access:servers!user=VRook_(WMF),access:servers!server=VRook_(WMF)/ for user VRook_(WMF), granting only read:users:groups!user,read:users:name!user,access:servers!server=VRook_(WMF)/.

Note access:servers!user=VRook_(WMF) dropped. I think this was fixed in https://github.com/jupyterhub/jupyterhub/pull/4063, and maybe that scope includes delete:servers.

Docs seem to indicate it doesn't, however. So the task is to figure out how do we configure jupyterhub to add delete:servers!user=VRook_(WMF) or delete:servers!server=VRook_(WMF)/ to the scope used by the hub page.

In T326164#8498932, @Chicocvenancio wrote:

In T326164#8497724, @rook wrote:

It is possible that this will be resolved with T317787

Not really. :/, this is the new Jupyterhub rbac.

Interesting, I was making this guess as I can still delete my server in minikube, and the major difference is that we are not introducing Pod Security Policies in minikube. Begging the question of if not psp, why is it working in minikube. Perhaps we are configuring something else differently in prod that we could adjust?

In T326164#8498955, @rook wrote:

In T326164#8498932, @Chicocvenancio wrote:

In T326164#8497724, @rook wrote:

It is possible that this will be resolved with T317787

Not really. :/, this is the new Jupyterhub rbac.

Interesting, I was making this guess as I can still delete my server in minikube, and the major difference is that we are not introducing Pod Security Policies in minikube. Begging the question of if not psp, why is it working in minikube. Perhaps we are configuring something else differently in prod that we could adjust?

Beyond kubernetes the db is very different in minikube, so maybe something is still borked with user tokens in the prod db. Its a good place to investigate, I'll see if I have some time later today.

Beyond kubernetes the db is very different in minikube, so maybe something is still borked with user tokens in the prod db. Its a good place to investigate, I'll see if I have some time later today.

I would guess it is not db related, unless it is an sqlite vs mysql kind of thing. I created a parallel trove DB and pointed a magnum cluster at it today, so all the data is new. I'm seeing the same "Action is not authorized with current scopes" in the magnum deployed PAWS.

If this is related to:

[W 2023-01-12 15:41:21.308 JupyterHub base:89] Blocking Cross Origin API request.  Referer: https://hub.paws-dev.codfw1dev.wmcloud.org/hub/home, Host: hub.paws-dev.codfw1dev.wmcloud.org, Host URL: http://hub.paws-dev.codfw1dev.wmcloud.org/hub/
[W 2023-01-12 15:41:21.310 JupyterHub scopes:804] Not authorizing access to /hub/api/users/VRook_(WMF)/server. Requires any of [delete:servers], not derived from scopes []
[W 2023-01-12 15:41:21.310 JupyterHub web:1796] 403 DELETE /hub/api/users/VRook_(WMF)/server (10.100.0.0): Action is not authorized with current scopes; requires any of [delete:servers]
[W 2023-01-12 15:41:21.316 JupyterHub log:186] 403 DELETE /hub/api/users/VRook_(WMF)/server (@10.100.0.0) 18.32ms

it is likely the same issue as T326217

This is resolved when I disable https in paws-dev at the haproxy layer, or convincing the browser to not use https.
http://hub.paws-dev.codfw1dev.wmcloud.org/user/VRook_%28WMF%29/lab
Does seem to work (Firefox likes to send it back to https://hub.paws-dev.codfw1dev.wmcloud.org/user/VRook_%28WMF%29/lab)

As such this does appear to be a duplicate of T326217