Sorry I missed the followup questions, but I think you have solved the mystery. Since I have ipblockexempt it didn't stop me from editing, but its likely my vpn provider was blocked.

Linkback from https://lists.wikimedia.org/hyperkitty/list/wikimedia-l@lists.wikimedia.org/message/DVA7I5TUHSNJAAYKOBD6O77FJT35E2MK/ where I cite this as an example of how WMF has delayed UCoC until the enforcemente guidelines are approved.

In T326164#8498955, @rook wrote:

In T326164#8498932, @Chicocvenancio wrote:

In T326164#8497724, @rook wrote:

It is possible that this will be resolved with T317787

Not really. :/, this is the new Jupyterhub rbac.

Interesting, I was making this guess as I can still delete my server in minikube, and the major difference is that we are not introducing Pod Security Policies in minikube. Begging the question of if not psp, why is it working in minikube. Perhaps we are configuring something else differently in prod that we could adjust?

Some sections of this, mostly host path parts, will become irrelevant with T308873 other parts remain unclear to me on why we need them. Aside from by default needing a psp, are there known benefits of our current setup?

In T326164#8497724, @rook wrote:

It is possible that this will be resolved with T317787

In T326160#8496160, @Bugreporter wrote:

In T326160#8496065, @Dominicbm wrote:

I tried again, and get a different error message:

400 : Bad Request
Invalid client_id parameter value.

I also get the same issue.

(my best guess is the authenticator is storing the client-id in the auth state and that has now changed in the upgrade)

Could you try one more time? I've deleted your user from the db, it should be recreated on login for you.

Its probably related to the user being running at the time of upgrade, let me try to manually clear the tokens in the db.

In T326167#8495997, @rook wrote:

If I'm understanding that network policy shouldn't 172.16.1.129 be permitted under 172.16.0.0/12?

This seems to be the new networkpolicy blocking private ip addresses.

Didn't manage to delete the server, just logged out and back in.

More I'm curious, added me where? I'm not in the list of admins in values.yaml, though I have access to the admin tab (well did until today)

Still never known how I had admin before though. I always assumed it was pulling from cloud vps, but that seems unexpected?

https://jupyterhub.readthedocs.io/en/stable/rbac/roles.html# seems to be the new way of defining admins. I see the admin tab but nothing is there for me right now.

Uhh, it now works... Maybe there was a cached logged in SA for me?

That seems fine to me. This is RBAC internal to jupyterhub to do oauth dances between the various components. I'm reading code/docs trying to figure out what needs to change.

In T326160#8495643, @rook wrote:

Oh, an account that was not logged in at the time of the upgrade can auth. Maybe it is something about accounts that were active during the upgrade?

That may parallel why it seemed to work in minikube.

Seems the oauth tokens for the internal service accounts in jupyterhub don't have the right scopes, weird that this doesn't seem to be mentioned in the upgrade guide.

Why was a rollback necessary? Is trying to fix the new version an option?

Probably a migration was applied with the upgrade and alembic sees it in the table but not in the migration code of the rolled back version? Nuking the db is always an option, will only disrupt currently running pods and log everyone out. There might be a rollback migration code for the new migration, or it might even be compatible needing only the a DELETE in the revision, that would need to be figured it out if keeping the db is desired.

Things are working again for me. I see no 400 errors in the logs for the past 10 minutes.

I confirm its down for me as well.

Reading the github issue I guess moving to notebook-based culling is the way to improve things, I can try to take a look at that.

the hub pod has some in-memory state, and (worse) some shared state with the proxy pod. https://github.com/jupyterhub/zero-to-jupyterhub-k8s/issues/1364 is main upstream issue on this, I think.

https://github.com/toolforge/paws/pull/173 lgtm, Will make the PR upstream.
https://github.com/jupyterhub/jupyter-rsession-proxy/pull/130 is the upstream PR to track, we can use a fork if it takes a while to get merged.

Probably https://github.com/jupyterhub/jupyter-rsession-proxy/blob/master/jupyter_rsession_proxy/__init__.py#L139

I created https://github.com/jupyterhub/jupyter-rsession-proxy/issues/129 upstream for this. I think should be a pretty straightforward change there to get things working. As I see it, NB_USER should be the source of truth for users in jupyterhub land.

I think volume persistence is the biggest blocker here, right?

Feels like your firefox is keeping jupyterlab open after it has a disconnect (Very possibly only a disconnect from the nfs), without telling you that it isn't connected.

I think there is something a bit more esoteric going on. If the the pod wasn't running or didn't have access to the filesystem creating a new file would not be possible. Might be browser session, pod v jupyterhub authentication or websocket connection failing. I would look into the hub and proxy logs for any error around the routes to the user pod and in the user pod itself for any error message the next time this happens.

Have you tried using the absolute path?

Sounds about right to me.

yeah, we haven't used in a few years.

The lack of an uptick in new weekly editors on fawiki during the ip block indicates to me a big difference from ptwiki's implementation. My lack of farsi prevents a deeper investigation, but in ptwiki the interface was hacked to point logged out users at the point of edit. The edit button was changed to point directly to the log-in page and banner explaining the need to create an account linking to the reasoning was put there.

@Isaac indeed. Fixed the link to a permalink to avoid future issues.

6.6.5 was merged.

Other bank transfer methods are also erroring for me.

@EMartin its before we actually make the transaction.

All for the upgrade but just for the record I don't see a path for exploitation here. All requests to openrefine go through the proxy and only authenticated requests pass. And if you're authenticated into a user server, well RCE is the main feature for jupyterhub.

LGTM. Opinionated ways to run the more basic tasks is a good way to reduce the burden on new users.

I guess portaudio19-dev is the missing apt package here.

Thanks to @yuvipanda on the assistance on this. Would've taken me hours to understand it.

Just noting we received a report of this bug in the Portuguese wikipedia telegram group today.

I had to rollback as the new Host validation in openrefine 3.5 broke it with the current settings. I'm not quite sure what best to do here, since we're already proxying it through the jupyter server we could disable the validation. But I think I'll set it up to have the proper value in the next few days.

PR in https://github.com/toolforge/paws/pull/110

Merged and deployed

[2021-11-17 13:47:56,895: ERROR/ForkPoolWorker-14] Task video2commons.backend.worker.main[ea3650a1-4dcd-4341-95a3-7afa84e43b94] raised unexpected: FileNotFoundError(2, 'No such file or directory')
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/celery/app/trace.py", line 450, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/celery/app/trace.py", line 731, in __protected_call__
    return self.run(*args, **kwargs)
  File "/srv/v2c/video2commons/backend/worker.py", line 132, in main
    pywikibot.Site('commons', 'commons', user=username).login()
  File "/usr/local/lib/python3.7/dist-packages/pywikibot/site/_apisite.py", line 358, in login
    if self.userinfo['name'] == self.user():
  File "/usr/local/lib/python3.7/dist-packages/pywikibot/site/_apisite.py", line 465, in userinfo
    uidata = uirequest.submit()
  File "/usr/local/lib/python3.7/dist-packages/pywikibot/data/api.py", line 1739, in submit
    self._add_defaults()
  File "/usr/local/lib/python3.7/dist-packages/pywikibot/data/api.py", line 1245, in _add_defaults
    and self.site.mw_version >= '1.25wmf5'):
  File "/usr/local/lib/python3.7/dist-packages/pywikibot/site/_apisite.py", line 953, in mw_version
    mw_ver = MediaWikiVersion(self.version())
  File "/usr/local/lib/python3.7/dist-packages/pywikibot/site/_apisite.py", line 929, in version
    version = self.siteinfo.get('generator', expiry=1).split(' ')[1]
  File "/usr/local/lib/python3.7/dist-packages/pywikibot/site/_siteinfo.py", line 296, in get
    preloaded = self._get_general(key, expiry)
  File "/usr/local/lib/python3.7/dist-packages/pywikibot/site/_siteinfo.py", line 237, in _get_general
    default_info = self._get_siteinfo(props, expiry)
  File "/usr/local/lib/python3.7/dist-packages/pywikibot/site/_siteinfo.py", line 162, in _get_siteinfo
    data = request.submit()
  File "/usr/local/lib/python3.7/dist-packages/pywikibot/data/api.py", line 2022, in submit
    self._write_cache(self._data)
  File "/usr/local/lib/python3.7/dist-packages/pywikibot/data/api.py", line 2014, in _write_cache
    with open(self._cachefile_path(), 'wb') as f:
FileNotFoundError: [Errno 2] No such file or directory: '/srv/v2c/apicache-py3/644d1c68a83935b749c8608f05fd23e5e5c1e18f5dce61a6c873727d7791a896'

Not sure how to debug this, is this perhaps a known issue with pywikibot?

Fixed on Fixed on https://github.com/toolforge/video2commons/commit/53d51e96d5db24afe3725eb88bdf90f5884fdd0a.

This error should be fixed with the upgrade to python3.

yeah, let me see if I can get python3 working on the backend