In T341378#8998519, @SEgt-WMF wrote:

Thank you @Chicocvenancio for reading my question and adding your ideas! Yeah, I just run the code with another Wiki account I have and it went well, so I'm guessing every user gets a limited number of SQL runs?

Running your code I realized that for loop only runs once, so even if it is leaking connections it probably is not your main issue. But since the max_connections are per user they might be coming from another notebook/process.

This is not a PAWS issue as much as a connection limit on the replicas issue. I'm not quite sure what pymysql and Pandas are doing there but it seems connections are leaking beyond each iteration of

def query_wikis(query, df):
    wikis = df.database_code.unique().tolist()
    combined_result = pd.DataFrame()
    for wiki in wikis:
        result = pd_query(query, wiki)
        result['wiki_db'] = wiki
        combined_result = pd.concat([combined_result, result])
    return combined_result

This is a change in behavior for jupyterlab 4 impacting labpawspublic. The property we are using is having RTC: prepended to it. I cannot figure out where to get the proper path, in fact I cannot find the documentation for jupyterlab's 4.0 api... (3.0 docs).
A quick solution is to remove the RTC: there.

LGTM, just apply the same changes to the other tools.

Hopefully resolved with current updates.

Should be fixed now.

Local uploads should be working again.

@Andrew https://wikitech.wikimedia.org/wiki/Help:Trove_database_user_guide#Before_you_start

@taavi I thought that as well, but wikitech docs link to this template, and there is no need for a project beyond the db.

Sorry I missed the followup questions, but I think you have solved the mystery. Since I have ipblockexempt it didn't stop me from editing, but its likely my vpn provider was blocked.

Linkback from https://lists.wikimedia.org/hyperkitty/list/wikimedia-l@lists.wikimedia.org/message/DVA7I5TUHSNJAAYKOBD6O77FJT35E2MK/ where I cite this as an example of how WMF has delayed UCoC until the enforcemente guidelines are approved.

In T326164#8498955, @rook wrote:

In T326164#8498932, @Chicocvenancio wrote:

In T326164#8497724, @rook wrote:

It is possible that this will be resolved with T317787

Not really. :/, this is the new Jupyterhub rbac.

Interesting, I was making this guess as I can still delete my server in minikube, and the major difference is that we are not introducing Pod Security Policies in minikube. Begging the question of if not psp, why is it working in minikube. Perhaps we are configuring something else differently in prod that we could adjust?

Some sections of this, mostly host path parts, will become irrelevant with T308873 other parts remain unclear to me on why we need them. Aside from by default needing a psp, are there known benefits of our current setup?

In T326164#8497724, @rook wrote:

It is possible that this will be resolved with T317787

In T326160#8496160, @Bugreporter wrote:

In T326160#8496065, @Dominicbm wrote:

I tried again, and get a different error message:

400 : Bad Request
Invalid client_id parameter value.

I also get the same issue.

(my best guess is the authenticator is storing the client-id in the auth state and that has now changed in the upgrade)

Could you try one more time? I've deleted your user from the db, it should be recreated on login for you.

Its probably related to the user being running at the time of upgrade, let me try to manually clear the tokens in the db.

In T326167#8495997, @rook wrote:

If I'm understanding that network policy shouldn't 172.16.1.129 be permitted under 172.16.0.0/12?

This seems to be the new networkpolicy blocking private ip addresses.

Didn't manage to delete the server, just logged out and back in.

More I'm curious, added me where? I'm not in the list of admins in values.yaml, though I have access to the admin tab (well did until today)

Still never known how I had admin before though. I always assumed it was pulling from cloud vps, but that seems unexpected?

https://jupyterhub.readthedocs.io/en/stable/rbac/roles.html# seems to be the new way of defining admins. I see the admin tab but nothing is there for me right now.

Uhh, it now works... Maybe there was a cached logged in SA for me?

That seems fine to me. This is RBAC internal to jupyterhub to do oauth dances between the various components. I'm reading code/docs trying to figure out what needs to change.

In T326160#8495643, @rook wrote:

Oh, an account that was not logged in at the time of the upgrade can auth. Maybe it is something about accounts that were active during the upgrade?

That may parallel why it seemed to work in minikube.

Seems the oauth tokens for the internal service accounts in jupyterhub don't have the right scopes, weird that this doesn't seem to be mentioned in the upgrade guide.

Why was a rollback necessary? Is trying to fix the new version an option?

Probably a migration was applied with the upgrade and alembic sees it in the table but not in the migration code of the rolled back version? Nuking the db is always an option, will only disrupt currently running pods and log everyone out. There might be a rollback migration code for the new migration, or it might even be compatible needing only the a DELETE in the revision, that would need to be figured it out if keeping the db is desired.

Things are working again for me. I see no 400 errors in the logs for the past 10 minutes.

I confirm its down for me as well.

Reading the github issue I guess moving to notebook-based culling is the way to improve things, I can try to take a look at that.

the hub pod has some in-memory state, and (worse) some shared state with the proxy pod. https://github.com/jupyterhub/zero-to-jupyterhub-k8s/issues/1364 is main upstream issue on this, I think.

https://github.com/toolforge/paws/pull/173 lgtm, Will make the PR upstream.
https://github.com/jupyterhub/jupyter-rsession-proxy/pull/130 is the upstream PR to track, we can use a fork if it takes a while to get merged.

Probably https://github.com/jupyterhub/jupyter-rsession-proxy/blob/master/jupyter_rsession_proxy/__init__.py#L139

I created https://github.com/jupyterhub/jupyter-rsession-proxy/issues/129 upstream for this. I think should be a pretty straightforward change there to get things working. As I see it, NB_USER should be the source of truth for users in jupyterhub land.

I think volume persistence is the biggest blocker here, right?

Feels like your firefox is keeping jupyterlab open after it has a disconnect (Very possibly only a disconnect from the nfs), without telling you that it isn't connected.

I think there is something a bit more esoteric going on. If the the pod wasn't running or didn't have access to the filesystem creating a new file would not be possible. Might be browser session, pod v jupyterhub authentication or websocket connection failing. I would look into the hub and proxy logs for any error around the routes to the user pod and in the user pod itself for any error message the next time this happens.

Have you tried using the absolute path?

Sounds about right to me.

yeah, we haven't used in a few years.

The lack of an uptick in new weekly editors on fawiki during the ip block indicates to me a big difference from ptwiki's implementation. My lack of farsi prevents a deeper investigation, but in ptwiki the interface was hacked to point logged out users at the point of edit. The edit button was changed to point directly to the log-in page and banner explaining the need to create an account linking to the reasoning was put there.

@Isaac indeed. Fixed the link to a permalink to avoid future issues.

6.6.5 was merged.

Other bank transfer methods are also erroring for me.

@EMartin its before we actually make the transaction.

All for the upgrade but just for the record I don't see a path for exploitation here. All requests to openrefine go through the proxy and only authenticated requests pass. And if you're authenticated into a user server, well RCE is the main feature for jupyterhub.

LGTM. Opinionated ways to run the more basic tasks is a good way to reduce the burden on new users.

I guess portaudio19-dev is the missing apt package here.

Thanks to @yuvipanda on the assistance on this. Would've taken me hours to understand it.