Mentioned in SAL (#wikimedia-cloud) [2023-06-13T18:16:00Z] <Rook> Rolling jupyterlab back to last known working version T338981

@Ponor to verify is paws working for you in its current state?

@rook I used it for an hour or two yesterday, and this morning it's still running fine. I think this was the most reasonable decision. I don't think I miss anything from the newer versions, and tbh I don't remember seeing anything *that* new in them. Would it be possible to install and run 4.x from a different URL in the future? I'd be happy to help testing, just LMK.
Many thanks!

@Ponor thank you for offering to see if you can reproduce. It is (was) possible to run paws in the other datacenter as a separate install on a different URL. Though that's kind of down, it's suggested T338778 will restore access when complete. Though we're stalled until we can get in there. I agree, it isn't amazingly critical that we get to 4.0 right away, just that we do unblock ourselves on that front eventually. Anyways when we can get into the dev cluster I'll push a newer jupyterlab version there and let you know.

Getting closer. The old cluster is down, and deploying a new one fails in T339961

vivian-rook opened https://github.com/toolforge/paws/pull/296

Hi @Ponor the codfw1dev dc is still not deploying k8s, so we're going to try things in the eqiad1 dc. I've deployed
https://github.com/toolforge/paws/pull/296
to test-hub-paws.wmcloud.org which has jupyterlab 3.6.3 I believe this is the version we deployed where you first noticed the interface failing. For now could you log in there and identify if you can recreate the issue?

A few things to keep in mind, this looks mostly identical to the cluster running on hub-paws.wmcloud.org. You can tell the difference with pip list | grep jupyterlab in a terminal. So be careful not to confuse them.

Also since we're in the same dc, we're using the same nfs. You will see, and be able to edit/delete any of your files as normal. They are very much so the same files. Don't delete anything that you don't want deleted.

Thank you!

@rook: after a quick test I can confirm these old traits:

• the browser complaining about a script slowing it down on the load of the "problematic" notebook (works fine in v. 3.4.8),
• the view resetting itself to top of the notebook upon running some 30 cells in sequence (browser console showing: TypeError: e is undefined; TypeError: undefined has no properties, and similar - for index.es6.js:257:16),
• jupyterlab failing to load that same notebook (appears blank)

In Firefox, about:performance typically shows 3–5× higher "energy impact" of jupyterlab 3.6.3 at test-hub-paws when notebook is loaded but no code is running.

@Ponor thank you for verifying that. Can you go back to test-hub-paws.wmcloud.org and restart your server? Now it should have most of the things we add for PAWS removed (No OpenRefine, no R, no Julia, etc), can you verify that this implementation is working fine for you?

@rook : no change I'm afraid. Some files were not loading, and the one that did load was kind of slow (slowing down my whole browser).

I had to !pip install a few modules (pandas, pywikibot, mwparserfromhell) in order to run the notebook.
TypeError: e is undefined was now coming from jlab_core.4d5114e25256eeddf4e4.js:2:1606197.

In T338981#8991720, @Ponor wrote:

@rook : no change I'm afraid. Some files were not loading, and the one that did load was kind of slow (slowing down my whole browser).

Oh this is interesting! So after installing the dependencies, you are seeing the same underlying problem of the interface being unusable?

There were signs of it being weird even before I !pip installed anything (that slowness). It's the same JS problem as before.

I'm also seeing this:
15:03:05.585 Source map error: Error: request failed with status 404
Resource URL: https://test-hub-paws.wmcloud.org/user/PonoRoboT/static/lab/jlab_core.4d5114e25256eeddf4e4.js?v=4d5114e25256eeddf4e4
Source Map URL: jlab_core.4d5114e25256eeddf4e4.js.map?v=4d5114e25256eeddf4e4

@Ponor Thank you for your continued help on this matter. I've put test-hub-paws.wmcloud.org back to only having an updated jupyterlab (3.6.3), could you log into it with a user that hasn't logged into PAWS before? Or create a test user for that. Then see if you see the same problem when you run a notebook with that user?

@rook no problem at all!
I logged in with my secondary account. No change. Slow server start, slow file load with browser complaints about a script. Resets with TypeError: e is undefined.
Noticed at least 20 error messages in the browser console as the server was respawning of the following kind:

15:17:20.443 Unsatisfied version 3.6.3 of shared singleton module @jupyterlab/filebrowser (required ^3.6.5) remoteEntry.e598f2de1483a54c09e7.js:1:4719
15:17:20.446 Unsatisfied version 3.6.3 of shared singleton module @jupyterlab/ui-components (required ^3.6.5) remoteEntry.e598f2de1483a54c09e7.js:1:4719
...
15:17:20.598 Unsatisfied version 3.6.3 from @jupyterlab/application-top of shared singleton module @jupyterlab/notebook (required ^3.6.4) remoteEntry.936b971eda36f85d6cca.js:1:5549
15:17:20.563 Unsatisfied version 3.6.3 from @jupyterlab/application-top of shared singleton module @jupyterlab/settingregistry (required ^4.0.2) 
15:17:20.588 Unsatisfied version 3.6.3 from @jupyterlab/application-top of shared singleton module @jupyterlab/settingregistry (required ^3.6.4)

It looks like your other user (Ponorić) logged into paws back in 2021. I'm trying to identify if there was something added to your user home directories that is causing the malfunction. I was hoping to do this through the dev dc, but that isn't working so we're sharing nfs, and thus files. Would it be alright if cleared out all the files in the home dir for Ponorić then trying again?

To verify, sorry I know I've asked this elsewhere. You're running Firefox on Linux?

@rook Hm... sorry, I do not remember logging in before. I did check if there was https://public-paws.wmcloud.org/User:Ponorić/ - there wasn't.
Sure, let's clear it out and I'll try again.

I am running Firefox on Linux. But I had the same issue in Chrome and Chromium on Linux AND in Chrome on Windows.

@Ponor ok I've cleared out the home directory. I don't have too much hope for it, but go ahead and give it a try...Though when you first log in, before you copy any files in place. Does everything seem to work fine at that point?

Thank you for verifying on browser and os. All the more strange, I'm running Firefox on Linux and can't reproduce...Well if you see it again on the new user I'll try to recreate by duplicating the directory that you make with files.

Did everything from Chrome/Linux now. Before I copied any files: I only thought the server took too long to respawn... some 2 minutes. Firefox complains about a script being too slow before I click on any files, Chrome does not.

There are many files in the TEST dir. I made a notebook copy that should be safe to run: forVRook.ipynb
Click to open: asks which kernel I want to use, but gets unresponsive for almost a minute. I usually keep the file browser or the TOC open on the left.
Run cell after cell (shift-enter), don't wait for them to execute.
Usually already half way through it will get stuck and the view will go back to the top. At that point I am no longer able to save (disabled). So I saved as forVRookCOPY.ipynb

Try opening them both at the same time; one may work fine, but the other won't. Sometimes it takes more than one full pass, top to bottom, but eventually it does get stuck.

(My) network problem, perhaps? Maybe you're "too close" (in any sense) to the servers, and I am not (though... hm... optimum.com, East Coast)

No issues with 3.4.8 since the day you rolled back.

@rook At some point someone mentioned this ever growing file .jupyter_ystore.db and NFS. I was just reading https://jupyterhub.readthedocs.io/en/stable/explanation/database.html, where they specifically say "The SQLite database should not be used on NFS. SQLite uses reader/writer locks to control access to the database. This locking mechanism might not work correctly if the database file is kept on an NFS filesystem. This is because fcntl() file locking is broken on many NFS implementations. Therefore, you should avoid putting SQLite database files on NFS since it will not handle well multiple processes which might try to access the file at the same time."

Is that what's going on? More often than not, I will have two or more notebooks open, and these newer JupyterLab versions will try save them way too often (writing things to the database at the same time?).

(I might be confusing hub and lab, but if hub has issues with NFS with its infrequent requests, maybe lab has them too)

@Ponor I was able to get what you mentioned reproduced! At least mostly, looks like running a lot of cells one at a time (shift+enter) locks up the system some of the time. Seems to more if I reload? Increasing the cpu may help, but it isn't obviously better.

In T338981#8997435, @Ponor wrote:

@rook At some point someone mentioned this ever growing file .jupyter_ystore.db and NFS. I was just reading https://jupyterhub.readthedocs.io/en/stable/explanation/database.html, where they specifically say "The SQLite database should not be used on NFS. SQLite uses reader/writer locks to control access to the database. This locking mechanism might not work correctly if the database file is kept on an NFS filesystem. This is because fcntl() file locking is broken on many NFS implementations. Therefore, you should avoid putting SQLite database files on NFS since it will not handle well multiple processes which might try to access the file at the same time."

Is that what's going on? More often than not, I will have two or more notebooks open, and these newer JupyterLab versions will try save them way too often (writing things to the database at the same time?).

(I might be confusing hub and lab, but if hub has issues with NFS with its infrequent requests, maybe lab has them too)

I like your idea way better. Let me see what more I can learn about it. Thank you for the direction

The slowness being an errant sqlite db is looking like a reasonable line of investigation.

@PAWS:~$ pip list | grep jupyterlab\ 
jupyterlab                3.4.8
@PAWS:~$ ls -l /proc/1/fd
total 0
lrwx------. 1 tools.paws tools.paws 64 Jul  7 15:37 0 -> /dev/null
l-wx------. 1 tools.paws tools.paws 64 Jul  7 15:37 1 -> 'pipe:[1293463625]'
lrwx------. 1 tools.paws tools.paws 64 Jul  7 15:56 11 -> 'socket:[1293458245]'
lrwx------. 1 tools.paws tools.paws 64 Jul  7 15:56 12 -> 'anon_inode:[eventpoll]'
lrwx------. 1 tools.paws tools.paws 64 Jul  7 15:56 13 -> 'socket:[1293464871]'
lrwx------. 1 tools.paws tools.paws 64 Jul  7 15:56 14 -> 'socket:[1293464872]'
lrwx------. 1 tools.paws tools.paws 64 Jul  7 15:56 17 -> /dev/pts/ptmx
l-wx------. 1 tools.paws tools.paws 64 Jul  7 15:37 2 -> 'pipe:[1293463626]'
lrwx------. 1 tools.paws tools.paws 64 Jul  7 15:37 3 -> 'anon_inode:[eventpoll]'
lrwx------. 1 tools.paws tools.paws 64 Jul  7 15:37 4 -> 'socket:[1293457903]'
lrwx------. 1 tools.paws tools.paws 64 Jul  7 15:37 5 -> 'socket:[1293457904]'
lrwx------. 1 tools.paws tools.paws 64 Jul  7 15:37 6 -> 'anon_inode:[eventpoll]'
lrwx------. 1 tools.paws tools.paws 64 Jul  7 15:37 7 -> 'socket:[1293457905]'
lrwx------. 1 tools.paws tools.paws 64 Jul  7 15:37 8 -> 'socket:[1293457906]'
lrwx------. 1 tools.paws tools.paws 64 Jul  7 15:37 9 -> 'socket:[1293457924]'

@PAWS:~$ pip list | grep jupyterlab\ 
jupyterlab                3.6.3
@PAWS:~$ ls /proc/1/fd -l
ls: cannot access '/proc/1/fd/18': No such file or directory
total 0
lrwx------. 1 tools.paws tools.paws 64 Jul  7 15:48 0 -> /dev/null
l-wx------. 1 tools.paws tools.paws 64 Jul  7 15:48 1 -> 'pipe:[9616123]'
lrwx------. 1 tools.paws tools.paws 64 Jul  7 15:56 10 -> 'socket:[9621157]'
lrwx------. 1 tools.paws tools.paws 64 Jul  7 16:04 12 -> /home/paws/.jupyter_ystore.db
lrwx------. 1 tools.paws tools.paws 64 Jul  7 15:56 13 -> 'socket:[9619043]'
lrwx------. 1 tools.paws tools.paws 64 Jul  7 16:04 14 -> 'socket:[9642208]'
lrwx------. 1 tools.paws tools.paws 64 Jul  7 15:56 15 -> /dev/pts/ptmx
lrwx------. 1 tools.paws tools.paws 64 Jul  7 15:56 16 -> 'socket:[9622552]'
lrwx------. 1 tools.paws tools.paws 64 Jul  7 15:56 17 -> 'socket:[9621176]'
l?????????? ? ?          ?           ?            ? 18
lrwx------. 1 tools.paws tools.paws 64 Jul  7 16:04 19 -> 'socket:[9644124]'
l-wx------. 1 tools.paws tools.paws 64 Jul  7 15:48 2 -> 'pipe:[9616124]'
lrwx------. 1 tools.paws tools.paws 64 Jul  7 15:48 3 -> 'anon_inode:[eventpoll]'
lrwx------. 1 tools.paws tools.paws 64 Jul  7 15:48 4 -> 'socket:[9614240]'
lrwx------. 1 tools.paws tools.paws 64 Jul  7 15:48 5 -> 'socket:[9614241]'
lrwx------. 1 tools.paws tools.paws 64 Jul  7 15:48 6 -> /home/paws/.local/share/jupyter/file_id_manager.db
lrwx------. 1 tools.paws tools.paws 64 Jul  7 15:48 7 -> 'socket:[9619027]'
lrwx------. 1 tools.paws tools.paws 64 Jul  7 15:48 8 -> /home/paws/.jupyter_ystore.db
lrwx------. 1 tools.paws tools.paws 64 Jul  7 15:48 9 -> /home/paws/.local/share/jupyter/nbsignatures.db

@Ponor I've pulled the various changes that R and Sparql make and bumped the cores from 1 to 2. This isn't really a state that I could redeploy, though could you restart your server and check to see if it is working (if slowly) for you? It is still slow, in particular for a minute or two after startup. But so far I haven't been able to get it to really freeze like I have before.

Slowness seems to be an observed issue https://discourse.jupyter.org/t/cripplingly-slow-ui-am-i-the-only-one/5351/5 seems to describe the same issue with larger notebooks. Though the whole thread discusses the same kind of issue. Firefox and Chrome make frequent appearances, seemingly opera and safari were working better (I didn't realize opera was still around). But switching browsers doesn't feel like a real solution.

Nah :(
I checked twice in Chrome and once in Firefox, shift-entering only when the previous run was complete. Surprisingly, it didn't take more than 10 cells for the reset-to-top to happen. I wouldn't say that the overall slowness is my problem (though I notice it from time to time, especially - as you say - in the first few minutes). Right now it's the cell freeze, followed by the focus reset to the topmost cell. Also, just a few hours ago my forVRookCOPY.ipynb was working fine, now I cannot even open it. No changes were made except for the cell runs, and the notebook was automatically saved.

Can't even say this is a browser problem, because my local installation of jupyter-lab 4.0.2 works just fine in that same browser (though: no hub, no containers, and nothing happening over the net)

@Ponor Thank you for checking on the current test. In your chrome and firefox, are you running any extensions? Noscript, ad blocker, etc?

@rook I do have a few extensions in FF, and none in Chrome since I rarely use it. All recent test were done in private/incognito windows, which is supposed to reduce the number of running extensions. I will disable all FF extensions and see if that helps.

I don't know if anything changed since Friday, but in the last ~90 minutes I was unable to trigger the cell freeze + view reset behavior, though jupyterlab tab switching is waaay slower than it used to be (10-20 seconds), and both notebooks with your name fail to load until I tell them to "Reload from Disk".

I am also unable to save any changes to the notebooks (reminds me of T310622). When I hover their name in jupyterlab file browser I get "Created: 2023-07-10" but "Modified 2023-07-07" (or -06).

Not sure if I mentioned it before, but every now and then I'm seeing the following browser console message "Failed to fetch ipywidgets through the "jupyter.widget.control" comm channel, fallback to fetching individual model state. Reason: Control comm was closed too early". I don't know what that means.

In T338981#9001813, @Ponor wrote:

@rook I do have a few extensions in FF, and none in Chrome since I rarely use it. All recent test were done in private/incognito windows, which is supposed to reduce the number of running extensions. I will disable all FF extensions and see if that helps.

I take it it wasn't an extension?

I don't know if anything changed since Friday, but in the last ~90 minutes I was unable to trigger the cell freeze + view reset behavior, though jupyterlab tab switching is waaay slower than it used to be (10-20 seconds), and both notebooks with your name fail to load until I tell them to "Reload from Disk".

Does this mean the current state doesn't do the freeze/reset thing (It's slow, but not catastrophically failing?)

I am also unable to save any changes to the notebooks (reminds me of T310622). When I hover their name in jupyterlab file browser I get "Created: 2023-07-10" but "Modified 2023-07-07" (or -06).

So nothing at all is saving in the test environment? Could the same file be open in the production environment as well, maybe causing a file lock problem?

Not sure if I mentioned it before, but every now and then I'm seeing the following browser console message "Failed to fetch ipywidgets through the "jupyter.widget.control" comm channel, fallback to fetching individual model state. Reason: Control comm was closed too early". I don't know what that means.

Hmm...I don't know what that means either. https://discourse.jupyter.org/t/jupyterhub-widgets-fail-to-load/18218 suggests there may be a missing module in one of the installed libs. I'm not immediately convinced that is the issue, as you mentioned seeing the problem with a minimal install as well. Though it could be part of the minimal stuff.

@rook No, it wasn't an extension: I've had all extensions disabled for a day, and right now the "thing" at test-hub-paws.wmcloud.org is pretty much unusable (even worse: unpredictably unusable): extremely slow, the view is resetting itself again today, and second after second the console is throwing (now that's new!) messages like

17:25:00.741 GET
wss://test-hub-paws.wmcloud.org/user/Ponorić/api/yjs/json:notebook:247cafee-e7a0-4eef-beda-3a71c3e691a4?sessionId=ce366ebf-0bfd-4cb7-bc12-859cf7438003
[HTTP/1.1 101 Switching Protocols 221ms]

	
GET
	wss://test-hub-paws.wmcloud.org/user/Ponori%C4%87/api/yjs/json:notebook:247cafee-e7a0-4eef-beda-3a71c3e691a4?sessionId=ce366ebf-0bfd-4cb7-bc12-859cf7438003
Status
101
Switching Protocols
VersionHTTP/1.1
Transferred354 B (0 B size)

    	
    Connection
    	upgrade
    content-security-policy
    	frame-ancestors 'self'; report-uri /user/Ponori%C4%87/api/security/csp-report
    Date
    	Tue, 11 Jul 2023 21:25:00 GMT
    sec-websocket-accept
    	zquZZwrlbnxmDfsvQaS0ce7CoEo=
    Server
    	nginx/1.18.0
    upgrade
    	websocket
    x-content-type-options
    	nosniff
    x-jupyterhub-version
    	4.0.0
    	
    Accept
    	*/*
    Accept-Encoding
    	gzip, deflate, br
    Accept-Language
    	en-US,en;q=0.5
    Cache-Control
    	no-cache
    Connection
    	keep-alive, Upgrade
    Cookie
    	jupyterhub-user-Ponori%C4%87=2|1:0|10:1689109966|28:jupyterhub-user-Ponori%C4%87|40:N0RzaW1OWUV1WlZzT0VBVGs5RmVldm1FQWhUQ0Vy|e60cd72148372c7530c1fe4fb41af03882ef3cc714dbd1a3454fcef8ab59bb64; _xsrf=2|44d32d7e|b6df8d62f74253a3865beaad2268897c|1689109966; jupyterhub-session-id=6f7ebdbe23f9473e9bc2ac180cbc2a0c
    Host
    	test-hub-paws.wmcloud.org
    Origin
    	https://test-hub-paws.wmcloud.org
    Pragma
    	no-cache
    Sec-Fetch-Dest
    	websocket
    Sec-Fetch-Mode
    	websocket
    Sec-Fetch-Site
    	same-origin
    Sec-WebSocket-Extensions
    	permessage-deflate
    Sec-WebSocket-Key
    	mGCXG5KqHEFIXxWD/wBrhA==
    Sec-WebSocket-Version
    	13
    Upgrade
    	websocket
    User-Agent
    	Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/115.0

I don't have the notebooks open anywhere else with my secondary account (though the system might THINK that I have, but that's out of my control). The files that I saved won't open any more, but the same files that I "saved as" will (and I've seen that before).

Really don't know what to think of all this. Will keep testing, hoping we'll figure it out.

In T338981#9007127, @Ponor wrote:

Really don't know what to think of all this. Will keep testing, hoping we'll figure it out.

I think we should start trying more fundamental variables. I don't know if I've reproduced the freezing/resetting, though I am seeing the slowness, and it is quite slow. I suspect what you're seeing is a more pronounced experience with the slowness problem, so, hopefully, resolving the slowness will resolve the more catastrophic issues. I'm going to pull down the cluster at test-hub-paws.wmcloud.org and have deployed another cluster at hub-paws-dev.codfw1dev.wmcloud.org, it's running the same code as production, just with the jupyterlab 3.6.3 as described, at time of writing, in https://github.com/toolforge/paws/pull/296 It is however completely separate from production at this point, on its own cluster, database, nfs, in a separate dc. You shouldn't find any of your files there when you log in (If you do, please let me know).
If you could give it a try and verify that the problem is still there that would be great. This all being said I suspect you will experience just the same problem there.

So would you be up for installing minikube on your system? The paws install process is described in https://github.com/toolforge/paws Though is basically:

git clone https://github.com/toolforge/paws.git
cd paws
minikube start --kubernetes-version=v1.23.15
minikube addons enable ingress
helm repo add jupyterhub https://jupyterhub.github.io/helm-chart/
helm dep up paws/
kubectl create namespace paws-dev
helm -n paws-dev install dev paws/ --timeout=50m

When that finishes it should print out a little more instruction about modifying your hosts file

Get the IP of minikube with:
minikube ip
Add these lines to your hosts file:
<ip address> hub.paws.local
<ip address> public.hub.paws.local

At which point you should be able to access the local version at hub.paws.local

If that all seems to be working without issue (it should be running the version in prod at this point)

git checkout T338981
helm -n paws-dev upgrade dev paws/ --timeout=50m

At that point restart your server at hub.paws.local and see if you can recreate the slowness/freezing/resetting problems

Also verify that the right image got pulled with:

kubectl get -o yaml pod | grep quay.io/wikimedia-paws-prod/singleuser:

Everything should return as: pr-296
for example image: quay.io/wikimedia-paws-prod/singleuser:pr-296

If this is working it suggests that the problem is either NFS (My primary suspect) or network lag or pod security policy or the database. I don't believe there are any other bits that are different at this point between the minikube and production versions. But NFS is not used for the home directory, so if it is working locally we can have some ideas. I at least can't reproduce the slowness in the local install.

vivian-rook closed https://github.com/toolforge/paws/pull/296

@rook: apologies, I've been mostly away from my laptop for over a week. Just got an email update from this T. I can do some testing, but it won't be before Friday night.
I haven't read the instructions yet. Before I do, please let me know if today's closure will affect my ability to test. Sorry again, and thank you.

In T338981#9025314, @Ponor wrote:

@rook: apologies, I've been mostly away from my laptop for over a week. Just got an email update from this T. I can do some testing, but it won't be before Friday night.
I haven't read the instructions yet. Before I do, please let me know if today's closure will affect my ability to test. Sorry again, and thank you.

Oh no problem. Sorry there would be a minor change in things for testing. hub-paws-dev.codfw1dev.wmcloud.org is still the same. Though for minikube you would want to test off of https://github.com/toolforge/paws/pull/308 I had to close the old branch and do a two commit because T342144 was preventing the singleuser container to build, and I haven't merged it yet in case anyone is using the old libs. Though the long and short of it is you can still build off of the T338981 branch for minikube.

I might make some more changes to other bits of PAWS and circle back if it updates anything for testing on your end.

@rook I followed your recipe and was able to test everything off of T338981. (Un)surprisingly, all the symptoms are still there: the notebooks that I used at PAWS are slow in FF, Chrome and Chromium, the javascript errors are still there, and the view is resetting itself. Those same notebooks work fine in my regular local installation of jupyter-lab.
If you think of some other tests I can try, I'll be happy to help (though I can't say I know what exactly I'm doing, and will probably not have time to learn more about it). I wonder if the guys who reported the saving issue in T310622 would see this slowness with their "problematic" notebooks as well.

@Ponor to verify, you tried installing on minikube local to your laptop, and saw the same issue? The same issue has been seen on https://hub-paws-dev.codfw1dev.wmcloud.org/ and minikube?

@rook Yes, my local minikube behaves pretty much the same as those at test-hub-paws and hub-paws-dev (with the notebooks that I know would trigger the problem).

@Ponor would you mind seeing if you can repeat what you had been seeing on https://hub-paws-dev.codfw1dev.wmcloud.org/ again? It's now running the beta release of the new helm chart, which updates a number of things. If we're lucky that is working, if not, it's probably time to open an issue on https://github.com/jupyterhub/zero-to-jupyterhub-k8s/issues which I would ask for you to do as I cannot reproduce what you are seeing to answer questions about it.

Good news, @rook. Nothing in the last 60 minutes of heavy testing seemed suspicious: my notebooks are loading, running reasonably fast, no resets, no JS error messages in the console. It lets me save the notebooks (save button is not disabled this time). FF is showing low energy impact of the paws window in about:performance. Only switching between jupyter notebook tabs takes a little too long, 1 to 2 seconds, but that's pretty much all that's different from my local jupyterLAB.
Previously, it wouldn't take me more than 1-2 minutes to see all the glitches. I hope this solves the problem; will let you know if anything pops up.
Many, many thanks!

This is great news @Ponor! I'll see if they release the production version soon, otherwise I'll probably go ahead and deploy the beta to PAWS prod as is. Thank you for the update, and please do let me know if anything comes up.