Page MenuHomePhabricator

upgrade jupyter-rsession-proxy
Closed, ResolvedPublic

Description

T298568 uncovered that an upgrade to jupyter-rsession-proxy from 1.4 to 2 broke rstudio in paws. A pr is open to deploy an older jupyter-rsession-proxy, this ticket is to track getting the new jupyter-rsession-proxy working.

https://github.com/jupyterhub/jupyter-rsession-proxy/blob/master/CHANGELOG.md

Event Timeline

rook renamed this task from upgrade rstudio to upgrade jupyter-rsession-proxy.Feb 19 2022, 7:17 PM
rook created this task.

Following the install instructions from:
https://github.com/jupyterhub/jupyter-rsession-proxy
installing rstudio-server-2022.02.0-443-amd64.deb (as described in https://www.rstudio.com/products/rstudio/download-server/debian-ubuntu/)
does not seem to get rstudio working with a newer jupyter-rsession-proxy.

possibly related:
https://github.com/jupyterhub/jupyter-rsession-proxy/issues/93

This seems to work on an ubuntu vm. Not so much with our paws container. Maybe it is an auth issue. Or that rstudio-server isn't starting in the container.

Log (/home/paws/.local/share/rstudio/log/rserver.log) gives errors like:

2022-06-08T20:38:59.080734Z [rserver] ERROR Could not find details for server user 'VRook (WMF)'; LOGGED FROM: virtual rstudio::core::ProgramStatus rstudio::server::Options::read(int, char* const*, std::ostream&) src/cpp/server/ServerOptions.cpp:299
2022-06-08T20:38:59.081383Z [rserver] ERROR system error 2 (No such file or directory) [description: User not found., user-value: VRook (WMF)]; OCCURRED AT rstudio::core::Error rstudio::core::system::User::Impl::populateUser(rstudio::core::system::User::Impl::GetPasswdFunc<T>&, T) [with T = const char*  rstudio::core::system::User::Impl::GetPasswdFunc<T> = std::function<int(const char*, passwd*, char*, long unsigned int, passwd**)>] src/cpp/shared_core/system/User.cpp:68; LOGGED FROM: virtual rstudio::core::ProgramStatus rstudio::server::Options::read(int, char* const*, std::ostream&) src/cpp/server/ServerOptions.cpp:300

Does appear to be an auth issue. Namely that there is both the users username, such as VRook (WMF) and tools.paws being offered to rstudio. It appears to be looking for the latter in /etc/passwd where it does not exist. Two potential ways of getting this working are to add a duplicate entry in /etc/passwd with the username. Another method of getting this working is to update:
https://github.com/toolforge/paws/blob/643e112be91c355b8769caeab9e814dea9172b7d/paws/values.yaml#L158
to be:
spawner.environment['USER'] = 'tools.paws'

Both feel like they would break unrelated things, and I don't have an immediate way to implement the former of them. It also remains unclear to me how this worked on older rstudio-server versions.

Adding either of:

'--auth-validate-users=0',
'--auth-minimum-user-id=0',

to the rserver options in jupyter-rsession-proxy doesn't seem to have an effect.

@Chicocvenancio @yuvipanda @Majavah thoughts on how to approach this? In particular what we might break by updating the user to tools.paws, or alternative methods that we might get rstudio running in an updated context?

I created https://github.com/jupyterhub/jupyter-rsession-proxy/issues/129 upstream for this. I think should be a pretty straightforward change there to get things working. As I see it, NB_USER should be the source of truth for users in jupyterhub land.

That seems in the right direction, though I don't think it is quite all of the issue. I was working on that last night and when I set
'--server-user=tools.paws',
statically in rsession-proxy I get a little further, but end at a new error:

Could not connect to the R session on RStudio Server.
Error occured during transmission (6)

The --server-user bit does look updated:
/usr/lib/rstudio-server/bin/rserver --auth-none=1 --www-frame-origin=same --www-port=46549 --www-verify-user-agent=0 --secure-cookie-key-file=/tmp/tmph5vx3uoe --server-user=tools.paws --www-root-path=/user/VRook_%28WMF%29/rstudio/ --database-config-file=/tmp/tmpq3dbmz83/tmpq43wi0a2
Though I don't see anything for rsession, it might run/exit faster than I can catch it though.

This works fine if I add:
VRook (WMF):x:52771:52771:Default user,,,:/home/paws:/bin/bash
to /etc/passwd
So it would seem that rserver is still getting "VRook (WMF)" from somewhere beyond the command line. Though that gives me a potential lead, maybe the proxy is passing it to rsession or somewhere else as well. More to look at :)

Couldn't decide between consistency and minimal changes. Seemed to work without it. Though I'll update.

https://github.com/toolforge/paws/pull/173 lgtm, Will make the PR upstream.
https://github.com/jupyterhub/jupyter-rsession-proxy/pull/130 is the upstream PR to track, we can use a fork if it takes a while to get merged.