T298568 uncovered that an upgrade to jupyter-rsession-proxy from 1.4 to 2 broke rstudio in paws. A pr is open to deploy an older jupyter-rsession-proxy, this ticket is to track getting the new jupyter-rsession-proxy working.
https://github.com/jupyterhub/jupyter-rsession-proxy/blob/master/CHANGELOG.md