I'd like to create Python 3.6 kernels for my projects—and ideally even upgrade my whole base virtualenv to 3.6.
The main thing I want is to use f strings, which will drastically simplify the way I template SQL queries.
With https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/480041, it seems like we should at least be able to get Python 3.6 on the SWAP host, although it doesn't seem to be there right now.
This isn't just a simple upgrade, so we're taking it as normal priority and it won't be done in the immediate future
It requires to recompile all deps to 3.6 (including jupyter itself) so it is an upgrade of the whole platform that might not be trvial
The extra tricky think about this SWAP setup is ALL user jupyterhub virtualenvs will have to be reinstalled. This means that every user of SWAP will have to start with a blank virtualenv in their jupyterhub env.
From my perspective, I don't particularly care if the base virtualenv is running on Python 3.6—I just want to be able to choose to start a Python 3.6 kernel for specific projects (roughly like I can chose to start a Python 3, PySpark, or Spark Scala kernel now).
But it sounds like there's no way to do that without running the base virtualenv on Python 3.6 too?
Hmm, I'm not sure if it would work, but it might. We should be able to at least install the python 3.6 binary (we have deb packages for it now :) ). I don't think anyone would mind if we just did that, and you could try it on your own (since you are an ADVANCED SWAPPER :D )
As FYI we have now Python3.7 + libpython3.7 on notebooks:
elukey@notebook1003:~$ python3.7 Python 3.7.1 (default, Dec 16 2018, 12:33:36) [GCC 6.3.0 20170516] on linux Type "help", "copyright", "credits" or "license" for more information. >>>
Caveat: since those nodes are still Stretch (that ships with 3.5), we had to use backports only for the interpreter and the standard library. This means that there might be packages in the Debian stretch repos requiring python3.5 and not working with 3.7, therefore python3.7 is usable only to bootstrap it in a virtual environment. Let me know if this works :)
Very neat! To clarify, this means we can now use Python 3.7 via the command line, but not in the SWAP installation of Jupyter, right? In that case, I'd like for us to keep this task open, since the ultimate ask is to be able to use Python 3.6 or greater without having to leave SWAP.
You can use python3.7 in your venv when you create the notebook, so in theory after that you should be able to use it (but I could be wrong, very ignorant about SWAP currently). The main trouble right now is using python3.7 via "CLI" since without the right venv it cannot install libraries (since the system ones are not compatible, they only work with 3.5).
You can use python3.7 in your venv when you create the notebook
All of the default python notebook kernels we make available in JupyterHub use the same auto created venv. It *might* be possible to make a python 3.7 based venv and then create a custom kernel for it, but I haven't tried. (These instructions are for spark but the basic idea is the same and can be used to create a new python kernel.)
It looks like Pandas 1.0, which should be released in the next month or so, will require Python 3.6.1 or higher.
@Ottomata is anything different now that we have jupyterhub on swap nodes with Buster? (like stat1005/stat1008, not yet ready but close to be)
Likely the SWAP wheels that you've built for buster use Python 3.7, right? So, when folks log into it on a Buster node, they will get a copy of that Python 3.7 venv.
I look forward for the day when we don't have to deal with stale copies of venvs anymore.