Page MenuHomePhabricator

Provide Python 3.6+ on SWAP
Closed, ResolvedPublic

Description

I'd like to create Python 3.6 kernels for my projects—and ideally even upgrade my whole base virtualenv to 3.6.

The main thing I want is to use f strings, which will drastically simplify the way I template SQL queries.

With https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/480041, it seems like we should at least be able to get Python 3.6 on the SWAP host, although it doesn't seem to be there right now.

Event Timeline

Restricted Application changed the subtype of this task from "Deadline" to "Task". · View Herald TranscriptDec 24 2018, 5:46 AM
Tbayer added a subscriber: Tbayer.
kzimmerman moved this task from Triage to Tracking on the Product-Analytics board.Jan 3 2019, 9:09 PM
Milimetric added a subscriber: Milimetric.

This isn't just a simple upgrade, so we're taking it as normal priority and it won't be done in the immediate future

Nuria added a subscriber: Nuria.Jan 7 2019, 4:47 PM

It requires to recompile all deps to 3.6 (including jupyter itself) so it is an upgrade of the whole platform that might not be trvial

The extra tricky think about this SWAP setup is ALL user jupyterhub virtualenvs will have to be reinstalled. This means that every user of SWAP will have to start with a blank virtualenv in their jupyterhub env.

nshahquinn-wmf added a comment.EditedJan 7 2019, 10:31 PM

The extra tricky think about this SWAP setup is ALL user jupyterhub virtualenvs will have to be reinstalled.

From my perspective, I don't particularly care if the base virtualenv is running on Python 3.6—I just want to be able to choose to start a Python 3.6 kernel for specific projects (roughly like I can chose to start a Python 3, PySpark, or Spark Scala kernel now).

But it sounds like there's no way to do that without running the base virtualenv on Python 3.6 too?

Hmm, I'm not sure if it would work, but it might. We should be able to at least install the python 3.6 binary (we have deb packages for it now :) ). I don't think anyone would mind if we just did that, and you could try it on your own (since you are an ADVANCED SWAPPER :D )

mforns moved this task from Project Newpyter to Incoming on the Analytics board.Aug 19 2019, 5:45 PM
elukey added a subscriber: elukey.Aug 22 2019, 4:39 PM

As FYI we have now Python3.7 + libpython3.7 on notebooks:

elukey@notebook1003:~$ python3.7
Python 3.7.1 (default, Dec 16 2018, 12:33:36)
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

Caveat: since those nodes are still Stretch (that ships with 3.5), we had to use backports only for the interpreter and the standard library. This means that there might be packages in the Debian stretch repos requiring python3.5 and not working with 3.7, therefore python3.7 is usable only to bootstrap it in a virtual environment. Let me know if this works :)

Nuria closed this task as Resolved.Aug 26 2019, 3:43 PM
nshahquinn-wmf reopened this task as Open.Aug 28 2019, 2:34 PM

As FYI we have now Python3.7 + libpython3.7 on notebooks:

Caveat: since those nodes are still Stretch (that ships with 3.5), we had to use backports only for the interpreter and the standard library. This means that there might be packages in the Debian stretch repos requiring python3.5 and not working with 3.7, therefore python3.7 is usable only to bootstrap it in a virtual environment. Let me know if this works :)

Very neat! To clarify, this means we can now use Python 3.7 via the command line, but not in the SWAP installation of Jupyter, right? In that case, I'd like for us to keep this task open, since the ultimate ask is to be able to use Python 3.6 or greater without having to leave SWAP.

elukey added a comment.Sep 9 2019, 6:38 AM
In T212591#5445917, @Neil_P._Quinn_WMF wrote:

As FYI we have now Python3.7 + libpython3.7 on notebooks:

Caveat: since those nodes are still Stretch (that ships with 3.5), we had to use backports only for the interpreter and the standard library. This means that there might be packages in the Debian stretch repos requiring python3.5 and not working with 3.7, therefore python3.7 is usable only to bootstrap it in a virtual environment. Let me know if this works :)

Very neat! To clarify, this means we can now use Python 3.7 via the command line, but not in the SWAP installation of Jupyter, right? In that case, I'd like for us to keep this task open, since the ultimate ask is to be able to use Python 3.6 or greater without having to leave SWAP.

You can use python3.7 in your venv when you create the notebook, so in theory after that you should be able to use it (but I could be wrong, very ignorant about SWAP currently). The main trouble right now is using python3.7 via "CLI" since without the right venv it cannot install libraries (since the system ones are not compatible, they only work with 3.5).

You can use python3.7 in your venv when you create the notebook

All of the default python notebook kernels we make available in JupyterHub use the same auto created venv. It *might* be possible to make a python 3.7 based venv and then create a custom kernel for it, but I haven't tried. (These instructions are for spark but the basic idea is the same and can be used to create a new python kernel.)

elukey added a comment.Sep 9 2019, 1:30 PM

Ok I thought it was possible to create your own venv and use that, but I was wrong.

nshahquinn-wmf renamed this task from Provide Python 3.6 on SWAP to Provide Python 3.6+ on SWAP.Sep 9 2019, 4:11 PM

It looks like Pandas 1.0, which should be released in the next month or so, will require Python 3.6.1 or higher.

Bokeh 2.0, which was recently released, requires Python 3.6 or higher as well.

You can use python3.7 in your venv when you create the notebook

All of the default python notebook kernels we make available in JupyterHub use the same auto created venv. It *might* be possible to make a python 3.7 based venv and then create a custom kernel for it, but I haven't tried. (These instructions are for spark but the basic idea is the same and can be used to create a new python kernel.)

@Ottomata is anything different now that we have jupyterhub on swap nodes with Buster? (like stat1005/stat1008, not yet ready but close to be)

Ottomata added a comment.EditedMar 13 2020, 7:31 PM

Likely the SWAP wheels that you've built for buster use Python 3.7, right? So, when folks log into it on a Buster node, they will get a copy of that Python 3.7 venv.

I look forward for the day when we don't have to deal with stale copies of venvs anymore.

elukey closed this task as Resolved.Apr 14 2020, 2:28 PM
elukey claimed this task.
elukey set Final Story Points to 3.

Please log on stat1005 or stat1008 to get a Python 3.7 venv :)