Page MenuHomePhabricator

Document how to add library support in PAWS
Open, LowPublic

Description

Hello,

So I am starting to learn tensorflow, and thought I could use PAWS to run my scripts, all the more as I would like to see what I can do with this tool for Wikimedia projects.

But so far, even the most basic script trying to import the required libraries fails for me.

Is there a way to install a library, like saying to jupyter "pip install tensorflow", or any other way?

Maybe I missed it, but I didn't find explanations on that topic in the PAWS documentation.

Cheers

Event Timeline

Chicocvenancio triaged this task as Low priority.Mar 27 2019, 3:00 PM
Chicocvenancio moved this task from Backlog to Easy tasks on the PAWS board.
Chicocvenancio added a subscriber: Chicocvenancio.

This specific request can be solved by adding a !pip install tensorflow at the beginning of the notebook.
But yeah, we need to document this.

Actually, while the command does try to install the missing module, it fails to successfully do so:

Building wheels for collected packages: gast, termcolor, absl-py
  Running setup.py bdist_wheel for gast ... error
  Complete output from command /srv/paws/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-vfle1knv/gast/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmpso9dvo1ypip-wheel- --python-tag cp36:
  usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
     or: -c --help [cmd1 cmd2 ...]
     or: -c --help-commands
     or: -c cmd --help
  
  error: invalid command 'bdist_wheel'
  
  ----------------------------------------
  Failed building wheel for gast
  Running setup.py clean for gast
  Running setup.py bdist_wheel for termcolor ... error
  Complete output from command /srv/paws/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-vfle1knv/termcolor/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmp41_9n52bpip-wheel- --python-tag cp36:
  usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
     or: -c --help [cmd1 cmd2 ...]
     or: -c --help-commands
     or: -c cmd --help
  
  error: invalid command 'bdist_wheel'
  
  ----------------------------------------
  Failed building wheel for termcolor
  Running setup.py clean for termcolor
  Running setup.py bdist_wheel for absl-py ... error
  Complete output from command /srv/paws/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-vfle1knv/absl-py/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmp5q76k_wrpip-wheel- --python-tag cp36:
  /usr/lib/python3.6/distutils/dist.py:261: UserWarning: Unknown distribution option: 'long_description_content_type'
    warnings.warn(msg)
  usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
     or: -c --help [cmd1 cmd2 ...]
     or: -c --help-commands
     or: -c cmd --help
  
  error: invalid command 'bdist_wheel'
  
  ----------------------------------------
  Failed building wheel for absl-py
  Running setup.py clean for absl-py
Failed to build gast termcolor absl-py
Installing collected packages: wheel, gast, h5py, keras-applications, termcolor, keras-preprocessing, absl-py, pbr, mock, tensorflow-estimator, protobuf, markdown, grpcio, tensorboard, astor, tensorflow
  Running setup.py install for gast ... done
  Running setup.py install for termcolor ... done
  Running setup.py install for absl-py ... done
Successfully installed absl-py-0.7.1 astor-0.7.1 gast-0.2.2 grpcio-1.19.0 h5py-2.9.0 keras-applications-1.0.7 keras-preprocessing-1.0.9 markdown-3.1 mock-2.0.0 pbr-5.1.3 protobuf-3.7.1 tensorboard-1.13.1 tensorflow-1.13.1 tensorflow-estimator-1.13.0 termcolor-1.1.0 wheel-0.33.1

Is there something I can do about it @Chicocvenancio ?

Oups, my bad, actually, despite the error message, it works. I guess that the error are related to some admin (wheel) specific stuff that aren't really required to make a module installed in a functional manner.

Thank you again!

@Psychoslave yeah, that's just telling you it could not install some of the requirements by one method and tried others successfully. Mind that you will have to run the cell again if/when your PAWS server is restarted.

Thank you for the feedback @Chicocvenancio.

Almost out of topic but I thought I might document that on the wiki. Maybe I already have a developer account, but I didn't found the relevant password nor a way to get back a reset quickly if that the case. I saw that there was some documentation on the topic, but didn't have time yet to go through it. Hopefully I can find a bit of time for that this week-end, although I would rather focus on raising my skills on tensorflow and more, so any guidance that would help me to go quicker on that account setup/reset would be very welcome.

I can help you set up a developer account on another task or telegram, if you wish. But the preferred wiki for documentation would be mediawiki.org. PAWS is the first line of contact for a lot of Wikimedia developers and mediawiki is a friendlier and more visible alternative.

By the way @Chicocvenancio, what about making most popular module preinstalled?

I don't know how popular tensorflow is among PAWS users, but surely they are module that are popular and that would be convenient to have installed out of the box.

By the way @Chicocvenancio, what about making most popular module preinstalled?

I don't know how popular tensorflow is among PAWS users, but surely they are module that are popular and that would be convenient to have installed out of the box.

Its a trade off, every new package pre-installed into a server means a larger startup time and greater resource usage for all servers. In general I would want to use the minimum number of packages, while leaving novices able to do their work without an added step. Tensorflow, in my view, does not add sufficient value for novice users to offset the added usage for every server startup.

I could be convinced otherwise with more users calling for it to be installed.