Can't it be installed via pip?

In T166985#3316127, @zhuyifei1999 wrote:

Can't it be installed via pip?

No, as pip is only for Python packages. And those are regular binaries, you'd install with your distro package manager (apt-get, yum install, etc..)

@Krinkle, 10x for more sane description of task.

(oh I thought you meant install the python requests package)

Packages get installed via Puppet. The list of packages installed in toollabs, on exec nodes, is:

puppet/modules/toollabs/manifests/exec_environ.pp in the operations/puppet repo. See the "package{}" resources in there.

Question is though how the currently installed packages are not listed there.

The k8s packages are installed via Dockerfiles at rODIT.

In T166985#3316302, @Dzahn wrote:

Question is though how the currently installed packages are not listed there.

The packages for the Docker images used in the Kubernetes cluster are not handled by Puppet. They are instead managed with Dockerfiles generated by code in the rODIT operations-docker-images-toollabs-images repository.

@Dzahn I believe this user is in the k8s environment which is handled separately from SGE

I imagine the current state is explained with:

tools-bastion-03:~$ whereis pdftotext

pdftotext: /usr/bin/pdftotext /usr/bin/X11/pdftotext /usr/share/man/man1/pdftotext.1.gz

tools-exec-1414:~$ whereis pdftotext

pdftotext: /usr/bin/pdftotext /usr/bin/X11/pdftotext /usr/share/man/man1/pdftotext.1.gz

tools-worker-1001:~$ whereis pdftotext

pdftotext:

Where tools-worker-1001 is an eligible worker to run a container in question. I'm not sure we have worked out a process for new packages that are system wide on the k8s nodes. I am marking this task to talk over in our team meeting tomorrow.

pdftotext: /usr/bin/pdftotext /usr/bin/X11/pdftotext /usr/share/man/man1/pdftotext.1.gz

@Xelgen, can you run the tool you would like to build on Grid Engine rather than Kubernetes? It sounds like the software you need is available there today.

@bd808 Frankly 2 weeks ago I've found workaround for djvutoxml - I copied binary and it's libraries from Jessie installation to python venv bin folder, and then alternated binary with patchelf so it will look for shared libraries in new destination, and this works for me now.
But it's a really duct-tape solution, mundane to keep updated and secure in future, plus I believe there are at least half a dozen tools using djvutoxml, so it's something which other people will need when they'll migrate to Kubernetes.

As far as I understand in long term we plan to move from Grid Engine to K8s. If that's the case I may try to stick with K8s, make same trick to pdftotext utility, and when things will be sorted out, I'll just have to delete those modified files, and app will automatically switch to system wide binaries.

In T166985#3322895, @Xelgen wrote:

As far as I understand in long term we plan to move from Grid Engine to K8s. If that's the case I may try to stick with K8s, make same trick to pdftotext utility, and when things will be sorted out, I'll just have to delete those modified files, and app will automatically switch to system wide binaries.

Kubernetes is certainly hoped to be the more stable and full-featured replacement for grid engine. We are at an awkward phase of the project today however due mostly to a lack of staff to work on proper solutions for the new platform. The current webservice --backend=kubernetes system is a reasonable match for the use case of a tool that runs a single web process and can use language specific library management (composer for PHP, pip + virtualenv for Python, etc) to get all the things it needs. The Docker containers we are making available are limited to a single language runtime and install a very minimal set of additional libraries and binaries. We have about 230 tools using Kubernetes today which is about 1/3rd of the tools that probably could move over with minimal changes to the tool's code.

A year ago we thought that we would be able to create a giant Docker container that replicated the 'everything for everyone' setup that the grid engine hosts have been using. About six months ago @yuvipanda finally got around to trying to build such a container (T152089). The outcome there is not well documented, but it failed badly. Since building a giant monolith container is now off of the table as an intermediate solution, we need to rethink the near term plan for Kubernetes. The hoped for long term solution is still to select a FLOSS Platform as a Service solution that works with Kubernetes as a backend (T136264).

Unfortunately, the cloud-services-team does not currently have the resources to work on the PaaS project for the foreseeable future. We are currently down one Operations Engineer (we are hiring!) and we plan to add one more person to the team in the upcoming Wikimedia Foundation fiscal year (July 2017-June 2018). Adding more people will help with some of our issues, but we are planning to focus on paying down some very long term technical debt in our OpenStack environment and doing more outreach to our current and future customers in the coming year's plan. There is more detail on this in the Foundation annual plan draft on meta and our team tracking board for annual plan work.

Without the time and people needed to evaluate, select, and deploy a proper PaaS layer over Kubernetes, we are left with a couple of choices. We can add new apt managed packages to the Docker containers reactively as people request them on tickets like this one. Each time we do that we are a step closer to the currently unknown tipping point of having an image that is too big to manage effectively. Alternately we can suggest that tools which have needs that cannot be met by the current Docker containers stick with grid engine as their deployment runtime. That path doesn't help the maintainers of those tools move to the more robust Kubernetes platform, but it also doesn't pile on more tech debt for the cloud-services-team to unwind in the future.

The ideal mid-term solution would be for a small group of volunteers who are very interested in seeing Kubernetes be more widely used to become involved in moving things forward. There are very few things that need to be done in the evaluation process that can only be handled by paid staff. The process of creating the list of features that should be compared in PaaS solutions (T136265) and testing products against those criteria are things that could largely be done by volunteers with a bit of support from the cloud-services-team. With a surge of support for this work from our community, I would be willing to change the team's priorities and goals to find resources to assist in making Kubernetes a more general solution sooner rather than later. Without that kind of infusion of energy and resources however, a better Kubernetes PaaS is going to have to wait until we finish other projects that have been determined to be more critical.

@Xelgen If you desire you may compile the two utilities locally for your tool, instead of copying and altering the binaries themselves. I tested them and pasted the logs (much of the irrelevant outputs removed):

Some of the downsides would be:

• every time the binaries are called, or other compilation is needed, a ton of environment variables must be set for them to load/compile correctly (PATH=~/.local/bin:$PATH CPATH=~/.local/include LIBRARY_PATH=~/.local/lib PKG_CONFIG_PATH=~/.local/lib/pkgconfig).
• no automatic upgrades via apt, although you might be able to make a script to do so by running the commands to download, compile, & install.
• every time the system have a major upgrade (eg. jessie => stretch) backwards-compatibility may be broken and the programs may need to be recompiled.

PS: This is the fourth time I've done such local compiling/installing on tool labs. Most C/C++/Assembly programs, including larger programs like FFmpeg (which I compiled for grid for the embeddeddata bot), should work fine when installed this way.

I just remembered that Gentoo has a package manager, Portage, that installs packages by compiling them locally. If we would not allow users to define apt installs (because it requires root), would emerge be a solution?

In T166985#3336974, @zhuyifei1999 wrote:

I just remembered that Gentoo has a package manager, Portage, that installs packages by compiling them locally. If we would not allow users to define apt installs (because it requires root), would emerge be a solution?

If there is a non-root emerge equivalent for Debian Jessie, yes. Another option would be to install something like GNU stow in the images to at least make custom compilation a bit easier. The real solution however is T136264: Evaluate Kubernetes based workflow replacement options for SGE and ensuring that the selected solution supports some reasonable mechanism for building containers with custom software packages included. An example of this in other PaaS products is heroku-buildpack-apt.