Page MenuHomePhabricator

Create a Perl Docker image for use on the Toolforge Kubernetes cluster
Closed, ResolvedPublic

Description

Requested in https://lists.wikimedia.org/pipermail/cloud/2019-January/000517.html

We may be able to make a smaller base image by making sure that we install local::lib and carton.

Our Docker containers tend to implement opinionated patterns for deploying and running code. It would be good to get feedback from actual Perl webservice authors on options for this. Examples:

  • Same as grid engine default: run $HOME/public_html/*.pl files as CGI scripts
  • Run a FastCGI app from $HOME/www/perl/src
  • Run a PSGI app from $HOME/www/perl/src using uWSGI (similar to our Python2 and Python3 defaults)
  • PSGI run under Starman
  • ...

Event Timeline

i'd like to test somehow, which perl-modules are missing.

one answer in the mailing list was

webservice --backend=kubernetes TYPE shell will get you into a shell.

but that gave me just a message:
type must be one of golang,jdk8,nodejs,php5.6,php7.2,python,python2,ruby2,tcl

(however, i guess, i just need the debian package libhtml-template-perl for the perl module HTML::Template.)

type must be one of golang,jdk8,nodejs,php5.6,php7.2,python,python2,ruby2,tcl

We do not have any image at all for Perl yet, so that explains your failure with webservice --backend=kubernetes perl shell.

(however, i guess, i just need the debian package libhtml-template-perl for the perl module HTML::Template.)

Have you used any project local package manager (something that does not need root permissions) with your Perl projects in the past? I think our ideal solution would be something similar Python's venv or PHP's Composer which would allow each tool to install the 3rd party libraries it needs without us needing to place the union of all possible modules used by tools in the Perl Docker image itself. Ideally we would just be installing a core Perl runtime and the libraries/tools needed for local package management and then letting each tool maintainer install the extras their tool needs.

I've used CPAN locally at the old toolserver (ran by DaB.). Since there is labs, I just asked for some packages, and the admins installed them. For me this is an easy way.

In my opinion local package managing systems are a PITA. It's better to have centralized and centrally managed _system_ package bundles, where one can be sufficiently sure that everything works together.
All perl modules I use are packed for debian already, e.g.,
sudo apt-get install libcgi-pm-perl libdatetime-format-strptime-perl libhtml-template-perl libwww-perl
They are installed at the 'normal' labs server already.

I'm not used to work with docker images. I understand that those docker images should be small, because every tool get's its own image, right? So if it would be possible to install system packages in that container, that would be great, maybe something such as https://perlmaven.com/distributing-perl-script-using-docker?

Another possibility is to use lighttpd as on gridengine and just install missing modules like CGI.pm (created T250975 for that, but feel free to close as dupe)

Perl running webservices are still a blocker for gridengine to be replaced by kubernetes

The CheckWiki project is written in Perl 5. It already has a local library for addon Perl modules. It has both a webservice and grid jobs.

Change 778683 had a related patch set uploaded (by BryanDavis; author: Bryan Davis):

[operations/docker-images/toollabs-images@master] Add perl532-sssd

https://gerrit.wikimedia.org/r/778683

Change 778683 merged by jenkins-bot:

[operations/docker-images/toollabs-images@master] Add perl532-sssd

https://gerrit.wikimedia.org/r/778683

Mentioned in SAL (#wikimedia-cloud) [2022-04-23T16:51:28Z] <bd808> Built new perl532-sssd/{base,web} images and pushed to registry (T214343)

Change 785374 had a related patch set uploaded (by BryanDavis; author: Bryan Davis):

[operations/software/tools-webservice@master] k8s: add perl5.32 type

https://gerrit.wikimedia.org/r/785374

Change 785374 merged by jenkins-bot:

[operations/software/tools-webservice@master] k8s: add perl5.32 type

https://gerrit.wikimedia.org/r/785374

Change 785375 had a related patch set uploaded (by BryanDavis; author: Bryan Davis):

[operations/software/tools-webservice@master] d/changelog: Prepare for 0.82 release

https://gerrit.wikimedia.org/r/785375

Change 785375 merged by jenkins-bot:

[operations/software/tools-webservice@master] d/changelog: Prepare for 0.82 release

https://gerrit.wikimedia.org/r/785375

Mentioned in SAL (#wikimedia-cloud) [2022-04-25T14:56:20Z] <bd808> Rebuilding all docker images to pick up toolforge-webservice v0.82 (T214343)

bd808 claimed this task.

We did not come up with a good plan for self-service perl module management, but there is now a webservice --backend=kubernetes perl5.32 ... container which runs lighttpd with config to serve .pl files found in $HOME/public_html as CGI scripts. I believe this matches the general level of support we have for perl in webservice on the grid engine backend.

Change 789878 had a related patch set uploaded (by BryanDavis; author: Bryan Davis):

[cloud/toolforge/jobs-framework-api@main] containers: Add perl 5.32 container

https://gerrit.wikimedia.org/r/789878

Change 789878 merged by jenkins-bot:

[cloud/toolforge/jobs-framework-api@main] containers: Add perl 5.32 container

https://gerrit.wikimedia.org/r/789878

A Perl container would need to have MediaWiki::Bot

A Perl container would need to have MediaWiki::Bot

We have never installed MediaWiki::Bot on the job grid, so I'm not sure why it would be required in a container.

It is there on the job grid and is used by my Perl bots.

perl532-sssd/base/Dockerfile.template has a list of the packages in the container. libmediawiki-bot-perl is listed.

Is there any way to test it?

become $TOOL; webservice perl5.32 shell will give you an interactive shell inside the perl container. From there you should be able to use perl directly and/or run scripts.

My bots uses one Perl module that is not installed:
File::Slurp

The checkwiki project has its extra modules installed in a project sub folder and uses a lib statement to add them to the search path.
Example: use lib '/data/project/checkwiki/perl/lib/perl5';

Yes, I could do that but it is not my preferred option because:
(1) The module is currently available on the grid
(2) It defeats the purpose of having a perl container

Of course, I handle my own modules this way.

using a use lib card is messy, as the path will not exist on the development machine, so better to include a path on the call line eg
perl -I File-Slurp-9999.32/lib bin/use.pl

Closed but not resolved

Please open a new bug report for File::Slurp being missing from the container.