Page MenuHomePhabricator

Create Docker image for Toolforge that is purpose built to run pywikibot scripts
Closed, ResolvedPublic

Description

Many grid engine jobs are executions of pywikibot scripts. An unknown but presumably large number of these are using some variation of the recommended pywikibot process.

The WMCS team would like to find ways to reduce folks dependence on the grid engine. Making a simple way for folks to run pywikibot on the Kubernetes grid seems like a good place to start on this larger goal.

In response to a random musing on IRC, @JJMC89 reported that pywikibot makes a stable release ~4-7 times per year [ 1 ]. This seems like an easy pace to keep up with even without a CI/CD system for updating our Docker images.

I'm thinking a first attempt at this could look something like a docker image based on whatever our latest py3 base container is with pywikibot, its direct dependencies, and most/all of the currently globally installed python3 packages included. This would be similar to the upstream Dockerfile with a few other bells and whistles.

This image should also do as many convention over configuration things as possible with the ultimate goal being that running a pywikibot workload on the Kubernetes cluster looks something like (fake commands ahead!):

$ become my-cool-pwb-tool
$ pwb-k8s init
Checking for local scripts directory...
Local scripts directory not found. Creating $HOME/pwb
Checking for user files...
User files not found. Running generate_user_files.py
[whatever generate_user_files.py does happens here]
$ pwb-k8s run version.py
Pywikibot: [https] r-pywikibot-core.git (df69134, g1, 2020/03/30, 11:17:54, OUTDATED)
Release version: 3.1.dev0
requests version: 2.12.4
  cacerts: /etc/ssl/certs/ca-certificates.crt
    certificate test: ok
Python: 3.5.3 (default, Sep 27 2018, 17:25:39)
[GCC 6.3.0 20170516]
$ vim pwb/my_cool_script.py
$ pwb-k8s cron --hour 3 --minute 17 my_cool_script.py
CronJob created
$ kubectl get cronjob
NAME                    SCHEDULE     SUSPEND   ACTIVE   LAST SCHEDULE   AGE
my_cool_script.py       17 3 * * *   False     0        0m              0m
$ kubectl describe cronjob my_cool_script.py
Name:                          my_cool_script.py
Namespace:                     tool-my-cool-pwb-tool
Labels:                        name=my-cool-pwb-tool.my_cool_script.py
                               toolforge=tool
Annotations:                   <none>
Schedule:                      17 3 * * *
Concurrency Policy:            Allow
Suspend:                       False
Successful Job History Limit:  3
Failed Job History Limit:      1
Starting Deadline Seconds:     <unset>
Selector:                      <unset>
Parallelism:                   <unset>
Completions:                   <unset>
Pod Template:
  Labels:  toolforge=tool
  Containers:
   bot:
    Image:      docker-registry.tools.wmflabs.org/toolforge-python37-sssd-pwb:latest
    Port:       <none>
    Host Port:  <none>
    Args:
      python3
      /data/project/my-cool-pwb-tool/pwb/my_cool_script.py
    Environment:
      PYWIKIBOT_DIR:  /data/project/my-cool-pwb-tool/pwb
      HOME:           /data/project/my-cool-pwb-tool
    Mounts:           <none>
  Volumes:            <none>
Last Schedule Time:   <none>
Active Jobs:          <none>
Events:               <none>

Event Timeline

That all looks awesome, especially the cron part.

generate_user_files.py will generate a user-config.py file which needs to be in $PYWIKIBOT2_DIR.

Also, if we could autogenerate the https://www.mediawiki.org/wiki/Manual:Pywikibot/OAuth/Wikimedia part that would be nice.

And the cron thing could automatically redirect logs (both stdout and stderr) to somewhere and rotate them.

Just for the record: a stable release is deployed monthly after tests pass. Sometimes it needs some additional time to fix all known issues for that release. Every stable release is marked with the „stable“ tag. This is useful if the local repository is checked out from the remote repository (which also includes common scripts to be used) instead as a side package. The last stable release was published yesterday.

Also, if we could autogenerate the https://www.mediawiki.org/wiki/Manual:Pywikibot/OAuth/Wikimedia part that would be nice.

I think we could easily output a link to those instructions. I'm not sure that there is any reasonable way to actually automate submitting the grant request. If there was, I think that would be an XSS attack vector that we would need to fix on the MediaWiki side.

Change 603652 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[operations/docker-images/toollabs-images@master] Pywikibot container

https://gerrit.wikimedia.org/r/603652

Change 603652 merged by jenkins-bot:
[operations/docker-images/toollabs-images@master] Pywikibot container

https://gerrit.wikimedia.org/r/603652

nskaggs added a subscriber: Bstorm.

Change 764300 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[cloud/toolforge/jobs-framework-api@main] deployment: add tf-python39-pwb image

https://gerrit.wikimedia.org/r/764300

Change 764300 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[cloud/toolforge/jobs-framework-api@main] deployment: add tf-python39-pwb image

https://gerrit.wikimedia.org/r/764300

With this patch we could allow the piwikibot container for the toolforge-jobs tool, which should enable a similar workflow as described in the task description.

looking at the associated gerrit patches I assume this issue has been fixed and should be closed @bd808 ?

looking at the associated gerrit patches I assume this issue has been fixed and should be closed @bd808 ?

We do have a container, but I don't think we have really done any work to document its existence on wiki. It is also not integrated with the jobs framework yet (there is a patch, but it is not merged yet). I didn't write any useful notes here, but I think I remember that this stalled out in part because of a lack of planning for how we would make sure that the image was updated when new stable builds of pywikibot are released.

Another thing that would be nice here would be something like the $ pwb-k8s init action that I proposed in the task description to guide new users through whatever setup is needed. This could be implemented as some helper script that is packaged inside the container.

looking at the associated gerrit patches I assume this issue has been fixed and should be closed @bd808 ?

We do have a container, but I don't think we have really done any work to document its existence on wiki. It is also not integrated with the jobs framework yet (there is a patch, but it is not merged yet). I didn't write any useful notes here, but I think I remember that this stalled out in part because of a lack of planning for how we would make sure that the image was updated when new stable builds of pywikibot are released.

Another thing that would be nice here would be something like the $ pwb-k8s init action that I proposed in the task description to guide new users through whatever setup is needed. This could be implemented as some helper script that is packaged inside the container.

Acked

I believe I'm the main reason why the jobs framework patch wasn't merged yet. Although I don't think I've ever voiced my concerns outside of IRC chats which might be hard to find, so copying the latest conversation here (with permission from everyone involved):

20:06:52 <+wm-bb> <MaartenDammers> Did you consider making a docker image that supports pywikibot without venv out of the box?
20:08:27 <+wm-bb> <MaartenDammers> Or does https://wikitech.wikimedia.org/wiki/Help:Toolforge/Pywikibot need to be updated? I see we have https://phabricator.wikimedia.org/diffusion/ODIT/browse/master/python39-sssd/pwb/Dockerfile.template
20:09:40 <taavi> yes, T249787. iirc the last time someone (bd808) was working on that I was concerned on how to manage the pywikibot version in a way that doesn't break anyones code without warning but still lets us offer an up-to-date versions
20:09:41 <+stashbot> T249787: Create Docker image for Toolforge that is purpose built to run pywikibot scripts - https://phabricator.wikimedia.org/T249787
20:13:00 <+wm-bb> <MaartenDammers> Yes. That's an issue. You could link each image to a specific branch?
20:15:03 <taavi> PWB uses SemVer, right? I guess one option would be to offer images like python39-pwb7 for 7.x, -pwb8 for 8.x and so on
20:17:38 <+wm-bb> <MaartenDammers> And python39-pwb-stable / python39-pwb-current for the lazy maintainers? 😊
20:21:54 <taavi> something like that, yeah

If we end up implementing those kind of -stable aliases, those would have to be done on the build.py/docker registry level as the image name is converted to the actual container url during job creation.

We discussed this a bit on Telegram/IRC (wikimedia-cloud) triggered by the grid to Kubernetes migration (as pasted above). I'll try to describe my use case a bit.

I have all sorts of jobs running.

  • All are based on Pywikibot, most of them are in https://github.com/multichill/toollabs/tree/master/bot
  • Jobs are in multiple accounts
  • I use pywikibot current (master) which I rarely update (more about that later)
  • I use a venv because at some point direct usage stopped working
  • Jobs all use jsub from a crontab. Hourly, daily, weekly jobs somewhat evenly distributed over time
  • Development is done on my local laptop(s) and when deemed stable enough, moved to a job

So my basic workflow is pretty straightforward: Write or update some tool locally. Test it. Commit it. Git pull on Toolforge and create a jsub job if it isn't there yet.
I'm using the current (master) branch because that's the one I'm also using locally. Updating pywikibot usually causes breakage. I don't like tools breaking, but I really hate it when tools break at some random time because someone else did a software change, so that's why I'm controlling that part myself.

I doubt pinning to pywikibot7.x and pywikibot8.x will provide enough stability (based on past experience), so we might have to go for python39-pwb77 and python39-pwb78

I'm using the current (master) branch because that's the one I'm also using locally.

If I'm understanding this statement correctly you are currently using your own git clone of pywikibot and not the any of the /shared/pywikibot/... checkouts which are maintained by the pywikibot tool. If this is the case you can keep using the same workflow from Kubrernetes by choosing a Python runtime and maintaining your own pywikibot clone in your tool's $HOME.

Some remarks:

Pywikibot versioning
Master branch should not be used for production systems but many thanks for all test which are done with its usage ;-). Anyway using it for a docker images is not a good idea: Commits are made continuously but the development counter is not changed. And there might be new features which are not tested for a longer time and is not sure whether it always works right even we have deeptests running before merging.

For example Pywikibot 8 drops support for Python < 3.6.1 and MW < 1.27, prohibits some positional arguments of methods and maybe comes with new features: 2FA login support, rewritten token handling, expanded http exception handling, MW API format2 usage, installing as site-package from git repository, entry point for foreign scripts.

The minimal releas supporting Wikimedia sites is 6.6.1. All versions are following semver since 6.5.0 which means there are breaking changes expected in 7.0.0 or 8.0.0 against older releases but no breaking changes are made between major releases (but deprecation warnings may be thrown). All changes can be found in the changelog.

All releases are tagged with the version number which can be used to collect older releases. The stable branch is always the last publisched releas which is not under development. Patch releases like 7.7.1 are mostly backported changes from master to stable branch which contains L10N updates for new Wikimedia sites like in 7.7.1, 6.6.2-6.6.5 or important bugfixes like in 7.5.2, 7.5.1, 7.2.1, 7.1.1.

Docker
There are two dockerfile creation files shipped with pywikibot:

The last Update was 2020 when release 4.2 was published.

Python 3.10 looks stable enough to use it with the docker image but has few advantages like case-statements, better debugging support, TypeGuard etc.

Currently Pywikibot is only published on Python Package Index. Is it appropriate to publish releases on docker hub too?

Change 764300 abandoned by Arturo Borrero Gonzalez:

[cloud/toolforge/jobs-framework-api@main] deployment: add tf-python39-pwb image

Reason:

not working on this at the moment

https://gerrit.wikimedia.org/r/764300

Change 916795 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/docker-images/toollabs-images@master] Remove -pwb images in favour of upcoming buildservice images

https://gerrit.wikimedia.org/r/916795

Change 916795 merged by jenkins-bot:

[operations/docker-images/toollabs-images@master] Remove -pwb images in favour of upcoming buildservice images

https://gerrit.wikimedia.org/r/916795

Change 916795 merged by jenkins-bot:

[operations/docker-images/toollabs-images@master] Remove -pwb images in favour of upcoming buildservice images

https://gerrit.wikimedia.org/r/916795

Using buildservice as the basis for pywikibot tools is generally a good thing I think. Rather than providing a base image with external helpers as I initially imagined we could instead think about maintaining a GitLab project that provides a functional base for pywikibot buildservice containers. Folks could then fork this repo to start their own bot specific repo. Hopefully this would come with some instructions for how to keep things updated as newer versions of pywikibot are released.

Currently Pywikibot is only published on Python Package Index. Is it appropriate to publish releases on docker hub too?

Toolforge does not currently support "bring your own container" workflows that would be able to take advantage of a Pywikibot container from Docker hub or any other non-Toolforge container repository. There may be other reasons that publishing tagged containers would be useful for the Pywikibot community however.

This topic was discussed in the latest Toolforge admin meeting. To summarize:

taavi raised the priority of this task from Medium to High.

Per today's team WMCS meeting.

  • How that image will be built and made available to users is still an open question.

I sent an email to the Pywikibot list about this.

Image published, and I wrote some docs on https://wikitech.wikimedia.org/wiki/Help:Toolforge/Running_Pywikibot_scripts. I'm calling this done.