Page MenuHomePhabricator

Job queue runners for MediaWiki-Docker
Closed, ResolvedPublic

Description

The Docker image appears to start up a web server, but no job queue runner process. As a result, background tasks may not be handled well or at all -- for instance TimedMediaHandler video transcode jobs appear not to be executed at all.

Steps to reproduce:

  • install MW in Docker
  • install TimedMediaHandler extension
    • don't forget to run composer and update.php
  • fix the PHP upload settings to allow files over 2 MiB (T246930)
  • increase $wgMaxShellMemory (seems to be needed to get ffmpeg working)
  • upload a WebM video file on Special:Upload

Actual results:

  • No ffmpeg processes are launched
  • Transcode status at bottom of page remains stuff like "Added to Job queue 7 minutes, 11 seconds ago"

Expected results:

  • An ffmpeg process should be launched nearly immediately, running one of the transcodes
  • Transcode status at bottom of page should update as files complete

Workaround:
In a docker shell, manually run maintenance/runJobs.php or similar.

Event Timeline

I'm not sure about what the "80% of use cases" behavior should be here.

We could use Redis for job queue and have a cron job in the container that's regularly processing it, without too much trouble. On the other hand running docker-compose exec mediawiki maintenance/runJobs.php via the CLI might not be a bad alternative, because you might want to debug a job or have more control over when the job queue is processed.

I think principle of least surprise leads us to want a job queue that is processed automatically without manual intervention, as it's meant to be a continuously-operating part of the MediaWiki service that code can rely on being in working condition.

That said, sometimes you want to debug some detail of job execution, so having some way to disable the automatic processing would probably be nice.

Change 597826 had a related patch set uploaded (by Brennen Bearnes; owner: Brennen Bearnes):
[releng/dev-images@master] add stretch-php72-jobrunner for TimedMediaHandler

https://gerrit.wikimedia.org/r/597826

Change 597844 had a related patch set uploaded (by Brennen Bearnes; owner: Brennen Bearnes):
[mediawiki/core@master] DNM: docker: add mediawiki-jobrunner

https://gerrit.wikimedia.org/r/597844

Change 597826 merged by Jforrester:
[releng/dev-images@master] add jobrunner & tweak settings for TimedMediaHandler

https://gerrit.wikimedia.org/r/597826

Change 597844 merged by jenkins-bot:
[mediawiki/core@master] mediawiki-docker: Add a jobrunner container

https://gerrit.wikimedia.org/r/597844

Mentioned in SAL (#wikimedia-releng) [2020-06-17T19:46:02Z] <brennen> Updating dev-images docker-pkg files on primary contint for jobrunner changes (T246935)

brennen moved this task from Needs/Waiting Review to Done / Defunct on the User-brennen board.
brennen moved this task from In Progress to Done on the MediaWiki-Docker board.