Page MenuHomePhabricator

Job queue runners for MediaWiki-Docker
Closed, ResolvedPublic

Description

The Docker image appears to start up a web server, but no job queue runner process. As a result, background tasks may not be handled well or at all -- for instance TimedMediaHandler video transcode jobs appear not to be executed at all.

Steps to reproduce:

  • install MW in Docker
  • install TimedMediaHandler extension
    • don't forget to run composer and update.php
  • fix the PHP upload settings to allow files over 2 MiB (T246930)
  • increase $wgMaxShellMemory (seems to be needed to get ffmpeg working)
  • upload a WebM video file on Special:Upload

Actual results:

  • No ffmpeg processes are launched
  • Transcode status at bottom of page remains stuff like "Added to Job queue 7 minutes, 11 seconds ago"

Expected results:

  • An ffmpeg process should be launched nearly immediately, running one of the transcodes
  • Transcode status at bottom of page should update as files complete

Workaround:
In a docker shell, manually run maintenance/runJobs.php or similar.

Event Timeline

brion created this task.Mar 4 2020, 7:23 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 4 2020, 7:23 PM
kostajh added a subscriber: kostajh.Mar 4 2020, 8:51 PM

I'm not sure about what the "80% of use cases" behavior should be here.

We could use Redis for job queue and have a cron job in the container that's regularly processing it, without too much trouble. On the other hand running docker-compose exec mediawiki maintenance/runJobs.php via the CLI might not be a bad alternative, because you might want to debug a job or have more control over when the job queue is processed.

brion added a comment.Mar 4 2020, 9:45 PM

I think principle of least surprise leads us to want a job queue that is processed automatically without manual intervention, as it's meant to be a continuously-operating part of the MediaWiki service that code can rely on being in working condition.

That said, sometimes you want to debug some detail of job execution, so having some way to disable the automatic processing would probably be nice.

brennen moved this task from Backlog to In Progress on the User-brennen board.Apr 17 2020, 4:28 PM
brennen moved this task from In Progress to Next on the User-brennen board.May 13 2020, 4:56 PM

Change 597826 had a related patch set uploaded (by Brennen Bearnes; owner: Brennen Bearnes):
[releng/dev-images@master] add stretch-php72-jobrunner for TimedMediaHandler

https://gerrit.wikimedia.org/r/597826

Change 597844 had a related patch set uploaded (by Brennen Bearnes; owner: Brennen Bearnes):
[mediawiki/core@master] DNM: docker: add mediawiki-jobrunner

https://gerrit.wikimedia.org/r/597844

brennen moved this task from Next to In Progress on the User-brennen board.May 27 2020, 4:41 PM

Change 597826 merged by Jforrester:
[releng/dev-images@master] add jobrunner & tweak settings for TimedMediaHandler

https://gerrit.wikimedia.org/r/597826

Change 597844 merged by jenkins-bot:
[mediawiki/core@master] mediawiki-docker: Add a jobrunner container

https://gerrit.wikimedia.org/r/597844

Mentioned in SAL (#wikimedia-releng) [2020-06-17T19:46:02Z] <brennen> Updating dev-images docker-pkg files on primary contint for jobrunner changes (T246935)

brennen closed this task as Resolved.Fri, Jun 19, 2:27 AM
brennen moved this task from Needs/Waiting Review to Done / Defunct on the User-brennen board.
brennen moved this task from In Progress to Done on the MediaWiki-Docker board.