Page MenuHomePhabricator

Rewrite jobs-loop.sh in a proper programming language
Closed, ResolvedPublic

Description

In terms of complexity, jobs-loop.sh has become complex enough that its shell script implementation is awkward. Porting it to a proper language like PHP would increase the number of people who could meaningfully contribute to it.

The following features would be desirable:

  • The ability to wrap runJobs.php in a per-section PoolCounter pool. It has recently become apparent (Ie5bb11b0) that if the CPU power available to the job runners is going to be efficiently utilised, then the number of job runners running on any given DB master needs to be limited.
  • A configuration file. Currently Puppet configures jobs-loop.sh in two different ways simultaneously: by changing the command line parameters via JR_EXTRA_ARGS in mw-job-runner.default, and by altering the code itself by making the shell script be a template. I don't think either is an especially elegant configuration method.
  • The ability to check for the exit status from runJobs.php, somehow avoiding a tight loop of respawns.
  • An error log might be nice too.

Version: unspecified
Severity: normal
Whiteboard: perfsprint-13

Details

Reference
bz46770

Event Timeline

bzimport raised the priority of this task from to Medium.Nov 22 2014, 1:34 AM
bzimport set Reference to bz46770.
tstarling created this task.Apr 2 2013, 4:42 AM

Based on previous comments by Ryan Lane that we should prefer PHP or Python for production scripts, tools, etc. (and I agree with him for a number of reasons), I'm tweaking the bug summary from "Rewrite jobs-loop.sh in a proper programming language" to "Rewrite jobs-loop.sh in PHP or Python".

It could also cooperate with the init script a bit more, say, by forking and writing a PID file.

(In reply to comment #1)

Based on previous comments by Ryan Lane that we should prefer PHP or Python
for
production scripts, tools, etc. (and I agree with him for a number of
reasons),
I'm tweaking the bug summary from "Rewrite jobs-loop.sh in a proper
programming
language" to "Rewrite jobs-loop.sh in PHP or Python".

The best tool for the job should be used.

aaron added a comment.Apr 4 2013, 7:27 PM

Any rewrite should also try to absorb nextJobDB.php to avoid the overhead of shelling out to it all the time.

If we ever go for python, count me in.

I have filled Bug 68318 - Bugzilla component for mediawiki/services/jobrunner with appropriate default CC

Note the repository is missing tests :/