Many maintenance scripts that process a long series of independent pages or other items can benefit from parallelizing the work across multiple CPUs. This allows CPU-bound work to scale a lot faster, or service-bound work to wait on more things in parallel.
Proposed change:
* core change adding ParallelMaintenance and MediaWiki\Parallel\* helpers and porting several maint scripts: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/451099
* TimedMediaHandler change using ParallelMaintenance in a test script: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/TimedMediaHandler/+/232214
MediaWiki\Parallel\ForkStreamController is a refactoring of the existing OrderedStreamingForkController used by some CirrusSearch scripts. It takes input provided by an IDispatcher's `loop` method, passes the data as JSON serialization to child processes to run through an IConsumer's `process` method, and then collects the results over JSON again to the parent process the IDispatcher's `result` method.
ParallelMaintenance uses this interface to replace the Maintenance base class's default `execute` method with `loop`, `process`, and `result` methods to serve as the callbacks.
```lang=php
class Foo extends ParallelMaintenance {
// Handle any input on the parent thread, and
// pass any data as JSON-serializable form into
// the queue() method, where it gets funneled into
// a child process.
public function loop( \MediaWiki\Parallel\IController $controller ) {
for ( $i = 0; $i < 1000; $i++) {
$controller->queue( $i );
}
}
// On the child process, receives the queued value
// via JSON encode/decode. Here it's a number.
public function process( $count ) {
return str_repeat( '*', $count );
}
// On the parent thread, receives the work() return value
// via JSON encode/decode. Here it's a string.
public function result( $data ) {
$this->output( $data . "\n" );
}
}
```
The script gains a `--threads=N` option, and if a thread count is provided will automatically fork out separate processes, otherwise it'll process the work callback in-process.
Notes on connections and data availability:
* creating child processes with pcntl_fork is a Unix-only thing (eg Mac/Linux); this is not currently supported on Windows hosts, but they can run a single thread in-process.
* general MediaWiki setup state remains in memory in the child processes, but once they're forked each has an independent process
* when a child process is forked via MediaWiki\Parallel\ForkStreamController, it closes off connections, so DB connections will be reset. They should automatically reconnect on use.
* each child process is created once at the beginning, and will process 0 or more items during its lifetime
Some possible alternative implementations for parallel processing:
* using pthreads instead of pcntl_fork would be more compatible with Windows, but the pthreads extension for PHP doesn't seem to be well packaged and doesn't share global state, which would complicate threading setup.
* launching sub-processes through proc_open() and piping over stdin/out would also work on Windows, but again doesn't share global state, so would have to be able to launch a script that launches the right class.
* MediaWiki\Parallel\ExecStreamController provides the equivalent interface over proc_open, requiring the called script to manually launch a MediaWiki\Parallel\StreamWorker. Not yet exercised but can be added with a maintenance script as a 'router'.
* better tools for using the job queue, if you can rely on it for speed, could be useful; TimedMediaHandler's requeueTranscodes.php does manual throttling to keep from flooding the queue for instance.
Open questions:
* should this share more with the job queue infrastructure for job -> class routing and serialization?
* should this be expanded to be able to send jobs to the job queue?
* should IDispatcher and IConsumer be interfaces or just sets of callbacks? Either can always route to the other.
* bikeshed any naming/yak-shaving issues with the MediaWiki\Parallel\* classes and interfaces, or method naming? Any forseen method naming conflicts?