Page MenuHomePhabricator

Dashboard for internal MW job queue
Open, Needs TriagePublic

Description

Let's display information about the MediaWiki job queue in an extension, or create a new extension.

Background: An internal job queue tracks background tasks to be done such as edits to wiki pages that are not done directly in a browser. Examples: An import/upload of records; a search and replace by the ReplaceText extension; changes to other pages caused by an edit to a page that has Semantic/Cargo references to and from it.

The job queue can get so long that it takes weeks to clear. This is hard to diagnose unless the admin and user understand this internal architecture: By the default MediaWiki configuration, one background job is done for each front-end user operation. On some wikis which are basically databases, there are not many user clicks, and the job queue can stay full for a month.

In one case, it was found that upping $wgJobRunRate to 10 addressed the problem. In another case, a developer created an administrator-accessible button to execute runJobs.php.

So some extension should give some visibility into the number of jobs now in the queue, a bit of information about which process/task created some of them, and/or the rate at which they are being cleared, and a bit of advice if it is going slowly. The information offered by the special Web API URL like this one could be shown by the extension: http://aero.referata.com/w/api.php?action=query&meta=siteinfo&siprop=statistics
$wgJobRunRate could be explained there. runJobs.php can be explained. A button could be offered to speed up the job processing. I do not know what other info is in the job queue but if possible it should be shown by the extension

More background: https://www.mediawiki.org/wiki/Extension:Replace_Text#Known_issues

James Montalvo immediately understood the issue and made a little extension that would do some of this. Here is a copy of the github repo. I do not know enough to move it forward myself.
https://github.com/peterbenmeyer/JobQueueInspector

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMon, Aug 12, 5:29 AM

James Montalvo immediately understood the issue and made a little extension that would do some of this. Here is a copy of the github repo. I do not know enough to move it forward myself.

@Econterms: Just for clarity and to avoid misunderstandings, do you plan to work on fixing this task at the Wikimania Hackathon?

@Aklapper: I can't work on it this Hackathon but someday perhaps. (I had some logistical challenges.) Also I don't know how to program well enough in the mediawiki context. In the longer run I hope to find an extension author who wouldn't mind to merge this content into an existing extension. It is not a lot to add, and some mediawiki managers would benefit.

Econterms updated the task description. (Show Details)Thu, Aug 15, 6:16 AM
Econterms added a subscriber: Bryandamon.
Econterms updated the task description. (Show Details)Thu, Aug 15, 6:21 AM
Econterms updated the task description. (Show Details)

@Bryandamon: We discussed this at EMWCon 2019. It might not be difficult and it addresses one struggle in wikis using Cargo.

@Bryandamon: We discussed this at EMWCon 2019. It might not be difficult and it addresses one struggle in wikis using Cargo.

Yes, something like this would be very nice. In my case, I've both increased $wgJobRunRate (helps when rebuilding Cargo tables) and added the MagicNoCache extension magic word to a few templates. Cargo also sometimes needs a blank save to rebuild I think, would be nice to address that as well. Thanks for pushing this forward.