Page MenuHomePhabricator

Dashboard for internal MW job queue
Open, Needs TriagePublic

Description

Let's display information about the MediaWiki job queue in an extension, or create a new extension.

Background: An internal job queue tracks background tasks to be done such as edits to wiki pages that are not done directly in a browser. Examples: An import/upload of records; a search and replace by the ReplaceText extension; changes to other pages caused by an edit to a page that has Semantic/Cargo references to and from it.

The job queue can get so long that it takes weeks to clear. This is hard to diagnose unless the admin and user understand this internal architecture: By the default MediaWiki configuration, one background job is done for each front-end user operation. On some wikis which are basically databases, there are not many user clicks, and the job queue can stay full for a month.

In one case, it was found that upping $wgJobRunRate to 10 addressed the problem. In another case, a developer created an administrator-accessible button to execute runJobs.php.

So some extension should give some visibility into the number of jobs now in the queue, a bit of information about which process/task created some of them, and/or the rate at which they are being cleared, and a bit of advice if it is going slowly. The information offered by the special Web API URL like this one could be shown by the extension: http://aero.referata.com/w/api.php?action=query&meta=siteinfo&siprop=statistics
$wgJobRunRate could be explained there. runJobs.php can be explained. A button could be offered to speed up the job processing. I do not know what other info is in the job queue but if possible it should be shown by the extension

More background: https://www.mediawiki.org/wiki/Extension:Replace_Text#Known_issues

James Montalvo immediately understood the issue and made a little extension that would do some of this. Here is a copy of the github repo. I do not know enough to move it forward myself.
https://github.com/peterbenmeyer/JobQueueInspector

Event Timeline

James Montalvo immediately understood the issue and made a little extension that would do some of this. Here is a copy of the github repo. I do not know enough to move it forward myself.

@Econterms: Just for clarity and to avoid misunderstandings, do you plan to work on fixing this task at the Wikimania Hackathon?

@Aklapper: I can't work on it this Hackathon but someday perhaps. (I had some logistical challenges.) Also I don't know how to program well enough in the mediawiki context. In the longer run I hope to find an extension author who wouldn't mind to merge this content into an existing extension. It is not a lot to add, and some mediawiki managers would benefit.

Econterms updated the task description. (Show Details)

@Bryandamon: We discussed this at EMWCon 2019. It might not be difficult and it addresses one struggle in wikis using Cargo.

@Bryandamon: We discussed this at EMWCon 2019. It might not be difficult and it addresses one struggle in wikis using Cargo.

Yes, something like this would be very nice. In my case, I've both increased $wgJobRunRate (helps when rebuilding Cargo tables) and added the MagicNoCache extension magic word to a few templates. Cargo also sometimes needs a blank save to rebuild I think, would be nice to address that as well. Thanks for pushing this forward.

@Econterms: Hi! This task has been assigned to you a while ago. Could you maybe share an update? Do you still plan to work on this task?
If this task has been resolved in the meantime: Please update the task status (via Add Action...Change Status in the dropdown menu).
If this task is not resolved and only if you do not plan to work on this task anymore: Please consider removing yourself as assignee (via Add Action...Assign / Claim in the dropdown menu): That would allow others to work on this (in theory), as others won't think that someone is already working on this. Thanks! :)

Removing task assignee due to inactivity, as this open task has been assigned for more than two years. See the email sent to the task assignee on February 06th 2022 (and T295729).

Please assign this task to yourself again if you still realistically [plan to] work on this task - it would be welcome.

If this task has been resolved in the meantime, or should not be worked on ("declined"), please update its task status via "Add Action… 🡒 Change Status".

Also see https://www.mediawiki.org/wiki/Bug_management/Assignee_cleanup for tips how to best manage your individual work in Phabricator.

I hope to work on this at the upcoming hackathon, in the Washington DC space. (https://meta.wikimedia.org/wiki/DC_Hackathon_2022)