gunter.schmidt wrote:

I am sorry, I just found, that the cleanup-jobs seems to check this. So it will delete all similar jobs with just one step.

Thus the DB-load is not significantly higher than if it would have been not inserted in the first place.

The only drawback is, that you do not see the actual length of the jobqueue. One could make a group statement in
SpecialStatistics.php.

gunter.schmidt wrote:

One would need to add:

$numJobs = $dbr->selectField( 'job', 'COUNT(*)', '', $fname );

+ $numJobsGrouped = $dbr->selectField( 'job', 'count(DISTINCT job_title,job_cmd,job_namespace)', '', $fname);

$wgLang->formatNum( $images ),

+ $wgLang->formatNum( $numJobsGrouped )

in SpecialStatistics.php

and add some text to the Message: Sitestatstext

I am not sure about the database load of count DISTINCT on large Systems, so it might not be a good idea.

Another possible SELECT would be SELECT COUNT(*) AS C FROM job WHERE job_id IN (SELECT (job_id) FROM job GROUP BY
job_cmd,job_namespace, job_title).

robchur wrote:

We could just add a unique index on those three columns and use an INSERT IGNORE
when stuffing rows into the job queue, but I'd like another opinion on whether
or not the duplicates are, in fact, causing load that we need to be worried about.

The duplicates are used because the original checking on add was very expensive
(the inserts must be very fast, while the processing can take as long as it needs).

An INSERT IGNORE might not do too bad, though, dunno.

I didn't use a unique index in the original code because I imagined that at some stage in the future, we may want to add job types that require execution of duplicates. For example, a job type with no attached title, defined entirely by the last few bytes of a large job_params blob, would create duplicates in a (job_cmd,job_namespace,job_title) key. The current method is good enough for now, although I would like to switch to a specialised non-MySQL data structure at some stage.

• Tim Starling