We recently added 500 000 pages to our MaRDI portal, which created 100 000 000 DispatchChanges jobs.
user@mardi:~/portal-compose$ docker compose exec -T wikibase /var/www/html/maintenance/run showJobs --group ... DispatchChanges: 102242200 queued; 0 claimed (0 active, 0 abandoned); 0 delayed
From T349178 I see that those jobs should be completed in 15 minutes on average. This is certainly not the case for us
o.O
What version of Wikibase are you running?
The purpose of the DispatchChanges job is to refresh wikitext articles that make use of the data from Entities when those Entities change.
Is that something that is applicable to your setup?
@Michael, we upgrade the version to the latest from time to time. Currently, we are at wmf/1.42.0-wmf.10 we also switch the job-runner to redis. We observed that the jobs were all completed, however, this was because our redis container did not have persistent storage and was restarted.
Yestday, we created another 250k pages and there are still about 20M jobs
mardi-test-user@mardi02:~/portal-compose$ docker compose exec -T wikibase /var/www/html/maintenance/run showJobs --group CreateProfilePages: 0 queued; 6906 claimed (6906 active, 0 abandoned); 0 delayed cirrusSearchIncomingLinkCount: 0 queued; 0 claimed (0 active, 0 abandoned); 92398 delayed cirrusSearchLinksUpdate: 394353 queued; 6891 claimed (6891 active, 0 abandoned); 0 delayed cirrusSearchElasticaWrite: 212 queued; 1584 claimed (1584 active, 0 abandoned); 0 delayed wikibase-addUsagesForPage: 0 queued; 7008 claimed (7008 active, 0 abandoned); 0 delayed EntityChangeNotification: 0 queued; 16 claimed (16 active, 0 abandoned); 0 delayed wikibase-InjectRCRecords: 0 queued; 22 claimed (22 active, 0 abandoned); 0 delayed DispatchChanges: 19076540 queued; 3045 claimed (3045 active, 0 abandoned); 0 delayed refreshLinks: 0 queued; 14 claimed (14 active, 0 abandoned); 0 delayed htmlCacheUpdate: 289466 queued; 3077 claimed (3077 active, 0 abandoned); 0 delayed enotifNotify: 0 queued; 3123 claimed (3123 active, 0 abandoned); 0 delayed recentChangesUpdate: 0 queued; 2054 claimed (2054 active, 0 abandoned); 0 delayed
We used a cronjob to speed up job execution and run 100 jobs in parallel every minute
`
cust_func(){
docker compose exec -T wikibase /var/www/html/maintenance/run runJobs.php --maxjobs=7
}
for i in {1..100}
do
cust_func $i &
doneThat works well for normal operations but for mass-creating pages, there is a quite huge job backlog. We are still planning to create additional 10M pages, thus it makes sense to still investing a better job running infrastructure. My feeling is that DispatchChanges is different from other jobs. It occurs to me that The job handling overhead (creating the job, scheduling, logging...) is much higher than the actual job execution.
The purpose of the DispatchChanges job is to refresh wikitext articles that make use of the data from Entities when those Entities change.
Is that something that is applicable to your setup?
Yes, the pages display content from wikibase, such as https://portal.mardi4nfdi.de/wiki/Person:1619976
After quite some digging on how to install the Redis job-runner I found the following configuration
https://github.com/MaRDI4NFDI/portal-compose/blob/main/redis-jobrunner-conf.json
works for us. With 100 dedicated DispatchChanges jobs vs 10 other jobs, the DispatchChanges jobs that piled up to 35M (there were 20M this morning) are processed faster than new jobs arrive.
I am leaving this open, as a reminder to reduce the overall number of jobs.
Good to hear that you found a way to deal with those jobs.
Still, the overall situation is very strange. Importing pages/items should not trigger multitudes of those jobs. That is, one edit to an Item should trigger at most 1 DispatchChanges job, not tens of it.
In the most recent version 1.43 wmf.28 the large number of jobs is gone. But our database is suffering from a large number of link update jobs, which has not been the case in the past. The job status now looks like that
docker compose exec -T wikibase /var/www/html/maintenance/run showJobs --group cirrusSearchIncomingLinkCount: 0 queued; 0 claimed (0 active, 0 abandoned); 2 delayed cirrusSearchElasticaWrite: 18 queued; 0 claimed (0 active, 0 abandoned); 0 delayed constraintsTableUpdate: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed refreshLinks: 96 queued; 4 claimed (3 active, 1 abandoned); 0 delayed htmlCacheUpdate: 6 queued; 0 claimed (0 active, 0 abandoned); 0 delayed recentChangesUpdate: 2 queued; 0 claimed (0 active, 0 abandoned); 0 delayed refreshLinksPrioritized: 0 queued; 1 claimed (0 active, 1 abandoned); 0 delayed refreshLinksDynamic: 16 queued; 5 claimed (5 active, 0 abandoned); 0 delayed
but the jobs never complete. Even if all jobs are completed new jobs are scheduled even if there were no edits. Thus the large list of links that need to be refreshed (for whatever reason) must still be somwhere but not in the jobqoue.
I'm closing this issue, but if anyone has a pointer that explains how the refresh link jobs get scheduled and rescheduled. I would greatly apprechiate that.