Change dispatcher should limit the amount of filtered changes in one go otherwise it might waste its work because it didn't manage to finish in 60s when its lock gets invalidated. Some Wikis have a very sparse hitrate, where there are 800000 pending changes but only 1 of it needs to be dispatched to a wiki.
Perhaps start with a limit of 100 * batch size. Maybe add its own command line setting.
Description
Description
| Status | Assigned | Task | ||
|---|---|---|---|---|
| Open | None | T108944 [Epic] Improve change dispatching | ||
| Resolved | aude | T105592 [Task] change dispatcher should limit the amount of filtered changes in one go |
Comment Actions
Change 230514 had a related patch set uploaded (by Hoo man):
Don't look at more than (15 * chunk size * batch size) changes in ChangeDispatcher
Comment Actions
this is indeed a problem. Dispatcher gets stuck in a loop when finding pending changes on small wikis that might not have a lot of relevant changes.
13:44:12 Using sitelinks to target notifications. 13:44:12 Using subscriptions to target notifications. 13:44:12 Starting loop for unlimited passes or 1600 seconds 13:44:12 Picking a client wiki... 13:44:12 Finding pending changes for liwikisource 13:44:12 Checking sitelinks to liwikisource for 446 entities. 13:44:12 Retaining changes for 0 relevant entities. 13:44:21 Found 0 changes for items with relevant sitelinks. 13:44:21 Checking sitelinks to liwikisource for 514 entities. 13:44:21 Retaining changes for 0 relevant entities. 13:44:33 Found 0 changes for items with relevant sitelinks. 13:44:33 Checking sitelinks to liwikisource for 535 entities. 13:44:33 Retaining changes for 0 relevant entities. 13:44:44 Found 0 changes for items with relevant sitelinks. 13:44:44 Checking sitelinks to liwikisource for 530 entities. 13:44:44 Retaining changes for 0 relevant entities. 13:44:55 Found 0 changes for items with relevant sitelinks. 13:44:55 Checking sitelinks to liwikisource for 504 entities. 13:44:55 Retaining changes for 0 relevant entities. 13:45:06 Found 0 changes for items with relevant sitelinks. 13:45:07 Checking sitelinks to liwikisource for 496 entities. 13:45:07 Retaining changes for 0 relevant entities. 13:45:21 Found 0 changes for items with relevant sitelinks. 13:45:21 Checking sitelinks to liwikisource for 422 entities. 13:45:21 Retaining changes for 0 relevant entities. 13:45:32 Found 0 changes for items with relevant sitelinks. 13:45:32 Checking sitelinks to liwikisource for 425 entities. 13:45:32 Retaining changes for 0 relevant entities. 13:45:41 Found 0 changes for items with relevant sitelinks. 13:45:41 Checking sitelinks to liwikisource for 408 entities. 13:45:41 Retaining changes for 0 relevant entities. 13:45:44 Found 0 changes for items with relevant sitelinks. 13:45:44 Checking sitelinks to liwikisource for 434 entities. 13:45:44 Retaining changes for 0 relevant entities. 13:45:48 Found 0 changes for items with relevant sitelinks. 13:45:48 Checking sitelinks to liwikisource for 449 entities. 13:45:48 Retaining changes for 0 relevant entities. 13:45:53 Found 0 changes for items with relevant sitelinks. 13:45:53 Checking sitelinks to liwikisource for 439 entities. 13:45:53 Retaining changes for 0 relevant entities. 13:45:57 Found 0 changes for items with relevant sitelinks. 13:45:57 Checking sitelinks to liwikisource for 446 entities. 13:45:57 Retaining changes for 0 relevant entities. 13:46:01 Found 0 changes for items with relevant sitelinks. 13:46:01 Checking sitelinks to liwikisource for 438 entities. 13:46:01 Retaining changes for 0 relevant entities. 13:46:05 Found 0 changes for items with relevant sitelinks. 13:46:05 Checking sitelinks to liwikisource for 452 entities. 13:46:05 Retaining changes for 0 relevant entities. 13:46:09 Found 0 changes for items with relevant sitelinks. 13:46:10 Checking sitelinks to liwikisource for 460 entities. 13:46:10 Retaining changes for 0 relevant entities. 13:46:14 Found 0 changes for items with relevant sitelinks. 13:46:15 Checking sitelinks to liwikisource for 460 entities. 13:46:15 Retaining changes for 0 relevant entities. 13:46:19 Found 0 changes for items with relevant sitelinks. 13:46:19 Checking sitelinks to liwikisource for 440 entities. 13:46:19 Retaining changes for 0 relevant entities.
Comment Actions
Change 233401 had a related patch set uploaded (by Aude):
Don't look at more than (15 * chunk size * batch size) changes in ChangeDispatcher
Comment Actions
Change 233401 merged by jenkins-bot:
Don't look at more than (15 * chunk size * batch size) changes in ChangeDispatcher
Comment Actions
Change 233411 had a related patch set uploaded (by Aude):
Update Wikidata - add limit for pending changes in dispatcher
Comment Actions
Change 233411 merged by jenkins-bot:
Update Wikidata - add limit for pending changes in dispatcher
Comment Actions
Change 233698 had a related patch set uploaded (by Aude):
Don't look at more than (15 * chunk size * batch size) changes in ChangeDispatcher
Comment Actions
Change 233698 merged by jenkins-bot:
Don't look at more than (15 * chunk size * batch size) changes in ChangeDispatcher
Comment Actions
Change 230514 merged by jenkins-bot:
Don't look at more than (15 * chunk size * batch size) changes in ChangeDispatcher