Page MenuHomePhabricator

[Task] change dispatcher should limit the amount of filtered changes in one go
Closed, ResolvedPublic

Description

Change dispatcher should limit the amount of filtered changes in one go otherwise it might waste its work because it didn't manage to finish in 60s when its lock gets invalidated. Some Wikis have a very sparse hitrate, where there are 800000 pending changes but only 1 of it needs to be dispatched to a wiki.
Perhaps start with a limit of 100 * batch size. Maybe add its own command line setting.

Event Timeline

JanZerebecki raised the priority of this task from to High.
JanZerebecki updated the task description. (Show Details)
JanZerebecki added a project: Wikidata.
JanZerebecki added subscribers: JanZerebecki, hoo, aude.

Change 230514 had a related patch set uploaded (by Hoo man):
Don't look at more than (15 * chunk size * batch size) changes in ChangeDispatcher

https://gerrit.wikimedia.org/r/230514

Jonas renamed this task from change dispatcher should limit the amount of filtered changes in one go to [Task] change dispatcher should limit the amount of filtered changes in one go.Aug 13 2015, 2:41 PM

this is indeed a problem. Dispatcher gets stuck in a loop when finding pending changes on small wikis that might not have a lot of relevant changes.

13:44:12 Using sitelinks to target notifications.
13:44:12 Using subscriptions to target notifications.
13:44:12 Starting loop for unlimited passes or 1600 seconds
13:44:12     Picking a client wiki...
13:44:12     Finding pending changes for liwikisource
13:44:12     Checking sitelinks to liwikisource for 446 entities.
13:44:12     Retaining changes for 0 relevant entities.
13:44:21     Found 0 changes for items with relevant sitelinks.
13:44:21     Checking sitelinks to liwikisource for 514 entities.
13:44:21     Retaining changes for 0 relevant entities.
13:44:33     Found 0 changes for items with relevant sitelinks.
13:44:33     Checking sitelinks to liwikisource for 535 entities.
13:44:33     Retaining changes for 0 relevant entities.
13:44:44     Found 0 changes for items with relevant sitelinks.
13:44:44     Checking sitelinks to liwikisource for 530 entities.
13:44:44     Retaining changes for 0 relevant entities.
13:44:55     Found 0 changes for items with relevant sitelinks.
13:44:55     Checking sitelinks to liwikisource for 504 entities.
13:44:55     Retaining changes for 0 relevant entities.
13:45:06     Found 0 changes for items with relevant sitelinks.
13:45:07     Checking sitelinks to liwikisource for 496 entities.
13:45:07     Retaining changes for 0 relevant entities.
13:45:21     Found 0 changes for items with relevant sitelinks.
13:45:21     Checking sitelinks to liwikisource for 422 entities.
13:45:21     Retaining changes for 0 relevant entities.
13:45:32     Found 0 changes for items with relevant sitelinks.
13:45:32     Checking sitelinks to liwikisource for 425 entities.
13:45:32     Retaining changes for 0 relevant entities.
13:45:41     Found 0 changes for items with relevant sitelinks.
13:45:41     Checking sitelinks to liwikisource for 408 entities.
13:45:41     Retaining changes for 0 relevant entities.
13:45:44     Found 0 changes for items with relevant sitelinks.
13:45:44     Checking sitelinks to liwikisource for 434 entities.
13:45:44     Retaining changes for 0 relevant entities.
13:45:48     Found 0 changes for items with relevant sitelinks.
13:45:48     Checking sitelinks to liwikisource for 449 entities.
13:45:48     Retaining changes for 0 relevant entities.
13:45:53     Found 0 changes for items with relevant sitelinks.
13:45:53     Checking sitelinks to liwikisource for 439 entities.
13:45:53     Retaining changes for 0 relevant entities.
13:45:57     Found 0 changes for items with relevant sitelinks.
13:45:57     Checking sitelinks to liwikisource for 446 entities.
13:45:57     Retaining changes for 0 relevant entities.
13:46:01     Found 0 changes for items with relevant sitelinks.
13:46:01     Checking sitelinks to liwikisource for 438 entities.
13:46:01     Retaining changes for 0 relevant entities.
13:46:05     Found 0 changes for items with relevant sitelinks.
13:46:05     Checking sitelinks to liwikisource for 452 entities.
13:46:05     Retaining changes for 0 relevant entities.
13:46:09     Found 0 changes for items with relevant sitelinks.
13:46:10     Checking sitelinks to liwikisource for 460 entities.
13:46:10     Retaining changes for 0 relevant entities.
13:46:14     Found 0 changes for items with relevant sitelinks.
13:46:15     Checking sitelinks to liwikisource for 460 entities.
13:46:15     Retaining changes for 0 relevant entities.
13:46:19     Found 0 changes for items with relevant sitelinks.
13:46:19     Checking sitelinks to liwikisource for 440 entities.
13:46:19     Retaining changes for 0 relevant entities.

Change 233401 had a related patch set uploaded (by Aude):
Don't look at more than (15 * chunk size * batch size) changes in ChangeDispatcher

https://gerrit.wikimedia.org/r/233401

Change 233401 merged by jenkins-bot:
Don't look at more than (15 * chunk size * batch size) changes in ChangeDispatcher

https://gerrit.wikimedia.org/r/233401

Change 233411 had a related patch set uploaded (by Aude):
Update Wikidata - add limit for pending changes in dispatcher

https://gerrit.wikimedia.org/r/233411

Change 233411 merged by jenkins-bot:
Update Wikidata - add limit for pending changes in dispatcher

https://gerrit.wikimedia.org/r/233411

Change 233698 had a related patch set uploaded (by Aude):
Don't look at more than (15 * chunk size * batch size) changes in ChangeDispatcher

https://gerrit.wikimedia.org/r/233698

Change 233698 merged by jenkins-bot:
Don't look at more than (15 * chunk size * batch size) changes in ChangeDispatcher

https://gerrit.wikimedia.org/r/233698

Change 230514 merged by jenkins-bot:
Don't look at more than (15 * chunk size * batch size) changes in ChangeDispatcher

https://gerrit.wikimedia.org/r/230514

All patches merged. It was said that this still needs tests.

merged the tests now, too

Change 235468 merged by jenkins-bot:
Add test for maxChunks in ChangeDispatcher

https://gerrit.wikimedia.org/r/235468