[Task] change dispatcher should limit the amount of filtered changes in one go
Closed, ResolvedPublic

Description

Change dispatcher should limit the amount of filtered changes in one go otherwise it might waste its work because it didn't manage to finish in 60s when its lock gets invalidated. Some Wikis have a very sparse hitrate, where there are 800000 pending changes but only 1 of it needs to be dispatched to a wiki.
Perhaps start with a limit of 100 * batch size. Maybe add its own command line setting.

JanZerebecki updated the task description. (Show Details)
JanZerebecki raised the priority of this task from to High.
JanZerebecki added a project: Wikidata.
JanZerebecki added subscribers: JanZerebecki, hoo, aude.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 11 2015, 2:28 PM
JanZerebecki updated the task description. (Show Details)Jul 11 2015, 2:41 PM
JanZerebecki set Security to None.

Change 230514 had a related patch set uploaded (by Hoo man):
Don't look at more than (15 * chunk size * batch size) changes in ChangeDispatcher

https://gerrit.wikimedia.org/r/230514

Jonas renamed this task from change dispatcher should limit the amount of filtered changes in one go to [Task] change dispatcher should limit the amount of filtered changes in one go.Aug 13 2015, 2:41 PM
aude added a comment.Aug 24 2015, 1:47 PM

this is indeed a problem. Dispatcher gets stuck in a loop when finding pending changes on small wikis that might not have a lot of relevant changes.

13:44:12 Using sitelinks to target notifications.
13:44:12 Using subscriptions to target notifications.
13:44:12 Starting loop for unlimited passes or 1600 seconds
13:44:12     Picking a client wiki...
13:44:12     Finding pending changes for liwikisource
13:44:12     Checking sitelinks to liwikisource for 446 entities.
13:44:12     Retaining changes for 0 relevant entities.
13:44:21     Found 0 changes for items with relevant sitelinks.
13:44:21     Checking sitelinks to liwikisource for 514 entities.
13:44:21     Retaining changes for 0 relevant entities.
13:44:33     Found 0 changes for items with relevant sitelinks.
13:44:33     Checking sitelinks to liwikisource for 535 entities.
13:44:33     Retaining changes for 0 relevant entities.
13:44:44     Found 0 changes for items with relevant sitelinks.
13:44:44     Checking sitelinks to liwikisource for 530 entities.
13:44:44     Retaining changes for 0 relevant entities.
13:44:55     Found 0 changes for items with relevant sitelinks.
13:44:55     Checking sitelinks to liwikisource for 504 entities.
13:44:55     Retaining changes for 0 relevant entities.
13:45:06     Found 0 changes for items with relevant sitelinks.
13:45:07     Checking sitelinks to liwikisource for 496 entities.
13:45:07     Retaining changes for 0 relevant entities.
13:45:21     Found 0 changes for items with relevant sitelinks.
13:45:21     Checking sitelinks to liwikisource for 422 entities.
13:45:21     Retaining changes for 0 relevant entities.
13:45:32     Found 0 changes for items with relevant sitelinks.
13:45:32     Checking sitelinks to liwikisource for 425 entities.
13:45:32     Retaining changes for 0 relevant entities.
13:45:41     Found 0 changes for items with relevant sitelinks.
13:45:41     Checking sitelinks to liwikisource for 408 entities.
13:45:41     Retaining changes for 0 relevant entities.
13:45:44     Found 0 changes for items with relevant sitelinks.
13:45:44     Checking sitelinks to liwikisource for 434 entities.
13:45:44     Retaining changes for 0 relevant entities.
13:45:48     Found 0 changes for items with relevant sitelinks.
13:45:48     Checking sitelinks to liwikisource for 449 entities.
13:45:48     Retaining changes for 0 relevant entities.
13:45:53     Found 0 changes for items with relevant sitelinks.
13:45:53     Checking sitelinks to liwikisource for 439 entities.
13:45:53     Retaining changes for 0 relevant entities.
13:45:57     Found 0 changes for items with relevant sitelinks.
13:45:57     Checking sitelinks to liwikisource for 446 entities.
13:45:57     Retaining changes for 0 relevant entities.
13:46:01     Found 0 changes for items with relevant sitelinks.
13:46:01     Checking sitelinks to liwikisource for 438 entities.
13:46:01     Retaining changes for 0 relevant entities.
13:46:05     Found 0 changes for items with relevant sitelinks.
13:46:05     Checking sitelinks to liwikisource for 452 entities.
13:46:05     Retaining changes for 0 relevant entities.
13:46:09     Found 0 changes for items with relevant sitelinks.
13:46:10     Checking sitelinks to liwikisource for 460 entities.
13:46:10     Retaining changes for 0 relevant entities.
13:46:14     Found 0 changes for items with relevant sitelinks.
13:46:15     Checking sitelinks to liwikisource for 460 entities.
13:46:15     Retaining changes for 0 relevant entities.
13:46:19     Found 0 changes for items with relevant sitelinks.
13:46:19     Checking sitelinks to liwikisource for 440 entities.
13:46:19     Retaining changes for 0 relevant entities.

Change 233401 had a related patch set uploaded (by Aude):
Don't look at more than (15 * chunk size * batch size) changes in ChangeDispatcher

https://gerrit.wikimedia.org/r/233401

Change 233401 merged by jenkins-bot:
Don't look at more than (15 * chunk size * batch size) changes in ChangeDispatcher

https://gerrit.wikimedia.org/r/233401

Change 233411 had a related patch set uploaded (by Aude):
Update Wikidata - add limit for pending changes in dispatcher

https://gerrit.wikimedia.org/r/233411

Change 233411 merged by jenkins-bot:
Update Wikidata - add limit for pending changes in dispatcher

https://gerrit.wikimedia.org/r/233411

JanZerebecki reassigned this task from hoo to aude.Aug 25 2015, 9:55 AM

Change 233698 had a related patch set uploaded (by Aude):
Don't look at more than (15 * chunk size * batch size) changes in ChangeDispatcher

https://gerrit.wikimedia.org/r/233698

Change 233698 merged by jenkins-bot:
Don't look at more than (15 * chunk size * batch size) changes in ChangeDispatcher

https://gerrit.wikimedia.org/r/233698

Change 230514 merged by jenkins-bot:
Don't look at more than (15 * chunk size * batch size) changes in ChangeDispatcher

https://gerrit.wikimedia.org/r/230514

All patches merged. It was said that this still needs tests.

aude moved this task from Doing to Done on the Wikidata-Sprint-2015-08-18 board.Aug 28 2015, 10:53 AM
aude moved this task from Done to Doing on the Wikidata-Sprint-2015-08-18 board.
aude moved this task from Doing to Review on the Wikidata-Sprint-2015-09-01 board.Sep 2 2015, 2:55 PM
daniel closed this task as Resolved.Sep 2 2015, 3:14 PM

merged the tests now, too

daniel moved this task from Review to Done on the Wikidata-Sprint-2015-09-01 board.Sep 2 2015, 3:14 PM

Change 235468 merged by jenkins-bot:
Add test for maxChunks in ChangeDispatcher

https://gerrit.wikimedia.org/r/235468