Page MenuHomePhabricator
Create Task
Maniphest T108944

[Epic] Improve change dispatching
Open, MediumPublic
Actions

Description

We want to have a reliable and secure dispatching mechanism without any bugs.

Priority list, after discussion with Katie, Jan, Adam, and Daniel on 2015-09-03:

  1. T110528: [Task] Use JOIN to find changes relevant for a given wiki
  2. T111356: [Story] wb_changes.change_info should only contain the minimum of information required for processing the change on the client.
  3. T111056: [Task] Replace ChangesTable orm abstraction with ChangeStore and ChangeLookup
  4. T107722: [Task] Make client ChangeHandler scale by batching updates
  5. T48643: [Story] Dispatching via job queue (instead of cron script) (needs thought, experimentation, and discussion)

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedjcrespoT118162 Wikibase dispatchChanges.php runs slow, creates an absurd amount of database connections
 ResolvedkarapayneWMDET265198 Don’t require extra cronjobs / maintenance scripts for Wikibase
 OpenNoneT108944 [Epic] Improve change dispatching
 ResolvedJanZerebeckiT108130 [Bug] "Warning: array_key_exists() expects parameter 2 to be array, null given" in the change dispatcher
 ResolvedNoneT108246 [Task] Enlarge wb_changes.change_info
 Resolved• AddshoreT75647 [Story] Log/graph dispatch lag
 DeclinedNoneT65411 [Story] wikis not removed from wb_changes_dispatch table when removed from localClientDatabases setting
 InvalidLydia_PintscherT50140 DispatchStats refactoring and API
 ResolvedaudeT105592 [Task] change dispatcher should limit the amount of filtered changes in one go
 ResolvedkarapayneWMDET48643 [Story] Dispatching via job queue (instead of cron script)
 ResolveddanielT70380 Factor dispatch logic out of maintenance script
 InvalidNoneT70381 [Task] [Dispatch via delayed job] Implement the trigger job
 InvalidNoneT70382 [Task] [Dispatch via delayed job] schedule trigger jobs on each change
 InvalidNoneT111353 [Task] understand MW jobQueue system and write RFC on how dispatching will use it
 ResolvedMichaelT290368 Create a dispatcher per-entity POC job
 ResolvedMichaelT290985 Enable new dispatching mechanism on beta
 OpenNoneT107722 [Task] Make client ChangeHandler scale by batching updates
 ResolvedhooT138409 [Task] Terms usage tracking: Only track parts of the language fallback chain that were actually used
 OpenNoneT146838 Allow seeking in UsageLookup::getPagesUsing results
 ResolvedaudeT109088 [Task] Profiling and investigate how to improve performance of change dispatcher
 InvalidNoneT111008 Deserialize only the site link diff when dispatching changes
 InvalidNoneT109821 [Task] Avoid unserializing metadata, especially diffs, when processing changes
 OpenNoneT110528 [Task] Use JOIN to find changes relevant for a given wiki
 ResolvedhooT109825 [Task] Use uppercase entity ID in wb_changes table.
 OpenNoneT111161 Subscribe client wikis immediately after adding sitelinks
 ResolvedaudeT112245 [Task] Configure wikidata.org to rely on the wb_changes_subscription table for dispatching.
 ResolvedNoneT110339 [Task] Deploy usage tracking on all client wikis on the Wikimedia cluster
 ResolvedaudeT107999 [Task] Enable Phase 2 for Wikibooks
 ResolvedNoneT98286 [Task] Deploy usage tracking and arbitrary access to zhwiki
 ResolvedLydia_PintscherT89002 multi-lingual label usage [Tracking]
 ResolveddanielT90563 Discuss: multi-lingual label usage tracking
 ResolveddanielT92288 Increase width of the eu_aspect column to allow inclusion of language codes.
 ResolveddanielT93055 Implement tracking of label use per parser target language
 ResolveddanielT93056 Include fallback languages in label tracking.
 ResolveddanielT93057 add eu_page_touched field to wbc_entity_usage_tracking
 ResolveddanielT93066 Investigate tracking update mechanism for "non-standard language" views.
 ResolveddanielT96373 Usage tracking update / purging on edit and re-parse (tracking)
 ResolvedNoneT96374 Track "touched" timestamp for each entry in the usage tracking table.
 InvaliddanielT99509 [Task] Ignore stale usage entries when processing changes
 ResolveddanielT99510 UsageTracking: Use LinksUpdateComplete hook instead ArticleEditUpdates
 ResolveddanielT99512 UsageTracking: Use new ParserCacheSaveComplete hook to track usage info.
 ResolveddanielT99513 UsageTracking: split the "update" operation for usage entries into "add" and "prune".
 ResolveddanielT99511 Introduce ParserCacheSave hook
 ResolveddanielT99693 UsageTracking: create integration test for usage tracking updates
 ResolveddanielT99774 Track usage in all namespaces, since arbitrary access allows data access from any page.
 ResolvedhooT101253 [RFC] How to track description usage
 DuplicateNoneT101280 Render labels in the user language on the client (via Lua or parser function), split parser cache
 ResolvedhooT75460 [Story] Make Lua functions default to the user's language on multilingual wikis
 ResolvedaudeT111142 Enable usage tracking on wikinews, wikibooks, and commons
 OpenNoneT111356 [Story] wb_changes.change_info should only contain the minimum of information required for processing the change on the client.
 ResolvedhooT110343 [Task] Rewrite EntityChange, drop dependency on ORMRow and ORMTable
 ResolvedhooT111056 [Task] Replace ChangesTable orm abstraction with ChangeStore and ChangeLookup
 ResolvedhooT126874 Don't save alias/ description diffs in wb_changes
 Resolved• AddshoreT151679 Document how change dispatching works
 ResolvedLadsgroupT151681 DispatchChanges: Avoid long-lasting connections to the master DB
ResolvedLadsgroupT151993 Implement ChangeDispatchCoordinator based on RedisLockManager
 ResolvedAndrewT155042 Increase quota for wikidata-dev project
 ResolvedLadsgroupT155190 Build an environment to test change dispatching using Redis-based locking
 DeclinedNoneT155196 Vagrant 1.9.1 provision failure on Trusty using role::labs:mediawiki_vagrant
 ResolvedLadsgroupT157308 Create test script for ChangeDispatchCoordinator
 ResolvedLadsgroupT159826 Use redis-based lock manager in dispatch changes in production
 ResolvedhooT159828 Use redis-based lock manager for dispatchChanges on test.wikidata.org
 ResolvedLadsgroupT162556 Consider only updating wb_changes_dispatch after a successful run
 ResolvedLadsgroupT162557 Use replica for reading the last dispatch position (chd_seen)
 OpenNoneT162558 Use memcached (or something similar) to keep the latest chd_seen state, only flush to table every once in a while
 ResolveddanielT113606 [Task] dispatchChanges should dispatch to local wiki if client functionality is enabled on the repo
 OpenNoneT171784 Merge LockManagerSqlChangeDispatchCoordinator into SqlChangeDispatchCoordinator, remove legacy locking
 DeclinedNoneT173008 Create a maintenance script for pruning stale entity subscriptions and run periodically
 ResolvedhooT179060 Dispatchers occasionally seem to "freeze" for certain wikis
 Resolved• AddshoreT178624 enwiki dispatch lag is 8 hours
 Resolved• AddshoreT178652 Wikidata dispatchers should use a LockManager with a short TTL
 OpenNoneT179008 Wikidata change propagation: introduce --batch-grace parameter to replace --dispatch-interval
 DeclinedNoneT179006 Wikidata change propagation: set reduce dispatcher randomness and shorten dispatch interval

Event Timeline

• Jonas created this task.Aug 13 2015, 2:02 PM
• Jonas raised the priority of this task from to Medium.
• Jonas updated the task description. (Show Details)
• Jonas added subscribers: • Jonas, Lydia_Pintscher.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 13 2015, 2:02 PM
• Jonas added a subtask: T108130: [Bug] "Warning: array_key_exists() expects parameter 2 to be array, null given" in the change dispatcher.Aug 13 2015, 2:03 PM
• Jonas added projects: Epic, Wikidata.
• Jonas set Security to None.
Reedy removed a project: Epic.Aug 13 2015, 2:03 PM
Lydia_Pintscher added a subtask: T75647: [Story] Log/graph dispatch lag.Aug 13 2015, 2:03 PM
Reedy added a project: Epic.Aug 13 2015, 2:05 PM
• Jonas added a subtask: T65411: [Story] wikis not removed from wb_changes_dispatch table when removed from localClientDatabases setting.Aug 13 2015, 2:06 PM
• Jonas added subtasks: T50140: DispatchStats refactoring and API, T105592: [Task] change dispatcher should limit the amount of filtered changes in one go, T48643: [Story] Dispatching via job queue (instead of cron script), T70382: [Task] [Dispatch via delayed job] schedule trigger jobs on each change, T70381: [Task] [Dispatch via delayed job] Implement the trigger job.
Lydia_Pintscher closed subtask T50140: DispatchStats refactoring and API as Invalid.Aug 13 2015, 2:07 PM
hoo added a subscriber: hoo.Aug 13 2015, 2:09 PM
Lydia_Pintscher added a subtask: T107722: [Task] Make client ChangeHandler scale by batching updates.Aug 13 2015, 2:21 PM
Lydia_Pintscher added a subtask: T109088: [Task] Profiling and investigate how to improve performance of change dispatcher.Aug 14 2015, 5:37 PM
Lydia_Pintscher moved this task from incoming to ready to go on the Wikidata board.Aug 17 2015, 3:23 PM
JanZerebecki closed subtask T108130: [Bug] "Warning: array_key_exists() expects parameter 2 to be array, null given" in the change dispatcher as Resolved.Aug 21 2015, 3:46 PM
aude closed subtask T109088: [Task] Profiling and investigate how to improve performance of change dispatcher as Resolved.Aug 31 2015, 2:22 PM
Lydia_Pintscher added a subtask: T111008: Deserialize only the site link diff when dispatching changes.Sep 1 2015, 11:36 AM
Lydia_Pintscher added a subtask: T109821: [Task] Avoid unserializing metadata, especially diffs, when processing changes.Sep 1 2015, 11:49 AM
aude added a subtask: T111056: [Task] Replace ChangesTable orm abstraction with ChangeStore and ChangeLookup.Sep 1 2015, 3:06 PM
aude added a subtask: T110528: [Task] Use JOIN to find changes relevant for a given wiki.Sep 1 2015, 3:17 PM
daniel closed subtask T105592: [Task] change dispatcher should limit the amount of filtered changes in one go as Resolved.Sep 2 2015, 3:14 PM
daniel closed subtask T111008: Deserialize only the site link diff when dispatching changes as Invalid.Sep 3 2015, 1:48 PM
daniel removed subtasks: T111353: [Task] understand MW jobQueue system and write RFC on how dispatching will use it, T70381: [Task] [Dispatch via delayed job] Implement the trigger job, T70382: [Task] [Dispatch via delayed job] schedule trigger jobs on each change.Sep 3 2015, 2:18 PM
daniel updated the task description. (Show Details)Sep 3 2015, 2:24 PM
JanZerebecki added a subtask: T111353: [Task] understand MW jobQueue system and write RFC on how dispatching will use it.Sep 10 2015, 6:49 PM
aude added a subtask: T110343: [Task] Rewrite EntityChange, drop dependency on ORMRow and ORMTable.Sep 10 2015, 10:33 PM
JanZerebecki closed subtask T111056: [Task] Replace ChangesTable orm abstraction with ChangeStore and ChangeLookup as Resolved.Oct 13 2015, 1:22 PM
JanZerebecki closed subtask T110343: [Task] Rewrite EntityChange, drop dependency on ORMRow and ORMTable as Resolved.
hoo reopened subtask T110343: [Task] Rewrite EntityChange, drop dependency on ORMRow and ORMTable as Open.Oct 13 2015, 3:22 PM
hoo closed subtask T110343: [Task] Rewrite EntityChange, drop dependency on ORMRow and ORMTable as Resolved.Oct 13 2015, 5:54 PM
• Addshore mentioned this in T118162: Wikibase dispatchChanges.php runs slow, creates an absurd amount of database connections.Nov 9 2015, 12:42 PM
hoo added a subtask: T126874: Don't save alias/ description diffs in wb_changes.Feb 13 2016, 5:02 PM
hoo closed subtask T126874: Don't save alias/ description diffs in wb_changes as Resolved.Feb 16 2016, 12:12 PM
daniel created subtask T151679: Document how change dispatching works.Nov 26 2016, 2:42 PM
daniel created subtask T151681: DispatchChanges: Avoid long-lasting connections to the master DB .Nov 26 2016, 3:07 PM
hoo closed subtask T75647: [Story] Log/graph dispatch lag as Resolved.Apr 9 2017, 6:05 PM
matej_suchanek added a subtask: T113606: [Task] dispatchChanges should dispatch to local wiki if client functionality is enabled on the repo.Apr 23 2017, 5:23 PM
Sjoerddebruin added a subscriber: Sjoerddebruin.Jul 19 2017, 4:52 PM
Restricted Application added a subscriber: PokestarFan. · View Herald TranscriptJul 25 2017, 8:16 AM
hoo added a subtask: T171784: Merge LockManagerSqlChangeDispatchCoordinator into SqlChangeDispatchCoordinator, remove legacy locking.Jul 26 2017, 8:45 PM
daniel closed subtask T113606: [Task] dispatchChanges should dispatch to local wiki if client functionality is enabled on the repo as Resolved.Jul 29 2017, 12:28 PM
hoo added a subtask: T173008: Create a maintenance script for pruning stale entity subscriptions and run periodically.Aug 10 2017, 3:22 PM
Halfak added a subtask: T173119: Implement more nuanced patrolling relevant changes in Wikidata based on usage.Aug 11 2017, 5:43 PM
hoo removed a subtask: T173119: Implement more nuanced patrolling relevant changes in Wikidata based on usage.Aug 11 2017, 10:27 PM
hoo added a subtask: T179060: Dispatchers occasionally seem to "freeze" for certain wikis.Oct 26 2017, 9:51 AM
hoo closed subtask T179060: Dispatchers occasionally seem to "freeze" for certain wikis as Resolved.Oct 30 2017, 10:40 AM
Liuxinyu970226 added a subscriber: Liuxinyu970226.Oct 30 2017, 12:13 PM
• Addshore added a subscriber: • Addshore.Dec 5 2017, 9:15 PM
thiemowmde closed subtask T111353: [Task] understand MW jobQueue system and write RFC on how dispatching will use it as Invalid.Dec 7 2017, 2:52 PM
• Addshore closed subtask T151679: Document how change dispatching works as Resolved.Dec 11 2017, 4:07 PM
hoo added subtasks: T179008: Wikidata change propagation: introduce --batch-grace parameter to replace --dispatch-interval, T179006: Wikidata change propagation: set reduce dispatcher randomness and shorten dispatch interval.Jan 21 2018, 10:41 AM
jcrespo mentioned this in T151681: DispatchChanges: Avoid long-lasting connections to the master DB .Jun 15 2018, 3:49 PM
• Marostegui closed subtask T151681: DispatchChanges: Avoid long-lasting connections to the master DB as Resolved.Jun 16 2018, 5:36 AM
• Addshore changed the status of subtask T179006: Wikidata change propagation: set reduce dispatcher randomness and shorten dispatch interval from Open to Stalled.Jun 25 2018, 9:55 AM
• Aleksey_WMDE added a subscriber: • Aleksey_WMDE.Aug 29 2018, 7:23 PM
• Addshore closed subtask T179006: Wikidata change propagation: set reduce dispatcher randomness and shorten dispatch interval as Declined.Sep 19 2018, 7:25 AM
• Addshore closed subtask T109821: [Task] Avoid unserializing metadata, especially diffs, when processing changes as Invalid.Jan 23 2019, 1:05 PM
hoo closed subtask T173008: Create a maintenance script for pruning stale entity subscriptions and run periodically as Declined.Apr 18 2019, 2:56 PM
Michael added a subscriber: Michael.Sep 30 2021, 4:41 PM
karapayneWMDE closed subtask T48643: [Story] Dispatching via job queue (instead of cron script) as Resolved.Oct 27 2021, 2:13 PM
Bugreporter closed subtask T65411: [Story] wikis not removed from wb_changes_dispatch table when removed from localClientDatabases setting as Declined.Feb 13 2022, 2:15 AM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL