Page MenuHomePhabricator
Create Task
Maniphest T110528

[Task] Use JOIN to find changes relevant for a given wiki
Closed, InvalidPublic
Actions

Description

To find changes relevant to wiki x, we should be able to join wb_changes against wb_changes_subscriptions. This would remove the need for programmatic filtering.

Related Objects
Search...

StatusSubtypeAssignedTask
 InvalidNoneT108944 [Epic] Improve change dispatching
 InvalidNoneT110528 [Task] Use JOIN to find changes relevant for a given wiki
 ResolvedhooT109825 [Task] Use uppercase entity ID in wb_changes table.
 ResolvedNoneT111161 Subscribe client wikis immediately after adding sitelinks
 ResolvedaudeT112245 [Task] Configure wikidata.org to rely on the wb_changes_subscription table for dispatching.
 ResolvedNoneT110339 [Task] Deploy usage tracking on all client wikis on the Wikimedia cluster
 ResolvedaudeT107999 [Task] Enable Phase 2 for Wikibooks
 ResolvedNoneT98286 [Task] Deploy usage tracking and arbitrary access to zhwiki
 ResolvedLydia_PintscherT89002 multi-lingual label usage [Tracking]
 ResolveddanielT90563 Discuss: multi-lingual label usage tracking
 ResolveddanielT92288 Increase width of the eu_aspect column to allow inclusion of language codes.
 ResolveddanielT93055 Implement tracking of label use per parser target language
 ResolveddanielT93056 Include fallback languages in label tracking.
 ResolveddanielT93057 add eu_page_touched field to wbc_entity_usage_tracking
 ResolveddanielT93066 Investigate tracking update mechanism for "non-standard language" views.
 ResolveddanielT96373 Usage tracking update / purging on edit and re-parse (tracking)
 ResolvedNoneT96374 Track "touched" timestamp for each entry in the usage tracking table.
 InvaliddanielT99509 [Task] Ignore stale usage entries when processing changes
 ResolveddanielT99510 UsageTracking: Use LinksUpdateComplete hook instead ArticleEditUpdates
 ResolveddanielT99512 UsageTracking: Use new ParserCacheSaveComplete hook to track usage info.
 ResolveddanielT99513 UsageTracking: split the "update" operation for usage entries into "add" and "prune".
 ResolveddanielT99511 Introduce ParserCacheSave hook
 ResolveddanielT99693 UsageTracking: create integration test for usage tracking updates
 ResolveddanielT99774 Track usage in all namespaces, since arbitrary access allows data access from any page.
 ResolvedhooT101253 [RFC] How to track description usage
 DuplicateNoneT101280 Render labels in the user language on the client (via Lua or parser function), split parser cache
 ResolvedhooT75460 [Story] Make Lua functions default to the user's language on multilingual wikis
 ResolvedaudeT111142 Enable usage tracking on wikinews, wikibooks, and commons

Event Timeline

daniel created this task.Aug 27 2015, 1:52 PM
daniel raised the priority of this task from to Needs Triage.
daniel updated the task description. (Show Details)
daniel subscribed.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 27 2015, 1:52 PM
daniel added a project: Wikidata.Aug 27 2015, 1:52 PM
daniel set Security to None.
daniel added a subtask: T110339: [Task] Deploy usage tracking on all client wikis on the Wikimedia cluster.
daniel added a subtask: T109825: [Task] Use uppercase entity ID in wb_changes table..
daniel added a project: Performance Issue.
aude mentioned this in T109088: [Task] Profiling and investigate how to improve performance of change dispatcher.Aug 31 2015, 2:22 PM
Lydia_Pintscher renamed this task from Use JOIN to find changes relevant for a given wiki to [Task] Use JOIN to find changes relevant for a given wiki.Sep 1 2015, 11:39 AM
Lydia_Pintscher triaged this task as Medium priority.
Lydia_Pintscher moved this task from incoming to ready to go on the Wikidata board.
aude added a parent task: T108944: [Epic] Improve change dispatching.Sep 1 2015, 3:17 PM
JanZerebecki subscribed.Sep 1 2015, 6:03 PM
daniel mentioned this in T108944: [Epic] Improve change dispatching.Sep 3 2015, 2:24 PM
aude closed subtask T110339: [Task] Deploy usage tracking on all client wikis on the Wikimedia cluster as Resolved.Sep 9 2015, 10:49 AM
Ricordisamoa subscribed.Sep 9 2015, 11:05 AM
daniel mentioned this in T112245: [Task] Configure wikidata.org to rely on the wb_changes_subscription table for dispatching..Sep 11 2015, 10:06 AM
hoo subscribed.Oct 8 2015, 10:18 AM
daniel mentioned this in T118162: Wikibase dispatchChanges.php runs slow, creates an absurd amount of database connections.Nov 10 2015, 11:24 AM
aude closed subtask T112245: [Task] Configure wikidata.org to rely on the wb_changes_subscription table for dispatching. as Resolved.Nov 10 2015, 4:48 PM
Lydia_Pintscher subscribed.Apr 23 2017, 6:06 PM

Is this still relevant?

hoo added a comment.Apr 23 2017, 8:16 PM
In T110528#3204506, @Lydia_Pintscher wrote:

Is this still relevant?

Yes, we are actually working on this right now (see the dependency T109825: [Task] Use uppercase entity ID in wb_changes table.).

hoo closed subtask T109825: [Task] Use uppercase entity ID in wb_changes table. as Resolved.Apr 26 2017, 4:23 PM
hoo mentioned this in T171263: Wikidata Dispatcher and Job Queue is overflowed.Jul 21 2017, 10:13 AM
Restricted Application added a subscriber: PokestarFan. · View Herald TranscriptJul 25 2017, 8:15 AM
hoo added a subscriber: Ladsgroup.Jul 26 2017, 10:55 AM

As far as I can see T111161: Subscribe client wikis immediately after adding sitelinks is the only blocker left for this.

hoo raised the priority of this task from Medium to High.Jul 26 2017, 10:59 AM

Rising priority: Dispatching is an ongoing problem and this will fix (most of?) this.

Also I guess the number of changes that actually need dispatching has decreased (as we have a lot of entities not used on any sister project) these days. Thus fixing this and increasing the job batch size (or completely changing how that works) would fix the dispatching bottleneck for now.

daniel mentioned this in T111161: Subscribe client wikis immediately after adding sitelinks.Jul 26 2017, 11:25 AM
daniel added a project: Wikidata-Former-Sprint-Board.

proposing for sprint as per hoo

Halfak added a subtask: T173123: Implement nuanced usage tracking for Wikidata.Aug 11 2017, 5:53 PM
hoo removed a subtask: T173123: Implement nuanced usage tracking for Wikidata.Aug 11 2017, 10:16 PM
Lydia_Pintscher removed a project: Wikidata-Former-Sprint-Board.Nov 21 2017, 8:54 AM
hoo mentioned this in T193733: Move dispatching of wikidata to a dedicated node.May 3 2018, 5:50 PM
Michael closed this task as Invalid.Mar 7 2024, 6:35 PM
Michael subscribed.

I think with the refactoring of how change dispatching works in Q3 2021, this is no longer applicable. Now RecentChangeSaveHookHandler schedules one job per change as long as there is any wiki subscribed to that entity. That happens in the process when the change has been made.

Michael closed subtask T111161: Subscribe client wikis immediately after adding sitelinks as Resolved.Mar 28 2024, 1:56 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits