Page MenuHomePhabricator

[Task] Improve deduplication of AddUsagesForPage job
Closed, ResolvedPublic

Description

the AddUsagesForPage job can be de-duplicated better, based primarily on title + parser language code.

This would help avoid unnecessary and redundant updates to the wbc_entity_usage table. For example, in cases where an item or a client page is edited again when we still have older jobs in the queue, then the older jobs can become no-ops and we execute only the most recent job for a page (+parser language).

even if the usages didn't change, we still update the touched timestamp, and want to avoid database updates like this when unnecessary.

Event Timeline

aude raised the priority of this task from to Needs Triage.
aude updated the task description. (Show Details)
aude subscribed.

we might be able to do this instead by ensuring the jobs get 'deduplicated' using a root job, with a per title + langCode key.

the language code can be either:

if allowDataAccessInUserLanguage === true, then it is the user language (from parser options), else it is the content language.

this would allow the jobs to be deduplicated as much as possible for multlingual wikis and those that are not.

Change 232459 had a related patch set uploaded (by Aude):
Set root job for AddUsagesForPageJob

https://gerrit.wikimedia.org/r/232459

aude renamed this task from Check page touched timestamp before executing AddUsagesForPage job to Set a root job for AddUsagesForPage job, for improved de-duplication.Aug 19 2015, 9:26 AM
aude claimed this task.
aude set Security to None.
aude updated the task description. (Show Details)
aude renamed this task from Set a root job for AddUsagesForPage job, for improved de-duplication to Improve deduplication of AddUsagesForPage job,.Aug 19 2015, 10:57 AM
aude updated the task description. (Show Details)
aude renamed this task from Improve deduplication of AddUsagesForPage job, to Improve deduplication of AddUsagesForPage job.Aug 20 2015, 10:12 AM
Jonas renamed this task from Improve deduplication of AddUsagesForPage job to [Task] Improve deduplication of AddUsagesForPage job.Aug 20 2015, 1:14 PM
Jonas triaged this task as Medium priority.

Change 232459 merged by jenkins-bot:
Improve deduplication of AddUsagesForPageJob

https://gerrit.wikimedia.org/r/232459

aude moved this task from Doing to Done on the Wikidata-Sprint-2015-08-18 board.
aude removed a project: Patch-For-Review.