Page MenuHomePhabricator
Create Task
Maniphest T201217

Load daily diffs to WDQS instead of weekly dumps
Closed, ResolvedPublic
Actions

Description

Convert category updates from weekly dumps to daily diffs at https://dumps.wikimedia.org/other/categoriesrdf/daily/

Details

SubjectRepoBranchLines +/-
Enable dailies everywhereoperations/puppetproduction+1 -1
Enable daily category diffs on testoperations/puppetproduction+1 -1
Add ability to load daily category dumps.operations/puppetproduction+44 -17
Use separate insert/delete statements to work around Blazegraph bugmediawiki/coremaster+43 -18
Add script for loading daily dumpwikidata/query/rdfmaster+39 -0
Fix target file for diff between weekly and dailyoperations/puppetproduction+3 -2
Customize query in gerrit

Event Timeline

Smalyshev created this task.Aug 4 2018, 12:54 AM
Restricted Application added a project: Wikidata. · View Herald TranscriptAug 4 2018, 12:54 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Smalyshev added a project: User-Smalyshev.Aug 4 2018, 12:55 AM

Currently does not work because of https://github.com/blazegraph/database/issues/100, workaround or fix required.

Smalyshev triaged this task as Medium priority.Aug 4 2018, 2:59 AM
Smalyshev moved this task from Backlog to Doing on the User-Smalyshev board.Aug 6 2018, 12:05 AM
gerritbot subscribed.Aug 6 2018, 12:15 AM

Change 450483 had a related patch set uploaded (by Smalyshev; owner: Smalyshev):
[operations/puppet@production] Fix target file for diff between weekly and daily

https://gerrit.wikimedia.org/r/450483

gerritbot added a project: Patch-For-Review.Aug 6 2018, 12:15 AM
gerritbot added a comment.Aug 6 2018, 7:53 AM

Change 450483 merged by ArielGlenn:
[operations/puppet@production] Fix target file for diff between weekly and daily

https://gerrit.wikimedia.org/r/450483

gerritbot added a comment.Aug 13 2018, 7:30 PM

Change 452462 had a related patch set uploaded (by Smalyshev; owner: Smalyshev):
[wikidata/query/rdf@master] Add script for loading daily dump

https://gerrit.wikimedia.org/r/452462

gerritbot added a comment.Aug 13 2018, 8:44 PM

Change 452569 had a related patch set uploaded (by Smalyshev; owner: Smalyshev):
[operations/puppet@production] Add ability to load daily category dumps.

https://gerrit.wikimedia.org/r/452569

gerritbot added a comment.Aug 13 2018, 9:13 PM

Change 452462 merged by jenkins-bot:
[wikidata/query/rdf@master] Add script for loading daily dump

https://gerrit.wikimedia.org/r/452462

gerritbot added a comment.Aug 13 2018, 10:37 PM

Change 450880 had a related patch set uploaded (by Smalyshev; owner: Smalyshev):
[mediawiki/core@master] Use separate insert/delete statements to work around Blazegraph bug

https://gerrit.wikimedia.org/r/450880

Smalyshev moved this task from Doing to In review on the User-Smalyshev board.Aug 13 2018, 10:37 PM
Smalyshev moved this task from Backlog to Needs review on the Discovery-Wikidata-Query-Service-Sprint board.
gerritbot added a comment.Aug 14 2018, 12:31 AM

Change 450880 merged by jenkins-bot:
[mediawiki/core@master] Use separate insert/delete statements to work around Blazegraph bug

https://gerrit.wikimedia.org/r/450880

ReleaseTaggerBot added a project: MW-1.32-notes (WMF-deploy-2018-08-14 (1.32.0-wmf.17)).Aug 14 2018, 1:00 AM
Jdforrester-WMF edited projects, added MW-1.32-notes (WMF-deploy-2018-08-21 (1.32.0-wmf.18)); removed MW-1.32-notes (WMF-deploy-2018-08-14 (1.32.0-wmf.17)).Aug 14 2018, 8:55 PM
gerritbot added a comment.Aug 20 2018, 5:27 PM

Change 454067 had a related patch set uploaded (by Smalyshev; owner: Smalyshev):
[operations/puppet@production] Enable daily category diffs on test

https://gerrit.wikimedia.org/r/454067

gerritbot added a comment.Aug 20 2018, 6:40 PM

Change 452569 merged by Gehel:
[operations/puppet@production] Add ability to load daily category dumps.

https://gerrit.wikimedia.org/r/452569

Smalyshev moved this task from In review to Doing on the User-Smalyshev board.Aug 20 2018, 6:58 PM
Smalyshev moved this task from Doing to Waiting/Blocked on the User-Smalyshev board.Aug 21 2018, 12:15 AM
Smalyshev moved this task from Waiting/Blocked to Doing on the User-Smalyshev board.Aug 24 2018, 11:27 PM
gerritbot added a comment.Aug 27 2018, 7:14 PM

Change 454067 merged by Gehel:
[operations/puppet@production] Enable daily category diffs on test

https://gerrit.wikimedia.org/r/454067

Smalyshev moved this task from Incoming to Current work on the Wikidata-Query-Service board.Aug 28 2018, 11:16 PM
gerritbot added a comment.Aug 29 2018, 4:00 PM

Change 456170 had a related patch set uploaded (by Smalyshev; owner: Smalyshev):
[operations/puppet@production] Enable dailies everywhere

https://gerrit.wikimedia.org/r/456170

gerritbot added a comment.Sep 10 2018, 8:01 PM

Change 456170 merged by Gehel:
[operations/puppet@production] Enable dailies everywhere

https://gerrit.wikimedia.org/r/456170

Smalyshev moved this task from Doing to Done on the User-Smalyshev board.Sep 10 2018, 8:57 PM
Smalyshev closed this task as Resolved.Sep 10 2018, 9:22 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL