Page MenuHomePhabricator
Create Task
Maniphest T349011

Improve data-reload cookbook based on graph split needs
Closed, ResolvedPublic
Actions

Description

As we prepare the graph split hosts in T347504, I've run into a few issues that need addressing:

  • when selecting a "-all" file the preceding "lexemes" one must be taken, e.g. if wikidata-20230925-all-BETA.ttl.bz2 is taken wikidata-20230922-lexemes-BETA.ttl.bz2 must be taken.
  • The cookbook tries to stop the wdqs-updater, but that fails because that service does not exist on the graph split hosts.
  • If you forget to add the "--no-depool" option, the cookbook doesn't fail until the end of the munging step (which takes several days); instead we should change it so that if the host is not behind LVS it will never try to pool/depool.

Details

Other Assignee
RKemper
Related Changes in Gerrit:
SubjectRepoBranchLines +/-
wdqs: fix roll restart on non-categories hostsoperations/cookbooksmaster+4 -1
wdqs.data-reload: add logic for graph_split hostsoperations/cookbooksmaster+64 -6
wdqs.data-reload: Add logic for graph_split hostsoperations/cookbooksmaster+89 -14
Customize query in gerrit

Related Objects
Search...

Event Timeline

bking created this task.Oct 16 2023, 4:44 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 16 2023, 4:44 PM
bking added a parent task: T347504: WDQS graph split: load data from dumps into new hosts.Oct 16 2023, 4:44 PM
bking claimed this task.Oct 16 2023, 5:19 PM
bking updated Other Assignee, added: RKemper.
bking mentioned this in T347504: WDQS graph split: load data from dumps into new hosts.
bking updated the task description. (Show Details)Oct 16 2023, 5:21 PM
gerritbot added a comment.Oct 16 2023, 9:44 PM

Change 966303 had a related patch set uploaded (by Bking; author: Bking):

[operations/cookbooks@master] wdqs.data-reload: Add logic for graph_split hosts

https://gerrit.wikimedia.org/r/966303

gerritbot added a project: Patch-For-Review.Oct 16 2023, 9:44 PM
bking added a subscriber: dcausse.Oct 16 2023, 10:00 PM

@dcausse We're working on the cookbook change, but this made me realize that the current reload cookbook always takes the latest lexemes dump, which isn't in sync with the wikidata dump (usually it's newer).

Do we always need the lexemes dump file to be older than the wikidata dump? Just trying to figure out whether this is a general problem, or specific to the graph split experiment.

Gehel moved this task from Incoming to Ready for Work on the Data-Platform-SRE board.Oct 17 2023, 3:30 PM
Gehel moved this task from Ready for Work to Quarterly Goals on the Data-Platform-SRE board.Oct 18 2023, 8:30 AM
Gehel triaged this task as Medium priority.Oct 18 2023, 8:33 AM
dcausse added a comment.Oct 18 2023, 9:32 AM

This is not a general problem, we want this particular order only when we want to be aligned with what we have in hdfs which is required in the following scenario:

  • initial bootstrap of the whole system (flink bootstrap state+initial data-reload)
  • the graph split evaluation where we want to be sure that the work we base on top of the hdfs data is the same as the data loaded into one blazegraph

It might just be enough to add two new args to the cookbook to allow specifying the files to download instead of picking latest.

Gehel moved this task from Quarterly Goals to In Progress on the Data-Platform-SRE board.Oct 20 2023, 8:25 AM
gerritbot added a comment.Oct 24 2023, 6:55 PM

Change 966303 abandoned by Ryan Kemper:

[operations/cookbooks@master] wdqs.data-reload: Add logic for graph_split hosts

Reason:

abandon this approach; we'll go with a simpler one

https://gerrit.wikimedia.org/r/966303

Maintenance_bot removed a project: Patch-For-Review.Oct 24 2023, 7:11 PM
gerritbot added a comment.Oct 24 2023, 7:31 PM

Change 968346 had a related patch set uploaded (by Bking; author: Bking):

[operations/cookbooks@master] wdqs.data-reload: add logic for graph_split hosts

https://gerrit.wikimedia.org/r/968346

gerritbot added a project: Patch-For-Review.Oct 24 2023, 7:31 PM
RKemper updated the task description. (Show Details)Nov 1 2023, 9:58 PM
gerritbot added a comment.Nov 1 2023, 10:19 PM

Change 968346 merged by Bking:

[operations/cookbooks@master] wdqs.data-reload: add logic for graph_split hosts

https://gerrit.wikimedia.org/r/968346

Maintenance_bot removed a project: Patch-For-Review.Nov 1 2023, 10:30 PM
bking closed this task as Resolved.Nov 2 2023, 3:58 PM
bking moved this task from In Progress to Done on the Data-Platform-SRE board.

With the above merge, we believe this issue has been addressed. Closing...

Stashbot added a comment.Nov 15 2023, 5:52 PM

Mentioned in SAL (#wikimedia-operations) [2023-11-15T17:52:39Z] <inflatador> bking@wdqs1024 reboot host to hopefully reduce data reload failures T349011

gerritbot added a comment.Jul 30 2025, 9:30 PM

Change #1174560 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/cookbooks@master] wdqs: fix roll restart on non-categories hosts

https://gerrit.wikimedia.org/r/1174560

gerritbot added a project: Patch-For-Review.Jul 30 2025, 9:30 PM
gerritbot added a comment.Jul 30 2025, 9:45 PM

Change #1174560 merged by Ryan Kemper:

[operations/cookbooks@master] wdqs: fix roll restart on non-categories hosts

https://gerrit.wikimedia.org/r/1174560

Maintenance_bot removed a project: Patch-For-Review.Jul 30 2025, 10:31 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits