Page MenuHomePhabricator
Create Task
Maniphest T415492

Request: make main and scholarly graphs available on WDQS test nodes
Closed, ResolvedPublic
Actions

Description

In order to test triple stores, we need main and scholarly graphs available on test nodes (wdqs1029, wdqs1030, wdqs1031, wdqs1032).

Could you initiate the transfer for us?
As a follow-up, I’d like us to start thinking about automated ways to move datasets generated by the batch pipelines.

It would be great if we could use storage that supports the S3 protocol, such as Ceph. The database backends we’re considering to replace Blazegraph have fast enough data ingestion to support reconciliation in a lambda-architecture style (links to Google Doc). This is a path I’d like to explore as we redesign the WDQS architecture.

We are thinking of the following setup:
wdqs1029 -> qlever main graph
wdqs1030 -> qlever scholarly graph
wdqs1031 -> virtuoso main graph
wdqs1032 -> virtuoso scholarly graph

wdqs1028 will be kept as full graph node, with qlever and virtuoso in multi-tenant that we can use for experimentation/break things at will.

We just need the datasets, and will take care of indexing ourselves.

AC

  • main and scholarly triples datasets are available on test eqiad nodes.

References

Related Objects

Event Timeline

gmodena created this task.Jan 26 2026, 9:18 AM
Restricted Application added a subscriber: • Aklapper. · View Herald TranscriptJan 26 2026, 9:18 AM
gmodena updated the task description. (Show Details)Jan 26 2026, 9:19 AM
gmodena updated the task description. (Show Details)
Maintenance_bot added a project: Wikidata.Jan 26 2026, 9:29 AM
Gehel subscribed.Jan 30 2026, 10:23 AM

Where is the sources of that data? The wikidata ttl dumps? or a munged version available on HDFS?

gmodena updated the task description. (Show Details)Feb 10 2026, 10:37 AM
gmodena added a comment.Feb 10 2026, 10:42 AM
In T415492#11569221, @Gehel wrote:

Where is the sources of that data? The wikidata ttl dumps? or a munged version available on HDFS?

I thought we only had splits for the munged version in nt format (on HDFS), but we'll happily take splits of the un-munged ttl dumps if available.

gmodena added a comment.Feb 13 2026, 2:55 PM
In T415492#11600879, @gmodena wrote:
In T415492#11569221, @Gehel wrote:

Where is the sources of that data? The wikidata ttl dumps? or a munged version available on HDFS?

I thought we only had splits for the munged version in nt format (on HDFS), but we'll happily take splits of the un-munged ttl dumps if available.

The snapshot we need would be
/wmf/data/discovery/wikidata/munged_n3_dump/wikidata/main/20260209 and /wmf/data/discovery/wikidata/munged_n3_dump/wikidata/scholarly/20260209 (or whatever the latest is once ready to transfer). My understanding is that to move from hdfs to wdqs nodes we need to run the data transfer cookbook, and that'd require SRE powers.

RKemper subscribed.Feb 13 2026, 4:29 PM
Gehel triaged this task as High priority.Feb 17 2026, 9:19 AM
Gehel edited projects, added: Data-Platform-SRE (2026-02-13 - 2026-03-06); removed: Data-Platform-SRE.
Gehel moved this task from Backlog - project to Backlog - operations on the Data-Platform-SRE (2026-02-13 - 2026-03-06) board.

WDP would like to have a first transfer soon. We can discuss how t automate this later.

gmodena added a comment.Feb 17 2026, 12:45 PM
In T415492#11621824, @Gehel wrote:

WDP would like to have a first transfer soon. We can discuss how t automate this later.

Thanks!

For context, this one off would be to let us index with virtuoso and qlever and have a baseline to experiment with traffic replay and query rewrites (part of T414782: Hypothesis WE2.5.1: setup triple stores in eqiad).

bking subscribed.Mar 2 2026, 7:02 PM

@gmodena can you confirm that you still need a transfer from HDFS? I ask because the ttl and nt dumps appear to be available on wdqs1028 via NFS:

bking@wdqs1028:/mnt/nfs/dumps-clouddumps1002.wikimedia.org/other/wikibase/wikidatawiki$ ls
20260116  20260123  20260202  20260211	20260220  dcatap.rdf	       latest-all.nt.gz		latest-lexemes.json.gz	latest-lexemes.ttl.gz
20260119  20260126  20260204  20260213	20260223  latest-all.json.bz2  latest-all.ttl.bz2	latest-lexemes.nt.bz2	latest-truthy.nt.bz2
20260120  20260128  20260206  20260216	20260225  latest-all.json.gz   latest-all.ttl.gz	latest-lexemes.nt.gz	latest-truthy.nt.gz
20260121  20260130  20260209  20260218	20260227  latest-all.nt.bz2    latest-lexemes.json.bz2	latest-lexemes.ttl.bz2

Let me know if I'm missing something.

gmodena added a comment.Mar 3 2026, 7:44 AM

@bking we do have access to the full xml dump, but we still need a transfer of the scholarly and main (munged) datasets that are generated on Hadoop.

BTullis claimed this task.Mar 3 2026, 9:37 AM
BTullis moved this task from Backlog - operations to In Progress on the Data-Platform-SRE (2026-02-13 - 2026-03-06) board.
BTullis added a comment.Mar 3 2026, 2:08 PM

I'm just checking the sizes of the files on HDFS that are present for these split sub-graphs.

btullis@an-launcher1003:~$ sudo -u hdfs kerberos-run-command hdfs hdfs dfs -du -s -h /wmf/data/discovery/wikidata/munged_n3_dump/wikidata/main/20260209
109.5 G  /wmf/data/discovery/wikidata/munged_n3_dump/wikidata/main/20260209

btullis@an-launcher1003:~$ sudo -u hdfs kerberos-run-command hdfs hdfs dfs -du -s -h /wmf/data/discovery/wikidata/munged_n3_dump/wikidata/scholarly/20260209
109.5 G  /wmf/data/discovery/wikidata/munged_n3_dump/wikidata/scholarly/20260209

It strikes me as a little odd that these directories are exactly the same size as each other, but we can come back to that.

For this first transfer, I think what I will do is to copy the files from HDFS to the local file system on an-master1004 and then use transfer.py to copy each dump type to the destination hosts.
The reason for this is that transfer.py can't access an HDFS source at the moment. Our sre.wdqs.data-transfer cookbook is only optimised for copying journal files between one host and another.
This might well work, but it wouldn't give you the named snapshot that you indicated from HDFS.

So I'm trying this first to create the first local copy.

btullis@an-master1004:/srv$ sudo mkdir tmp
btullis@an-master1004:/srv$ cd tmp
btullis@an-master1004:/srv/tmp$ sudo chown hdfs:hdfs .
btullis@an-master1004:/srv/tmp$ sudo -u hdfs kerberos-run-command hdfs hdfs dfs -get /wmf/data/discovery/wikidata/munged_n3_dump/wikidata/main/20260209 ./main-20260209
BTullis added a comment.Mar 3 2026, 3:04 PM

Those copy operations completed.

Now I am starting to copy the files to the destination hosts like this:

btullis@cumin1003:~$ sudo transfer.py an-master1004.eqiad.wmnet:/srv/tmp/main-20260209 wdqs1029.eqiad.wmnet:/srv/tmp
2026-03-03 15:03:45  INFO: About to transfer /srv/tmp/main-20260209 from an-master1004.eqiad.wmnet to ['wdqs1029.eqiad.wmnet']:['/srv/tmp'] (117599576516 bytes).
BTullis added a comment.Mar 3 2026, 3:47 PM

The two main versions are available now.

btullis@cumin1003:~$ sudo transfer.py an-master1004.eqiad.wmnet:/srv/tmp/main-20260209 wdqs1029.eqiad.wmnet:/srv/tmp
2026-03-03 15:03:45  INFO: About to transfer /srv/tmp/main-20260209 from an-master1004.eqiad.wmnet to ['wdqs1029.eqiad.wmnet']:['/srv/tmp'] (117599576516 bytes).
2026-03-03 15:11:05  WARNING: Original size on an-master1004.eqiad.wmnet is 117599576516 bytes but transferred size is 117599527364 bytes after copy to wdqs1029.eqiad.wmnet.
2026-03-03 15:11:06  INFO: Parallel checksum of source on an-master1004.eqiad.wmnet and the transmitted ones on wdqs1029.eqiad.wmnet match.
2026-03-03 15:11:06  INFO: 117599527364 bytes correctly transferred from an-master1004.eqiad.wmnet to wdqs1029.eqiad.wmnet
2026-03-03 15:11:08  INFO: Cleaning up....
btullis@cumin1003:~$ sudo transfer.py an-master1004.eqiad.wmnet:/srv/tmp/main-20260209 wdqs1031.eqiad.wmnet:/srv/tmp
2026-03-03 15:14:54  INFO: About to transfer /srv/tmp/main-20260209 from an-master1004.eqiad.wmnet to ['wdqs1031.eqiad.wmnet']:['/srv/tmp'] (117599576516 bytes).
2026-03-03 15:22:31  WARNING: Original size on an-master1004.eqiad.wmnet is 117599576516 bytes but transferred size is 117599527364 bytes after copy to wdqs1031.eqiad.wmnet.
2026-03-03 15:22:32  INFO: Parallel checksum of source on an-master1004.eqiad.wmnet and the transmitted ones on wdqs1031.eqiad.wmnet match.
2026-03-03 15:22:32  INFO: 117599527364 bytes correctly transferred from an-master1004.eqiad.wmnet to wdqs1031.eqiad.wmnet
2026-03-03 15:22:34  INFO: Cleaning up....

I also did a chown -R gmodena:wikidev /srv/tmp on both of these hosts, so you should be able to access the files now.

The transfers of the scholarly graph is starting now.

BTullis added a comment.Mar 3 2026, 4:49 PM

Those should all be available now.

btullis@cumin1003:~$ sudo cumin wdqs10[29-32].eqiad.wmnet 'du -sh /srv/tmp/*'
4 hosts will be targeted:
wdqs[1029-1032].eqiad.wmnet
OK to proceed on 4 hosts? Enter the number of affected hosts to confirm or "q" to quit: 4
===== NODE GROUP =====                                                                                                                                                                                             
(2) wdqs[1029,1031].eqiad.wmnet                                                                                                                                                                                    
----- OUTPUT of 'du -sh /srv/tmp/*' -----                                                                                                                                                                          
110G    /srv/tmp/main-20260209                                                                                                                                                                                     
===== NODE GROUP =====                                                                                                                                                                                             
(2) wdqs[1030,1032].eqiad.wmnet                                                                                                                                                                                    
----- OUTPUT of 'du -sh /srv/tmp/*' -----                                                                                                                                                                          
110G    /srv/tmp/scholarly-20260209                                                                                                                                                                                
================                                                                                                                                                                                                   
PASS |█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (4/4) [00:01<00:00,  3.19hosts/s]
FAIL |                                                                                                                                                                             |   0% (0/4) [00:01<?, ?hosts/s]
100.0% (4/4) success ratio (>= 100.0% threshold) for command: 'du -sh /srv/tmp/*'.
100.0% (4/4) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.

Please let me know if you find that they're not accessible.

I'm still dubious as to the identical size of the two dump types, but I'll defer to you @gmodena to check those.

BTullis closed this task as Resolved.Mar 3 2026, 4:50 PM
BTullis updated the task description. (Show Details)
BTullis moved this task from In Progress to Done on the Data-Platform-SRE (2026-02-13 - 2026-03-06) board.
gmodena added a comment.Mar 4 2026, 6:57 PM
In T415492#11669144, @BTullis wrote:

I'm still dubious as to the identical size of the two dump types, but I'll defer to you @gmodena to check those.

Yeah.. That's odd. I'm indexing and will run some sanity checks before running experiments. I'll ping you by Friday if something is off.

Thank you so much for the help with this transfer!

Gehel moved this task from Done to Reported on the Data-Platform-SRE (2026-02-13 - 2026-03-06) board.Mar 6 2026, 9:12 AM
gmodena mentioned this in T424256: request: data transfer: wikidata main-20260209 snapshot to wdqs2009.Apr 23 2026, 6:58 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits