Have a cron job to create RDF dumps enabled.
Which RDF formats?
Description
Description
Details
Details
Related Changes in Gerrit:
| Subject | Author | Repo | Branch | Lines +/- | |
|---|---|---|---|---|---|
| Add a script to create Wikidata ttl dumps | 2096 | operations/puppet | production | +82 -1 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T88728 Improve Wikimedia dumping infrastructure | |||
| Open | None | T88991 improve Wikidata dumps [tracking] | |||
| Resolved | hoo | T93658 cron job to create RDF dumps | |||
| Resolved | ArielGlenn | T72385 Wikidata JSON dump: file directory location should follow standard patterns | |||
| Stalled | None | T94019 Generate RDF from JSON | |||
| Open | None | T162351 Create a "page prop" RDF dump for Wikidata entities ("pagePropertiesRdf") | |||
| Open | None | T98320 [Task] Create dump of entity redirects (JSON or n-triples) |
Event Timeline
Comment Actions
TTL should be enough for now. Right now command line would be something like:
mwscript extensions/WikidataBuildResources/extensions/Wikibase/repo/maintenance/dumpRdf.php --wiki wikidatawiki --format ttl --output {filename}
but that could change (see T93488 and https://gerrit.wikimedia.org/r/#/c/198803/). So there might be several parts in the future. Right now there's just one and ttl should be enough.
Comment Actions
in production, it would just be:
mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/dumpRdf.php --wiki wikidatawiki --format ttl --output {filename}
instead of using "WikidataBuildResources"
Comment Actions
We need a variant of modules/snapshot/files/dumpwikidatajson.sh for this, which does the sharding and assembling the shards.
https://git.wikimedia.org/blob/operations%2Fpuppet.git/6abb1fd3a7b9e9baa0155732d0e96999ff925527/modules%2Fsnapshot%2Ffiles%2Fdumpwikidatajson.sh
Comment Actions
Ready to run manual command to create a dump:
filename=`date +'%Y%m%d'`; i=0; shards=4; while [ $i -lt $shards ]; do php /srv/mediawiki/multiversion/MWScript.php extensions/Wikidata/extensions/Wikibase/repo/maintenance/dumpRdf.php --wiki wikidatawiki --format ttl --shard $i --sharding-factor $shards 2>> /var/log/wikidatadump/dumpwikidatardf-$filename-$i.log | gzip > /mnt/data/xmldatadumps/temp/wikidataRdf-$filename.$i.gz & let i++; done
Comment Actions
Change 201003 had a related patch set uploaded (by Smalyshev):
T93658: create script for TTL dumps
Comment Actions
Change 201003 had a related patch set uploaded (by Hoo man):
Add a script to create Wikidata ttl dumps
Comment Actions
The cron is in place (but disabled for now). Once the code is deployed and I tested it manually for a bit, we can enable the automatic dump creation. If everything works out, this will be late next week (maybe Wednesday, Thursday).