A special version of upload.tar which only includes the most recent version of
files would be nice. Perheps only including files that are linked from the
current version of the articles.
Version: unspecified
Severity: enhancement
URL: http://dumps.wikimedia.org/
| Title | Reference | Author | Source Branch | Dest Branch | |
|---|---|---|---|---|---|
| Revert "Exclude deploy servers from target list if /srv/mediawiki is a symlink" | repos/releng/scap!128 | dancy | review/dancy/T329857-cleanup | master | |
| search: Disable auto versioning in glent | repos/data-engineering/airflow-dags!335 | ebernhardson | work/ebernhardson/glent-upload-auto-version | main | |
| search: Bump glent to 0.3.3 | repos/data-engineering/airflow-dags!334 | ebernhardson | work/ebernhardson/glent-0.3.3 | main | |
| search: Bump glent jar to 0.3.2 | repos/data-engineering/airflow-dags!324 | ebernhardson | work/ebernhardson/glent-0.3.2 | main | |
| search: Port transfer_to_es from airflow 1 | repos/data-engineering/airflow-dags!318 | ebernhardson | work/ebernhardson/transfer-to-es | main | |
| HivePartitionWriter: Cast values to appropriate types | repos/search-platform/discolytics!22 | ebernhardson | work/ebernhardson/partition-writer-cast-values | main | |
| Migrate search_satisfaction DAG | repos/data-engineering/airflow-dags!309 | pfischer | migrate-search-dag-search_satisfaction | main | |
| Migrate scripts used by search_satisfaction AirFlow DAG | repos/search-platform/discolytics!20 | pfischer | migrate-cli-for-search_satisfaction-dag | main | |
| Migrate glent_weekly DAG | repos/data-engineering/airflow-dags!292 | pfischer | migrate-search-dag-glent_weekly | main | |
| Exclude deploy servers from target list if /srv/mediawiki is a symlink | repos/releng/scap!103 | dancy | T329857 | master | |
| Migrate scripts used by ores_predictions AirFlow DAG | repos/search-platform/discolytics!17 | pfischer | migrate-cli-for-ores_predictions-dag | main | |
| search: Import popularity score from airflow 1 | repos/data-engineering/airflow-dags!266 | ebernhardson | work/ebernhardson/popularity_score | main | |
| search: Import query clicks from airflow 1 | repos/data-engineering/airflow-dags!265 | ebernhardson | work/ebernhardson/query_clicks | main | |
| search: Port incoming_links from airflow 1 | repos/data-engineering/airflow-dags!252 | ebernhardson | work/ebernhardson/incoming-links | main | |
| search: Port export_queries_to_relforge from airflow 1 | repos/data-engineering/airflow-dags!251 | ebernhardson | work/ebernhardson/export-queries-to-relforge | main | |
| Port incoming_links from airflow v1 | repos/search-platform/discolytics!15 | ebernhardson | work/ebernhardson/incoming_links | main | |
| Port export_queries_to_relforge | repos/search-platform/discolytics!12 | ebernhardson | work/ebernhardson/export-queries-to-relforge | main | |
| search: Port import_cirrus_indexes from airflow 1 | repos/data-engineering/airflow-dags!247 | ebernhardson | work/ebernhardson/import_cirrus_indexes | main | |
| Make scap pull exit if /srv/mediawiki is a symlink | repos/releng/scap!91 | dancy | review/dancy/no-pull-via-symlink | master | |
| Port import_cirrus_indexes | repos/search-platform/discolytics!11 | ebernhardson | work/ebernhardson/import_cirrus_indexes | main |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | None | T29939 Wikimedia should provide media dumps (files, images, etc.) | |||
| Resolved | ArielGlenn | T3298 A "current" version of the upload.tar dir would be nice |
Reopening: This feature-request is not about a recent dump but about a dump
containing only the most recent versions of files.
While actual .tar files are probably not feasible at our current level (~3TB for Commons files current versions only), getting some offsite image mirrors and redistribution is on the table. Tomasz, assigning this one to you since you'll be coordinating the data dump stuff.
These are now semi-available (I'm running them on an ad hoc basis, they are generated on a mirror site rather than one of our servers, we're still working out hardware issues with them, etc etc.) If you're willing to deal with directories moving around and possible inaccessibility, you can get these before the official announcement, from http://ftpmirror.your.org/pub/wikimedia/imagedumps/ in the tarballs/full directory and the tarballs/incrs directory. These are indeed current version only, per project *except* for commons.
If you want commons images, you should get them via rsync from rsync://ftpmirror.your.org/wikimedia-images/ and please see http://meta.wikimedia.org/wiki/Mirroring_Wikimedia_project_XML_dumps#Current_Mirrors for more information about what data is mirrored where.
Anyone on this bug that's not on the xmldatadumps-l list had better get on it, since that's generally where updates about this sort of thing will be sent.
Hmm I guess since the official announcement went out we can call this done, or close enough to done at any rate. (Everyone on the xmldatadumps-l list yet??)