Page MenuHomePhabricator

Create backups of user tables from decommissioned database servers
Closed, ResolvedPublic

Description

While user databases were provided with the disclaimer that there were no backups. Users migrating from the Toolserver were not informed of this (TS had User database backups).

In short, it would be a shame to let another tragedy as happened with the Toolserver purge happen again. 8 TB HDD should be enough and those are only $150.

Event Timeline

Dispenser triaged this task as High priority.Dec 28 2017, 10:41 PM
Dispenser created this task.
Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptDec 28 2017, 10:41 PM

@Dispenser, backup is fine, but have you thought about making GHEL work with the new setup?

@Dispenser are there particular databases/tables that you know should be backed up which have not already been taken care of by their maintainers? I'm not opposed to saving useful data, but I'm not sure I see the point in making a lot of mysql dump files that will never be looked at again. In the past when we have reached out to specific users with large databases the nearly unanimous response has been "I don't remember why I created that".

8 TB HDD should be enough and those are only $150.

That is a very simplistic view, enterprise grade disks are way more expensive than that- without taking into account shipping and support (even I cannot find 8TB consumer disks lower than $250-300 before taxes); 8TB are not enough to hold user databases; redundancy is needed to accommodate file storage, plus several dumps on different datacenters, and the redundant machines needed to perform such operations; also a method has to be setup that doesn't interfere with the normal operation of the servers (and many people create and drop tables continuously). Also some people has to setup, maintain and test a backup system, both the hardware and the software. There is also concerns about privacy- people are told to not store user's private data on databases, but there is no way to enforce or know that out of thousands of users. Note anyway that disk resources are not the reason (or at least the main one) why user databases are not backed up- backups are a serious thing, either they are done well, or they are not done and that is indicated as is.

Non-public, one time dumps "just in case" is something that is "easy" to implement (and we _do_ that every time a db is going to be deleted), a backup system is not.

A full archive seems unmanageable from an access control point of view. Releasing any of the data held in a non *_p database would require a data audit. I suppose we could dump and archive most of the *_p user and tool databases and give some longer term recovery window (e.g. 6-12 months). That would also allow a community member to upload the dumps to Internet Archive or a similar "permanent" storage if they felt it was incredibility valuable information.

A full archive seems unmanageable from an access control point of view. Releasing any of the data held in a non *_p database would require a data audit. I suppose we could dump and archive most of the *_p user and tool databases and give some longer term recovery window (e.g. 6-12 months). That would also allow a community member to upload the dumps to Internet Archive or a similar "permanent" storage if they felt it was incredibility valuable information.

Maybe we should start by getting a list users/tools databases present on labsdb1003 obviously excluding core wikis (and their views) as those do not need to be backuped.

bd808 added a comment.Jan 5 2018, 8:09 PM

Maybe we should start by getting a list users/tools databases present on labsdb1003 obviously excluding core wikis (and their views) as those do not need to be backuped.

https://tools.wmflabs.org/tool-db-usage/ provides one view of that. Here is a more raw view of them:

1(u3518@c1.labsdb) [information_schema]> show databases where `Database` regexp '^[psu][0-9]';
2+-------------------------------------+
3| Database |
4+-------------------------------------+
5| p50380g40014__test |
6| p50380g40030__AnomieBOT |
7| p50380g50491__rlrl_enwiki_p |
8| p50380g50491__rlrl_ptwiki_p |
9| p50380g50491__unlikely_enwiki_p |
10| p50380g50491__unlikely_ptwiki_p |
11| p50380g50491_common |
12| p50380g50491_inconsistent_redirects |
13| p50380g50491_oddlinks |
14| p50380g50491_rlrl_enwiki |
15| p50380g50491_unlikely |
16| p50380g50494_data |
17| p50380g50497__wikidb_commonswiki |
18| p50380g50497__wikidb_enwiki |
19| p50380g50497__wikidb_zhwiki |
20| p50380g50506__temp_transient |
21| p50380g50507__temp_transient |
22| p50380g50512__ |
23| p50380g50536__catscan2 |
24| p50380g50551__UTRS |
25| p50380g50553__ilc |
26| p50380g50592__interwikis_p |
27| p50380g50692__DPL_p |
28| p50380g50718__u_drtrigon |
29| p50380g50718__u_p50380g50718 |
30| p50380g50728__hostbot |
31| p50380g50728__hostbot_p |
32| p50380g50816__pop_stats |
33| p50380g50824__contributionsurveyor |
34| p50380g50838__geophotoreq |
35| p50380g50900__catscan2 |
36| p50380g50900_testdb |
37| p50380g50921__ghel_p |
38| p50380g50921__wma_p |
39| s51055__AnomieBOT |
40| s51058__dat |
41| s51073__toolserver |
42| s51111__common_p |
43| s51111__inconsistent_redirects_p |
44| s51111__oddlinks_p |
45| s51111__rlrl_enwiki_p |
46| s51111__rlrl_ptwiki_p |
47| s51111__unlikely_enwiki_p |
48| s51111__unlikely_ptwiki_p |
49| s51113__catscan2 |
50| s51127__dewiki_Bilderwunsch_p |
51| s51127__dewiki_new |
52| s51127__list_transient |
53| s51127__stats_p |
54| s51127__temp_transient |
55| s51127__toolserverdb |
56| s51154__catscan2 |
57| s51156__catscan2 |
58| s51187__xtools_tmp |
59| s51206__ptwikis_p |
60| s51306__copyright_p |
61| s51306__plagiabot |
62| s51312__u_s51312 |
63| s51329__afc_statistics |
64| s51362__erwin85 |
65| s51417__artplusfem_nyc_2015_moma |
66| s51698__yetkin2 |
67| s51739__topic_contribs |
68| s51755__tmp |
69| s51892_toolserverdb |
70| s51892_toolserverdb_p |
71| s51966__temp |
72| s52037_yicheng |
73| s52107_reports |
74| s52198__CrisisResponse |
75| s52252__wikidb |
76| s52334__catscan2 |
77| s52365__devtestwiki |
78| s52467__hashtags |
79| s52490__hashtags_p |
80| s52524__rwtest |
81| s52690__ |
82| s52690__p |
83| s52744__tif |
84| s53003__xtools_dev |
85| s53012__fikarummet_p |
86| s53024__sandbox_p |
87| s53151__spamwatch_test |
88| s53311__wikidata_usage_and_views_p |
89| s53381__extra |
90| u10276__tmp |
91| u11049__Revision_2015JanFeb |
92| u12219__wikied_p |
93| u2029__p |
94| u2029__quarry |
95| u2041__botvbot_p |
96| u2041__ores_p |
97| u2041__thr_p |
98| u2041_test |
99| u2043__infocycle |
100| u2117_test |
101| u2204__testing |
102| u2402__p |
103| u2745_analysis |
104| u2815__ |
105| u2815__p |
106| u2815__piwik |
107| u2815__test |
108| u2815__toolserver_s4b |
109| u2843__tmp |
110| u3182__wp10_p |
111| u3214_testdb |
112| u3356__ |
113| u3533_edit_req_db |
114| u3533_g13_database |
115| u3533_onlyuser |
116| u3617__tmp |
117| u3945__staeiou |
118| u3949__tmp |
119| u4301_robertb |
120| u4785__article |
121| u4974__ores_tmp_p |
122+-------------------------------------+
123117 rows in set (0.01 sec)

1(u3518@c3.labsdb) [(none)]> show databases where `Database` regexp '^[psu][0-9]';
2+---------------------------------------+
3| Database |
4+---------------------------------------+
5| p50380g50491__rlrl_cawiki_p |
6| p50380g50491__rlrl_frwiki_p |
7| p50380g50491__rlrl_lvwiki_p |
8| p50380g50491__unlikely_lvwiki_p |
9| p50380g50491_common |
10| p50380g50497__wikidb_arwiki |
11| p50380g50497__wikidb_commonswiki |
12| p50380g50497__wikidb_enwiki |
13| p50380g50497__wikidb_mediawikiwiki |
14| p50380g50497__wikidb_testwikidatawiki |
15| p50380g50497__wikidb_wikidatawiki |
16| p50380g50497__wikidb_zhwiki |
17| p50380g50497__wikidb_zhwikisource |
18| p50380g50506__temp_transient |
19| p50380g50506__temp_transient_s3 |
20| p50380g50506__temp_transient_s7 |
21| p50380g50507__temp_transient |
22| p50380g50507__temp_transient_s3 |
23| p50380g50507__temp_transient_s7 |
24| p50380g50536__catscan2 |
25| p50380g50536__catscan2_s3 |
26| p50380g50536__catscan2_s7 |
27| p50380g50552__inflections |
28| p50380g50584_bot |
29| p50380g50692__DPL_p |
30| p50380g50718__u_drtrigon |
31| p50380g50718__u_p50380g50718 |
32| p50380g50729__grantsbot |
33| p50380g50769__wvs2 |
34| p50380g50769__wvs2ds |
35| p50380g50838__geophotoreq |
36| p50380g50921__ghel_p |
37| p50380g50921__ghel_p_s3 |
38| p50380g50921__ghel_p_s7 |
39| p50380g50921__wma_p |
40| p50380g50921__wma_p_s3 |
41| p50380g50921__wma_p_s7 |
42| p50380g50995__oudadb |
43| p50380g51713__srwiki |
44| s51072__dwl_p |
45| s51111__common_p |
46| s51111__rlrl_cawiki_p |
47| s51111__rlrl_frwiki_p |
48| s51111__rlrl_lvwiki_p |
49| s51111__unlikely_lvwiki_p |
50| s51117__wikidb_dawiki |
51| s51117__wikidb_eswiki |
52| s51117__wikidb_frwiki |
53| s51117__wikidb_jawiki |
54| s51117__wikidb_kowiki |
55| s51117__wikidb_ruwiki |
56| s51117__wikidb_zh_yuewiki |
57| s51127__list_transient |
58| s51127__temp_transient |
59| s51127__temp_transient_s3 |
60| s51127__temp_transient_s7 |
61| s51154__catscan2 |
62| s51156__catscan2 |
63| s51156__catscan2_s3 |
64| s51156__catscan2_s7 |
65| s51187__xtools_tmp_s3 |
66| s51187__xtools_tmp_s7 |
67| s51206__pt |
68| s51312__u_s51312 |
69| s51329__afc_statistics |
70| s51362__erwin85 |
71| s51362__erwin85_s3 |
72| s51362__erwin85_s7 |
73| s51392__coord |
74| s51430__bswiki_first_page_revisions_p |
75| s51447__cawiki |
76| s51580__hroest |
77| s51837__MerlBot |
78| s51892_toolserverdb |
79| s51892_toolserverdb_p |
80| s51892_toolserverdb_p_s3 |
81| s51892_toolserverdb_p_s7 |
82| s51892_toolserverdb_s3 |
83| s51892_toolserverdb_s7 |
84| s51966__temp |
85| s52107_lime |
86| s52107_reports |
87| s52107_tools |
88| s52334__catscan2 |
89| s52481__citationhunt_bn |
90| s52481__citationhunt_ca |
91| s52481__citationhunt_cs |
92| s52481__citationhunt_el |
93| s52481__citationhunt_en |
94| s52481__citationhunt_es |
95| s52481__citationhunt_fi |
96| s52481__citationhunt_fr |
97| s52481__citationhunt_he |
98| s52481__citationhunt_it |
99| s52481__citationhunt_ja |
100| s52481__citationhunt_lv |
101| s52481__citationhunt_ml |
102| s52481__citationhunt_nb |
103| s52481__citationhunt_nn |
104| s52481__citationhunt_pl |
105| s52481__citationhunt_pt |
106| s52481__citationhunt_ro |
107| s52481__citationhunt_ru |
108| s52481__citationhunt_sv |
109| s52481__citationhunt_zh_hans |
110| s52481__citationhunt_zh_hant |
111| s52481__stats_global |
112| s52524__rwtest |
113| s52690__ |
114| s52690__p |
115| s52861__bwAPI_p |
116| s52861__fnbot |
117| s52861__request |
118| s52861__secWatch |
119| s52861__thanks_de |
120| s52957__wdcat |
121| s52960__testing |
122| s53012__fikarummet_p |
123| s53311__wikidata |
124| u2170__meta_p |
125| u2718__bswiki_first_page_revisions_p |
126| u2815__ |
127| u2815__delete_p |
128| u2815__p |
129| u3617__tmp |
130| u3710__stats |
131| u3949__tmp |
132| u4529_srwiki |
133+---------------------------------------+
134128 rows in set (0.02 sec)

Just the *_p dbs:

(u3518@c1.labsdb) [(none)]> show databases where `Database` regexp '^[psu][0-9].*\_p';
+------------------------------------+
| Database                           |
+------------------------------------+
| p50380g50491__rlrl_enwiki_p        |
| p50380g50491__rlrl_ptwiki_p        |
| p50380g50491__unlikely_enwiki_p    |
| p50380g50491__unlikely_ptwiki_p    |
| p50380g50592__interwikis_p         |
| p50380g50692__DPL_p                |
| p50380g50718__u_p50380g50718       |
| p50380g50728__hostbot_p            |
| p50380g50816__pop_stats            |
| p50380g50921__ghel_p               |
| p50380g50921__wma_p                |
| s51111__common_p                   |
| s51111__inconsistent_redirects_p   |
| s51111__oddlinks_p                 |
| s51111__rlrl_enwiki_p              |
| s51111__rlrl_ptwiki_p              |
| s51111__unlikely_enwiki_p          |
| s51111__unlikely_ptwiki_p          |
| s51127__dewiki_Bilderwunsch_p      |
| s51127__stats_p                    |
| s51206__ptwikis_p                  |
| s51306__copyright_p                |
| s51306__plagiabot                  |
| s51892_toolserverdb_p              |
| s52490__hashtags_p                 |
| s52690__p                          |
| s53012__fikarummet_p               |
| s53024__sandbox_p                  |
| s53311__wikidata_usage_and_views_p |
| u12219__wikied_p                   |
| u2029__p                           |
| u2041__botvbot_p                   |
| u2041__ores_p                      |
| u2041__thr_p                       |
| u2402__p                           |
| u2815__p                           |
| u2815__piwik                       |
| u3182__wp10_p                      |
| u4974__ores_tmp_p                  |
+------------------------------------+
39 rows in set (0.01 sec)
(u3518@c3.labsdb) [(none)]> show databases where `Database` regexp '^[psu][0-9].*\_p';
+---------------------------------------+
| Database                              |
+---------------------------------------+
| p50380g50491__rlrl_cawiki_p           |
| p50380g50491__rlrl_frwiki_p           |
| p50380g50491__rlrl_lvwiki_p           |
| p50380g50491__unlikely_lvwiki_p       |
| p50380g50692__DPL_p                   |
| p50380g50718__u_p50380g50718          |
| p50380g50921__ghel_p                  |
| p50380g50921__ghel_p_s3               |
| p50380g50921__ghel_p_s7               |
| p50380g50921__wma_p                   |
| p50380g50921__wma_p_s3                |
| p50380g50921__wma_p_s7                |
| s51072__dwl_p                         |
| s51111__common_p                      |
| s51111__rlrl_cawiki_p                 |
| s51111__rlrl_frwiki_p                 |
| s51111__rlrl_lvwiki_p                 |
| s51111__unlikely_lvwiki_p             |
| s51206__pt                            |
| s51430__bswiki_first_page_revisions_p |
| s51892_toolserverdb_p                 |
| s51892_toolserverdb_p_s3              |
| s51892_toolserverdb_p_s7              |
| s52481__citationhunt_pl               |
| s52481__citationhunt_pt               |
| s52690__p                             |
| s52861__bwAPI_p                       |
| s53012__fikarummet_p                  |
| u2170__meta_p                         |
| u2718__bswiki_first_page_revisions_p  |
| u2815__delete_p                       |
| u2815__p                              |
+---------------------------------------+
32 rows in set (0.02 sec)

Thanks!
I am a bit confused because _p core databases ie: enwiki_p are views:

mysql:root@localhost [enwiki_p]> show create table revision\G
*************************** 1. row ***************************
                View: revision
         Create View: CREATE ALGORITHM=UNDEFINED

Whereas those users _p databases aren't - ie:

mysql:root@localhost [s51892_toolserverdb_p_s3]> show create table language;
+----------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table    | Create Table                                                                                                                                                                                                                                          |
+----------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| language | CREATE TABLE `language` (
  `lang` varchar(16) NOT NULL,
  `english_name` varchar(255) DEFAULT NULL,
  `native_name` varchar(255) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT NULL,
  PRIMARY KEY (`lang`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1 |
+----------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

My point is, there might be normal user databases without _p on its name that are being used too?

bd808 added a comment.Jan 5 2018, 9:04 PM

My point is, there might be normal user databases without _p on its name that are being used too?

Yes, but at least in theory the non-_p databases are private to the creator.

https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#Privileges_on_the_database:

Users have all privileges and have access to all grant options on their databases. Database names ending with _p are granted read access for everyone.

This means that databases that do not end with _p could contain sensitive information stored by the user or tool that created them. This is was I was referencing in T183758#3878969 as an access control problem. If we archive the non-_p databases then they would need to be treated as potentially containing and go through some audit (like the source audit described for https://wikitech.wikimedia.org/wiki/Help:Toolforge/Abandoned_tool_policy) before being released to anyone other than the original creator.

Marostegui added a comment.EditedJan 5 2018, 9:08 PM

Ah! Gotcha! Missed that point - thanks for clarifying that.

We can try to backup the _p databases you listed then.

We need to be aware that labsdb1001 storage is broken and the disk might die while we are reading those though :(

I have backuped all the _p databases listed on T183758#3879024

ls -lh labsdb1003/
total 4.8G
-rw-r--r-- 1 root root  5.9M Jan  8 08:48 p50380g50491__rlrl_cawiki_p.sql.gz
-rw-r--r-- 1 root root   43M Jan  8 08:48 p50380g50491__rlrl_frwiki_p.sql.gz
-rw-r--r-- 1 root root   32K Jan  8 08:49 p50380g50491__rlrl_lvwiki_p.sql.gz
-rw-r--r-- 1 root root  1.7M Jan  8 08:49 p50380g50491__unlikely_lvwiki_p.sql.gz
-rw-r--r-- 1 root root  606M Jan  8 08:52 p50380g50692__DPL_p.sql.gz
-rw-r--r-- 1 root root   491 Jan  8 08:52 p50380g50718__u_p50380g50718.sql.gz
-rw-r--r-- 1 root root   40M Jan  8 08:54 p50380g50921__ghel_p_s3.sql.gz
-rw-r--r-- 1 root root   48M Jan  8 08:54 p50380g50921__ghel_p_s7.sql.gz
-rw-r--r-- 1 root root  523M Jan  8 08:53 p50380g50921__ghel_p.sql.gz
-rw-r--r-- 1 root root  148M Jan  8 08:58 p50380g50921__wma_p_s3.sql.gz
-rw-r--r-- 1 root root  123M Jan  8 08:58 p50380g50921__wma_p_s7.sql.gz
-rw-r--r-- 1 root root 1015M Jan  8 08:57 p50380g50921__wma_p.sql.gz
-rw-r--r-- 1 root root 1001M Jan  8 09:00 s51072__dwl_p.sql.gz
-rw-r--r-- 1 root root   97K Jan  8 09:00 s51111__common_p.sql.gz
-rw-r--r-- 1 root root  5.9M Jan  8 09:01 s51111__rlrl_cawiki_p.sql.gz
-rw-r--r-- 1 root root  122M Jan  8 09:01 s51111__rlrl_frwiki_p.sql.gz
-rw-r--r-- 1 root root   32K Jan  8 09:01 s51111__rlrl_lvwiki_p.sql.gz
-rw-r--r-- 1 root root  1.7M Jan  8 09:01 s51111__unlikely_lvwiki_p.sql.gz
-rw-r--r-- 1 root root   24M Jan  8 09:02 s51206__pt.sql.gz
-rw-r--r-- 1 root root  894K Jan  8 09:02 s51430__bswiki_first_page_revisions_p.sql.gz
-rw-r--r-- 1 root root  590K Jan  8 09:02 s51892_toolserverdb_p_s3.sql.gz
-rw-r--r-- 1 root root  590K Jan  8 09:02 s51892_toolserverdb_p_s7.sql.gz
-rw-r--r-- 1 root root  590K Jan  8 09:02 s51892_toolserverdb_p.sql.gz
-rw-r--r-- 1 root root   494 Jan  8 09:02 s52481__citationhunt_pl.sql.gz
-rw-r--r-- 1 root root   494 Jan  8 09:02 s52481__citationhunt_pt.sql.gz
-rw-r--r-- 1 root root   483 Jan  8 09:02 s52690__p.sql.gz
-rw-r--r-- 1 root root  3.6M Jan  8 09:02 s52861__bwAPI_p.sql.gz
-rw-r--r-- 1 root root  1.4K Jan  8 09:02 s53012__fikarummet_p.sql.gz
-rw-r--r-- 1 root root   16K Jan  8 09:02 u2170__meta_p.sql.gz
-rw-r--r-- 1 root root  2.6M Jan  8 09:02 u2718__bswiki_first_page_revisions_p.sql
-rw-r--r-- 1 root root  795K Jan  8 09:18 u2718__bswiki_first_page_revisions_p.sql.gz
-rw-r--r-- 1 root root  726M Jan  8 09:03 u2815__delete_p.sql.gz
-rw-r--r-- 1 root root  466M Jan  8 09:05 u2815__p.sql.gz
ls -lh labsdb1001/
total 9.1G
-rw-r--r-- 1 root root 1.5M Jan  8 09:22 p50380g50491__rlrl_enwiki_p.sql.gz
-rw-r--r-- 1 root root  65M Jan  8 09:22 p50380g50491__rlrl_ptwiki_p.sql.gz
-rw-r--r-- 1 root root 129M Jan  8 09:22 p50380g50491__unlikely_enwiki_p.sql.gz
-rw-r--r-- 1 root root  15M Jan  8 09:23 p50380g50491__unlikely_ptwiki_p.sql.gz
-rw-r--r-- 1 root root 1.9M Jan  8 09:23 p50380g50592__interwikis_p.sql.gz
-rw-r--r-- 1 root root 602M Jan  8 09:27 p50380g50692__DPL_p.sql.gz
-rw-r--r-- 1 root root  491 Jan  8 09:27 p50380g50718__u_p50380g50718.sql.gz
-rw-r--r-- 1 root root  15M Jan  8 09:27 p50380g50728__hostbot_p.sql.gz
-rw-r--r-- 1 root root 3.3G Jan  8 09:31 p50380g50816__pop_stats.sql.gz
-rw-r--r-- 1 root root 485M Jan  8 09:32 p50380g50921__ghel_p.sql.gz
-rw-r--r-- 1 root root 1.1G Jan  8 09:34 p50380g50921__wma_p.sql.gz
-rw-r--r-- 1 root root  45M Jan  8 09:35 s51111__common_p.sql.gz
-rw-r--r-- 1 root root 280K Jan  8 09:35 s51111__inconsistent_redirects_p.sql.gz
-rw-r--r-- 1 root root  23K Jan  8 09:35 s51111__oddlinks_p.sql.gz
-rw-r--r-- 1 root root 226M Jan  8 09:35 s51111__rlrl_enwiki_p.sql.gz
-rw-r--r-- 1 root root  60M Jan  8 09:35 s51111__rlrl_ptwiki_p.sql.gz
-rw-r--r-- 1 root root  46M Jan  8 09:36 s51111__unlikely_enwiki_p.sql.gz
-rw-r--r-- 1 root root  15M Jan  8 09:36 s51111__unlikely_ptwiki_p.sql.gz
-rw-r--r-- 1 root root 2.9M Jan  8 09:36 s51127__dewiki_Bilderwunsch_p.sql.gz
-rw-r--r-- 1 root root 1.4K Jan  8 09:36 s51127__stats_p.sql.gz
-rw-r--r-- 1 root root 1.1K Jan  8 09:36 s51206__ptwikis_p.sql.gz
-rw-r--r-- 1 root root 9.0M Jan  8 09:36 s51306__copyright_p.sql.gz
-rw-r--r-- 1 root root  489 Jan  8 09:36 s51306__plagiabot.sql.gz
-rw-r--r-- 1 root root 186K Jan  8 09:36 s51892_toolserverdb_p.sql.gz
-rw-r--r-- 1 root root  26M Jan  8 09:36 s52490__hashtags_p.sql.gz
-rw-r--r-- 1 root root 253M Jan  8 09:37 s52690__p.sql.gz
-rw-r--r-- 1 root root  13K Jan  8 09:37 s53012__fikarummet_p.sql.gz
-rw-r--r-- 1 root root  19K Jan  8 09:37 s53024__sandbox_p.sql.gz
-rw-r--r-- 1 root root 402M Jan  8 09:37 s53311__wikidata_usage_and_views_p.sql.gz
-rw-r--r-- 1 root root 421M Jan  8 09:38 u12219__wikied_p.sql.gz
-rw-r--r-- 1 root root  482 Jan  8 09:38 u2029__p.sql.gz
-rw-r--r-- 1 root root  38M Jan  8 09:38 u2041__botvbot_p.sql.gz
-rw-r--r-- 1 root root 931M Jan  8 09:51 u2041__ores_p.sql.gz
-rw-r--r-- 1 root root  86K Jan  8 09:45 u2041__thr_p.sql.gz
-rw-r--r-- 1 root root  12M Jan  8 09:45 u2402__p.sql.gz
-rw-r--r-- 1 root root  44M Jan  8 09:46 u2815__piwik.sql.gz
-rw-r--r-- 1 root root 812M Jan  8 09:46 u2815__p.sql.gz
-rw-r--r-- 1 root root 125M Jan  8 09:46 u3182__wp10_p.sql.gz
-rw-r--r-- 1 root root 5.2M Jan  8 09:46 u4974__ores_tmp_p.sql.gz

I have done one file per database, so it is easier to recover in case we have to, but tared it into one file:

-rw-r--r-- 1 root root 16G Jan  8 09:57 labsdb_backup.tar


tar -tvf labsdb_backup.tar
drwxr-xr-x root/root         0 2018-01-08 09:49 labsdb1001/
-rw-r--r-- root/root   1540413 2018-01-08 09:22 labsdb1001/p50380g50491__rlrl_enwiki_p.sql.gz
-rw-r--r-- root/root  67686263 2018-01-08 09:22 labsdb1001/p50380g50491__rlrl_ptwiki_p.sql.gz
-rw-r--r-- root/root 134297986 2018-01-08 09:22 labsdb1001/p50380g50491__unlikely_enwiki_p.sql.gz
-rw-r--r-- root/root  14872310 2018-01-08 09:23 labsdb1001/p50380g50491__unlikely_ptwiki_p.sql.gz
-rw-r--r-- root/root   1893487 2018-01-08 09:23 labsdb1001/p50380g50592__interwikis_p.sql.gz
-rw-r--r-- root/root 630229340 2018-01-08 09:27 labsdb1001/p50380g50692__DPL_p.sql.gz
-rw-r--r-- root/root       491 2018-01-08 09:27 labsdb1001/p50380g50718__u_p50380g50718.sql.gz
-rw-r--r-- root/root  14820225 2018-01-08 09:27 labsdb1001/p50380g50728__hostbot_p.sql.gz
-rw-r--r-- root/root 3491506105 2018-01-08 09:31 labsdb1001/p50380g50816__pop_stats.sql.gz
-rw-r--r-- root/root  507634491 2018-01-08 09:32 labsdb1001/p50380g50921__ghel_p.sql.gz
-rw-r--r-- root/root 1168737652 2018-01-08 09:34 labsdb1001/p50380g50921__wma_p.sql.gz
-rw-r--r-- root/root   46506188 2018-01-08 09:35 labsdb1001/s51111__common_p.sql.gz
-rw-r--r-- root/root     286297 2018-01-08 09:35 labsdb1001/s51111__inconsistent_redirects_p.sql.gz
-rw-r--r-- root/root      23366 2018-01-08 09:35 labsdb1001/s51111__oddlinks_p.sql.gz
-rw-r--r-- root/root  236275347 2018-01-08 09:35 labsdb1001/s51111__rlrl_enwiki_p.sql.gz
-rw-r--r-- root/root   62494301 2018-01-08 09:35 labsdb1001/s51111__rlrl_ptwiki_p.sql.gz
-rw-r--r-- root/root   47970227 2018-01-08 09:36 labsdb1001/s51111__unlikely_enwiki_p.sql.gz
-rw-r--r-- root/root   14899115 2018-01-08 09:36 labsdb1001/s51111__unlikely_ptwiki_p.sql.gz
-rw-r--r-- root/root    2963980 2018-01-08 09:36 labsdb1001/s51127__dewiki_Bilderwunsch_p.sql.gz
-rw-r--r-- root/root       1338 2018-01-08 09:36 labsdb1001/s51127__stats_p.sql.gz
-rw-r--r-- root/root       1044 2018-01-08 09:36 labsdb1001/s51206__ptwikis_p.sql.gz
-rw-r--r-- root/root    9412636 2018-01-08 09:36 labsdb1001/s51306__copyright_p.sql.gz
-rw-r--r-- root/root        489 2018-01-08 09:36 labsdb1001/s51306__plagiabot.sql.gz
-rw-r--r-- root/root     189543 2018-01-08 09:36 labsdb1001/s51892_toolserverdb_p.sql.gz
-rw-r--r-- root/root   26446474 2018-01-08 09:36 labsdb1001/s52490__hashtags_p.sql.gz
-rw-r--r-- root/root  264654985 2018-01-08 09:37 labsdb1001/s52690__p.sql.gz
-rw-r--r-- root/root      13015 2018-01-08 09:37 labsdb1001/s53012__fikarummet_p.sql.gz
-rw-r--r-- root/root      19360 2018-01-08 09:37 labsdb1001/s53024__sandbox_p.sql.gz
-rw-r--r-- root/root  420537419 2018-01-08 09:37 labsdb1001/s53311__wikidata_usage_and_views_p.sql.gz
-rw-r--r-- root/root  441124557 2018-01-08 09:38 labsdb1001/u12219__wikied_p.sql.gz
-rw-r--r-- root/root        482 2018-01-08 09:38 labsdb1001/u2029__p.sql.gz
-rw-r--r-- root/root   39321067 2018-01-08 09:38 labsdb1001/u2041__botvbot_p.sql.gz
-rw-r--r-- root/root      87298 2018-01-08 09:45 labsdb1001/u2041__thr_p.sql.gz
-rw-r--r-- root/root   12393968 2018-01-08 09:45 labsdb1001/u2402__p.sql.gz
-rw-r--r-- root/root  850514180 2018-01-08 09:46 labsdb1001/u2815__p.sql.gz
-rw-r--r-- root/root   45189333 2018-01-08 09:46 labsdb1001/u2815__piwik.sql.gz
-rw-r--r-- root/root  131022113 2018-01-08 09:46 labsdb1001/u3182__wp10_p.sql.gz
-rw-r--r-- root/root    5366707 2018-01-08 09:46 labsdb1001/u4974__ores_tmp_p.sql.gz
-rw-r--r-- root/root 3128833245 2018-01-08 09:55 labsdb1001/u2041__ores_p.sql.gz
drwxr-xr-x root/root          0 2018-01-08 09:18 labsdb1003/
-rw-r--r-- root/root    6141643 2018-01-08 08:48 labsdb1003/p50380g50491__rlrl_cawiki_p.sql.gz
-rw-r--r-- root/root   44656606 2018-01-08 08:48 labsdb1003/p50380g50491__rlrl_frwiki_p.sql.gz
-rw-r--r-- root/root      31975 2018-01-08 08:49 labsdb1003/p50380g50491__rlrl_lvwiki_p.sql.gz
-rw-r--r-- root/root    1736028 2018-01-08 08:49 labsdb1003/p50380g50491__unlikely_lvwiki_p.sql.gz
-rw-r--r-- root/root  634801813 2018-01-08 08:52 labsdb1003/p50380g50692__DPL_p.sql.gz
-rw-r--r-- root/root        491 2018-01-08 08:52 labsdb1003/p50380g50718__u_p50380g50718.sql.gz
-rw-r--r-- root/root  547379090 2018-01-08 08:53 labsdb1003/p50380g50921__ghel_p.sql.gz
-rw-r--r-- root/root   41831307 2018-01-08 08:54 labsdb1003/p50380g50921__ghel_p_s3.sql.gz
-rw-r--r-- root/root   50315110 2018-01-08 08:54 labsdb1003/p50380g50921__ghel_p_s7.sql.gz
-rw-r--r-- root/root 1063647794 2018-01-08 08:57 labsdb1003/p50380g50921__wma_p.sql.gz
-rw-r--r-- root/root  154719577 2018-01-08 08:58 labsdb1003/p50380g50921__wma_p_s3.sql.gz
-rw-r--r-- root/root  128866721 2018-01-08 08:58 labsdb1003/p50380g50921__wma_p_s7.sql.gz
-rw-r--r-- root/root 1048637664 2018-01-08 09:00 labsdb1003/s51072__dwl_p.sql.gz
-rw-r--r-- root/root      98620 2018-01-08 09:00 labsdb1003/s51111__common_p.sql.gz
-rw-r--r-- root/root    6141652 2018-01-08 09:01 labsdb1003/s51111__rlrl_cawiki_p.sql.gz
-rw-r--r-- root/root  126973377 2018-01-08 09:01 labsdb1003/s51111__rlrl_frwiki_p.sql.gz
-rw-r--r-- root/root      31957 2018-01-08 09:01 labsdb1003/s51111__rlrl_lvwiki_p.sql.gz
-rw-r--r-- root/root    1736027 2018-01-08 09:01 labsdb1003/s51111__unlikely_lvwiki_p.sql.gz
-rw-r--r-- root/root   24829114 2018-01-08 09:02 labsdb1003/s51206__pt.sql.gz
-rw-r--r-- root/root     915399 2018-01-08 09:02 labsdb1003/s51430__bswiki_first_page_revisions_p.sql.gz
-rw-r--r-- root/root     603811 2018-01-08 09:02 labsdb1003/s51892_toolserverdb_p.sql.gz
-rw-r--r-- root/root     603812 2018-01-08 09:02 labsdb1003/s51892_toolserverdb_p_s3.sql.gz
-rw-r--r-- root/root     603814 2018-01-08 09:02 labsdb1003/s51892_toolserverdb_p_s7.sql.gz
-rw-r--r-- root/root        494 2018-01-08 09:02 labsdb1003/s52481__citationhunt_pl.sql.gz
-rw-r--r-- root/root        494 2018-01-08 09:02 labsdb1003/s52481__citationhunt_pt.sql.gz
-rw-r--r-- root/root        483 2018-01-08 09:02 labsdb1003/s52690__p.sql.gz
-rw-r--r-- root/root    3692505 2018-01-08 09:02 labsdb1003/s52861__bwAPI_p.sql.gz
-rw-r--r-- root/root       1356 2018-01-08 09:02 labsdb1003/s53012__fikarummet_p.sql.gz
-rw-r--r-- root/root      16070 2018-01-08 09:02 labsdb1003/u2170__meta_p.sql.gz
-rw-r--r-- root/root  760700588 2018-01-08 09:03 labsdb1003/u2815__delete_p.sql.gz
-rw-r--r-- root/root  487761449 2018-01-08 09:05 labsdb1003/u2815__p.sql.gz
-rw-r--r-- root/root    2625923 2018-01-08 09:02 labsdb1003/u2718__bswiki_first_page_revisions_p.sql
-rw-r--r-- root/root     813792 2018-01-08 09:18 labsdb1003/u2718__bswiki_first_page_revisions_p.sql.gz

I have left this file at 3 locations:

es2019:/srv/tmp/labsdb_backup
dbstore1001:/srv/tmp/labsdb_backup
dbstore2001:/srv/tmp/labsdb_backup

However, I would like also to leave it on a server that cloud-services-team own, so it is there too. Which one should that be?
@bd808 can you also double check that all the databases you listed are there? Just double checking with another fresh pair of eyes :-)

Thanks!

bd808 added a comment.Jan 8 2018, 8:55 PM

However, I would like also to leave it on a server that cloud-services-team own, so it is there too. Which one should that be?

A /srv/scratch/T183758 directory on labstore1003 seems like a reasonable place to put a public copy of these dumps.

@bd808 can you also double check that all the databases you listed are there? Just double checking with another fresh pair of eyes :-)

All of the databases I named in T183758#3879024 are there. My search was not well constrained and caught things that contain "_p" but do not end with "_p", so there are some we should delete before publishing the dump files to labstore1003 (which will make them accessible by everyone on Toolforge):

  • labsdb1001/p50380g50718__u_p50380g50718.sql.gz
  • labsdb1001/p50380g50816__pop_stats.sql.gz
  • labsdb1001/u2815__piwik.sql.gz
  • labsdb1003/p50380g50718__u_p50380g50718.sql.gz
  • labsdb1003/p50380g50921__ghel_p_s3.sql.gz
  • labsdb1003/p50380g50921__ghel_p_s7.sql.gz
  • labsdb1003/p50380g50921__wma_p_s3.sql.gz
  • labsdb1003/p50380g50921__wma_p_s7.sql.gz
  • labsdb1003/s51206__pt.sql.gz
  • labsdb1003/s51892_toolserverdb_p_s3.sql.gz
  • labsdb1003/s51892_toolserverdb_p_s7.sql.gz
  • labsdb1003/s52481__citationhunt_pl.sql.gz
  • labsdb1003/s52481__citationhunt_pt.sql.gz

Thanks a lot @bd808 for double checking.
I have removed those files and this is now how it looks like:

tar -tvf labsdb_backup.tar
drwxr-xr-x root/root         0 2018-01-09 07:02 labsdb1001/
-rw-r--r-- root/root   1540413 2018-01-08 09:22 labsdb1001/p50380g50491__rlrl_enwiki_p.sql.gz
-rw-r--r-- root/root  67686263 2018-01-08 09:22 labsdb1001/p50380g50491__rlrl_ptwiki_p.sql.gz
-rw-r--r-- root/root 134297986 2018-01-08 09:22 labsdb1001/p50380g50491__unlikely_enwiki_p.sql.gz
-rw-r--r-- root/root  14872310 2018-01-08 09:23 labsdb1001/p50380g50491__unlikely_ptwiki_p.sql.gz
-rw-r--r-- root/root   1893487 2018-01-08 09:23 labsdb1001/p50380g50592__interwikis_p.sql.gz
-rw-r--r-- root/root 630229340 2018-01-08 09:27 labsdb1001/p50380g50692__DPL_p.sql.gz
-rw-r--r-- root/root  14820225 2018-01-08 09:27 labsdb1001/p50380g50728__hostbot_p.sql.gz
-rw-r--r-- root/root 507634491 2018-01-08 09:32 labsdb1001/p50380g50921__ghel_p.sql.gz
-rw-r--r-- root/root 1168737652 2018-01-08 09:34 labsdb1001/p50380g50921__wma_p.sql.gz
-rw-r--r-- root/root   46506188 2018-01-08 09:35 labsdb1001/s51111__common_p.sql.gz
-rw-r--r-- root/root     286297 2018-01-08 09:35 labsdb1001/s51111__inconsistent_redirects_p.sql.gz
-rw-r--r-- root/root      23366 2018-01-08 09:35 labsdb1001/s51111__oddlinks_p.sql.gz
-rw-r--r-- root/root  236275347 2018-01-08 09:35 labsdb1001/s51111__rlrl_enwiki_p.sql.gz
-rw-r--r-- root/root   62494301 2018-01-08 09:35 labsdb1001/s51111__rlrl_ptwiki_p.sql.gz
-rw-r--r-- root/root   47970227 2018-01-08 09:36 labsdb1001/s51111__unlikely_enwiki_p.sql.gz
-rw-r--r-- root/root   14899115 2018-01-08 09:36 labsdb1001/s51111__unlikely_ptwiki_p.sql.gz
-rw-r--r-- root/root    2963980 2018-01-08 09:36 labsdb1001/s51127__dewiki_Bilderwunsch_p.sql.gz
-rw-r--r-- root/root       1338 2018-01-08 09:36 labsdb1001/s51127__stats_p.sql.gz
-rw-r--r-- root/root       1044 2018-01-08 09:36 labsdb1001/s51206__ptwikis_p.sql.gz
-rw-r--r-- root/root    9412636 2018-01-08 09:36 labsdb1001/s51306__copyright_p.sql.gz
-rw-r--r-- root/root        489 2018-01-08 09:36 labsdb1001/s51306__plagiabot.sql.gz
-rw-r--r-- root/root     189543 2018-01-08 09:36 labsdb1001/s51892_toolserverdb_p.sql.gz
-rw-r--r-- root/root   26446474 2018-01-08 09:36 labsdb1001/s52490__hashtags_p.sql.gz
-rw-r--r-- root/root  264654985 2018-01-08 09:37 labsdb1001/s52690__p.sql.gz
-rw-r--r-- root/root      13015 2018-01-08 09:37 labsdb1001/s53012__fikarummet_p.sql.gz
-rw-r--r-- root/root      19360 2018-01-08 09:37 labsdb1001/s53024__sandbox_p.sql.gz
-rw-r--r-- root/root  420537419 2018-01-08 09:37 labsdb1001/s53311__wikidata_usage_and_views_p.sql.gz
-rw-r--r-- root/root  441124557 2018-01-08 09:38 labsdb1001/u12219__wikied_p.sql.gz
-rw-r--r-- root/root        482 2018-01-08 09:38 labsdb1001/u2029__p.sql.gz
-rw-r--r-- root/root   39321067 2018-01-08 09:38 labsdb1001/u2041__botvbot_p.sql.gz
-rw-r--r-- root/root      87298 2018-01-08 09:45 labsdb1001/u2041__thr_p.sql.gz
-rw-r--r-- root/root   12393968 2018-01-08 09:45 labsdb1001/u2402__p.sql.gz
-rw-r--r-- root/root  850514180 2018-01-08 09:46 labsdb1001/u2815__p.sql.gz
-rw-r--r-- root/root  131022113 2018-01-08 09:46 labsdb1001/u3182__wp10_p.sql.gz
-rw-r--r-- root/root    5366707 2018-01-08 09:46 labsdb1001/u4974__ores_tmp_p.sql.gz
-rw-r--r-- root/root 3128833245 2018-01-08 09:55 labsdb1001/u2041__ores_p.sql.gz
drwxr-xr-x root/root          0 2018-01-09 07:11 labsdb1003/
-rw-r--r-- root/root    6141643 2018-01-08 08:48 labsdb1003/p50380g50491__rlrl_cawiki_p.sql.gz
-rw-r--r-- root/root   44656606 2018-01-08 08:48 labsdb1003/p50380g50491__rlrl_frwiki_p.sql.gz
-rw-r--r-- root/root      31975 2018-01-08 08:49 labsdb1003/p50380g50491__rlrl_lvwiki_p.sql.gz
-rw-r--r-- root/root    1736028 2018-01-08 08:49 labsdb1003/p50380g50491__unlikely_lvwiki_p.sql.gz
-rw-r--r-- root/root  634801813 2018-01-08 08:52 labsdb1003/p50380g50692__DPL_p.sql.gz
-rw-r--r-- root/root  547379090 2018-01-08 08:53 labsdb1003/p50380g50921__ghel_p.sql.gz
-rw-r--r-- root/root 1063647794 2018-01-08 08:57 labsdb1003/p50380g50921__wma_p.sql.gz
-rw-r--r-- root/root 1048637664 2018-01-08 09:00 labsdb1003/s51072__dwl_p.sql.gz
-rw-r--r-- root/root      98620 2018-01-08 09:00 labsdb1003/s51111__common_p.sql.gz
-rw-r--r-- root/root    6141652 2018-01-08 09:01 labsdb1003/s51111__rlrl_cawiki_p.sql.gz
-rw-r--r-- root/root  126973377 2018-01-08 09:01 labsdb1003/s51111__rlrl_frwiki_p.sql.gz
-rw-r--r-- root/root      31957 2018-01-08 09:01 labsdb1003/s51111__rlrl_lvwiki_p.sql.gz
-rw-r--r-- root/root    1736027 2018-01-08 09:01 labsdb1003/s51111__unlikely_lvwiki_p.sql.gz
-rw-r--r-- root/root     915399 2018-01-08 09:02 labsdb1003/s51430__bswiki_first_page_revisions_p.sql.gz
-rw-r--r-- root/root     603811 2018-01-08 09:02 labsdb1003/s51892_toolserverdb_p.sql.gz
-rw-r--r-- root/root        483 2018-01-08 09:02 labsdb1003/s52690__p.sql.gz
-rw-r--r-- root/root    3692505 2018-01-08 09:02 labsdb1003/s52861__bwAPI_p.sql.gz
-rw-r--r-- root/root       1356 2018-01-08 09:02 labsdb1003/s53012__fikarummet_p.sql.gz
-rw-r--r-- root/root      16070 2018-01-08 09:02 labsdb1003/u2170__meta_p.sql.gz
-rw-r--r-- root/root  760700588 2018-01-08 09:03 labsdb1003/u2815__delete_p.sql.gz
-rw-r--r-- root/root  487761449 2018-01-08 09:05 labsdb1003/u2815__p.sql.gz
-rw-r--r-- root/root     813792 2018-01-08 09:18 labsdb1003/u2718__bswiki_first_page_revisions_p.sql.gz

I will wait for your ok before copying to labstore1003.
So far it is at:

es2019:/srv/tmp/labsdb_backup
dbstore1001:/srv/tmp/labsdb_backup
dbstore2001:/srv/tmp/labsdb_backup

md5sums are the same on the three files: d4bde7692b1c18e55079e57fa4aa8316

Marostegui moved this task from Triage to In progress on the DBA board.Jan 9 2018, 3:41 PM
bd808 added a comment.Jan 9 2018, 5:51 PM

I will wait for your ok before copying to labstore1003.

Looks good to me. Thanks for all the work on this @Marostegui.

Marostegui moved this task from In progress to Done on the DBA board.Jan 9 2018, 6:18 PM

Thanks for double checking.
I have left it at: labstore1003:/root/labsdb_backup.tar
And its md5sum looks good: d4bde7692b1c18e55079e57fa4aa8316

So I think we are done from the DBA side.

bd808 closed this task as Resolved.Jan 9 2018, 6:39 PM
bd808 assigned this task to Marostegui.

The dumps are now published on the "scratch" volume accessible from Toolforge:

$ find /data/scratch/T183758-user-db-archive
/data/scratch/T183758-user-db-archive
/data/scratch/T183758-user-db-archive/labsdb1001
/data/scratch/T183758-user-db-archive/labsdb1001/s51206__ptwikis_p.sql.gz
/data/scratch/T183758-user-db-archive/labsdb1001/u2815__p.sql.gz
/data/scratch/T183758-user-db-archive/labsdb1001/u2041__ores_p.sql.gz
/data/scratch/T183758-user-db-archive/labsdb1001/s53012__fikarummet_p.sql.gz
/data/scratch/T183758-user-db-archive/labsdb1001/u2029__p.sql.gz
/data/scratch/T183758-user-db-archive/labsdb1001/s51111__rlrl_enwiki_p.sql.gz
/data/scratch/T183758-user-db-archive/labsdb1001/u2041__thr_p.sql.gz
/data/scratch/T183758-user-db-archive/labsdb1001/s51892_toolserverdb_p.sql.gz
/data/scratch/T183758-user-db-archive/labsdb1001/s51127__dewiki_Bilderwunsch_p.sql.gz
/data/scratch/T183758-user-db-archive/labsdb1001/s51111__unlikely_ptwiki_p.sql.gz
/data/scratch/T183758-user-db-archive/labsdb1001/u12219__wikied_p.sql.gz
/data/scratch/T183758-user-db-archive/labsdb1001/p50380g50491__rlrl_enwiki_p.sql.gz
/data/scratch/T183758-user-db-archive/labsdb1001/s51111__rlrl_ptwiki_p.sql.gz
/data/scratch/T183758-user-db-archive/labsdb1001/s51306__copyright_p.sql.gz
/data/scratch/T183758-user-db-archive/labsdb1001/p50380g50921__ghel_p.sql.gz
/data/scratch/T183758-user-db-archive/labsdb1001/u3182__wp10_p.sql.gz
/data/scratch/T183758-user-db-archive/labsdb1001/s52490__hashtags_p.sql.gz
/data/scratch/T183758-user-db-archive/labsdb1001/p50380g50692__DPL_p.sql.gz
/data/scratch/T183758-user-db-archive/labsdb1001/s51111__inconsistent_redirects_p.sql.gz
/data/scratch/T183758-user-db-archive/labsdb1001/s52690__p.sql.gz
/data/scratch/T183758-user-db-archive/labsdb1001/s51111__unlikely_enwiki_p.sql.gz
/data/scratch/T183758-user-db-archive/labsdb1001/s51306__plagiabot.sql.gz
/data/scratch/T183758-user-db-archive/labsdb1001/s53024__sandbox_p.sql.gz
/data/scratch/T183758-user-db-archive/labsdb1001/p50380g50728__hostbot_p.sql.gz
/data/scratch/T183758-user-db-archive/labsdb1001/s53311__wikidata_usage_and_views_p.sql.gz
/data/scratch/T183758-user-db-archive/labsdb1001/p50380g50491__rlrl_ptwiki_p.sql.gz
/data/scratch/T183758-user-db-archive/labsdb1001/p50380g50491__unlikely_enwiki_p.sql.gz
/data/scratch/T183758-user-db-archive/labsdb1001/s51127__stats_p.sql.gz
/data/scratch/T183758-user-db-archive/labsdb1001/p50380g50491__unlikely_ptwiki_p.sql.gz
/data/scratch/T183758-user-db-archive/labsdb1001/u2402__p.sql.gz
/data/scratch/T183758-user-db-archive/labsdb1001/p50380g50592__interwikis_p.sql.gz
/data/scratch/T183758-user-db-archive/labsdb1001/u2041__botvbot_p.sql.gz
/data/scratch/T183758-user-db-archive/labsdb1001/s51111__common_p.sql.gz
/data/scratch/T183758-user-db-archive/labsdb1001/u4974__ores_tmp_p.sql.gz
/data/scratch/T183758-user-db-archive/labsdb1001/s51111__oddlinks_p.sql.gz
/data/scratch/T183758-user-db-archive/labsdb1001/p50380g50921__wma_p.sql.gz
/data/scratch/T183758-user-db-archive/labsdb1003
/data/scratch/T183758-user-db-archive/labsdb1003/p50380g50491__unlikely_lvwiki_p.sql.gz
/data/scratch/T183758-user-db-archive/labsdb1003/u2815__p.sql.gz
/data/scratch/T183758-user-db-archive/labsdb1003/s53012__fikarummet_p.sql.gz
/data/scratch/T183758-user-db-archive/labsdb1003/u2815__delete_p.sql.gz
/data/scratch/T183758-user-db-archive/labsdb1003/u2718__bswiki_first_page_revisions_p.sql.gz
/data/scratch/T183758-user-db-archive/labsdb1003/s51892_toolserverdb_p.sql.gz
/data/scratch/T183758-user-db-archive/labsdb1003/s51111__rlrl_frwiki_p.sql.gz
/data/scratch/T183758-user-db-archive/labsdb1003/u2170__meta_p.sql.gz
/data/scratch/T183758-user-db-archive/labsdb1003/p50380g50921__ghel_p.sql.gz
/data/scratch/T183758-user-db-archive/labsdb1003/s52861__bwAPI_p.sql.gz
/data/scratch/T183758-user-db-archive/labsdb1003/p50380g50692__DPL_p.sql.gz
/data/scratch/T183758-user-db-archive/labsdb1003/s52690__p.sql.gz
/data/scratch/T183758-user-db-archive/labsdb1003/s51111__rlrl_cawiki_p.sql.gz
/data/scratch/T183758-user-db-archive/labsdb1003/s51111__rlrl_lvwiki_p.sql.gz
/data/scratch/T183758-user-db-archive/labsdb1003/s51430__bswiki_first_page_revisions_p.sql.gz
/data/scratch/T183758-user-db-archive/labsdb1003/p50380g50491__rlrl_cawiki_p.sql.gz
/data/scratch/T183758-user-db-archive/labsdb1003/p50380g50491__rlrl_lvwiki_p.sql.gz
/data/scratch/T183758-user-db-archive/labsdb1003/s51111__unlikely_lvwiki_p.sql.gz
/data/scratch/T183758-user-db-archive/labsdb1003/s51111__common_p.sql.gz
/data/scratch/T183758-user-db-archive/labsdb1003/s51072__dwl_p.sql.gz
/data/scratch/T183758-user-db-archive/labsdb1003/p50380g50491__rlrl_frwiki_p.sql.gz
/data/scratch/T183758-user-db-archive/labsdb1003/p50380g50921__wma_p.sql.gz