Page MenuHomePhabricator

Identify tools hosting databases on labsdb100[13] and notify maintainers
Closed, ResolvedPublic

Description

Find out which tools/accounts are creating and updating user-managed databases co-located with the Wiki Replica data. These tools will need to be updated somehow to deal with the removal of user-managed dbs from the new Wiki Replica cluster.

Event Timeline

Initial list of accounts:

1SELECT
2 SUBSTRING_INDEX(table_schema, '_', 1) as tool
3 , count(1) as tables
4FROM information_schema.TABLES
5WHERE table_schema regexp '^[psu][0-9]'
6GROUP BY tool
7ORDER BY tables desc;
8
9-- labsdb1001.eqiad.wmnet
10
11+--------------+--------+
12| tool | tables |
13+--------------+--------+
14| s51187 | 355 |
15| p50380g50497 | 159 |
16| p50380g50491 | 105 |
17| u2815 | 79 |
18| s51111 | 73 |
19| p50380g50692 | 54 |
20| p50380g50512 | 49 |
21| s52365 | 49 |
22| u3532 | 46 |
23| u4817 | 35 |
24| p50380g50816 | 30 |
25| s51454 | 26 |
26| p50380g50728 | 21 |
27| p50380g50921 | 21 |
28| u12219 | 18 |
29| p50380g50494 | 17 |
30| s51362 | 16 |
31| s51127 | 14 |
32| p50380g50551 | 12 |
33| s51892 | 11 |
34| u3617 | 9 |
35| s52690 | 8 |
36| s53311 | 8 |
37| s52467 | 7 |
38| u2041 | 7 |
39| s51755 | 6 |
40| s51154 | 6 |
41| s53381 | 6 |
42| s51698 | 5 |
43| p50380g50553 | 5 |
44| s52490 | 5 |
45| u3533 | 4 |
46| s51329 | 4 |
47| p50380g50900 | 4 |
48| s52037 | 4 |
49| u2402 | 4 |
50| u2248 | 4 |
51| u11049 | 3 |
52| u3356 | 3 |
53| u3945 | 3 |
54| u4301 | 2 |
55| u2043 | 2 |
56| u3949 | 2 |
57| s51306 | 2 |
58| u3290 | 2 |
59| u3570 | 2 |
60| p50380g40014 | 1 |
61| s51058 | 1 |
62| u2117 | 1 |
63| s51172 | 1 |
64| s53012 | 1 |
65| u2843 | 1 |
66| u10276 | 1 |
67| s51055 | 1 |
68| s52744 | 1 |
69| s52107 | 1 |
70| u2745 | 1 |
71| s51223 | 1 |
72| s51417 | 1 |
73| u2029 | 1 |
74| u4974 | 1 |
75| p50380g50592 | 1 |
76| s51206 | 1 |
77| s52524 | 1 |
78| s53151 | 1 |
79| u3214 | 1 |
80| p50380g50838 | 1 |
81| s51073 | 1 |
82| s53024 | 1 |
83| u3182 | 1 |
84| p50380g50824 | 1 |
85| s51334 | 1 |
86+--------------+--------+
8772 rows in set (1.09 sec)
88
89-- labsdb1003.eqiad.wmnet
90
91+--------------+--------+
92| tool | tables |
93+--------------+--------+
94| s51187 | 688 |
95| s51117 | 343 |
96| p50380g50497 | 261 |
97| p50380g50921 | 138 |
98| u2815 | 89 |
99| s51362 | 86 |
100| p50380g50491 | 73 |
101| s51111 | 72 |
102| p50380g50769 | 39 |
103| s52107 | 35 |
104| u3710 | 33 |
105| s51892 | 33 |
106| s52861 | 25 |
107| u4529 | 20 |
108| p50380g50729 | 9 |
109| p50380g50553 | 8 |
110| p50380g50692 | 7 |
111| s51072 | 5 |
112| s51580 | 4 |
113| s52957 | 3 |
114| p50380g50552 | 3 |
115| s53311 | 3 |
116| s51206 | 2 |
117| u2170 | 2 |
118| s51447 | 2 |
119| u3949 | 2 |
120| s51430 | 1 |
121| s53012 | 1 |
122| s51392 | 1 |
123| u3617 | 1 |
124| s51837 | 1 |
125| u2718 | 1 |
126| s52690 | 1 |
127| s52524 | 1 |
128| s51172 | 1 |
129+--------------+--------+
13035 rows in set (1.36 sec)

New tool https://tools.wmflabs.org/tool-db-usage/ can be used to see this data better. It also decodes the owners where possible which will make contacting them a bit easier.

Initial list of accounts:
P5960

I've annotated the labsdb1001 section with wikitech usernames, and notes about broken links and probable maintainers, at P6184.
It should be suitable for (A) Sending a massmessage reminder to all maintainers (after a quick alphabetical shuffle and trim), (B) perhaps pointing the notified maintainers towards, so that they can easily connect the alphanumerical string to their own tool?
I'll do the same for labsdb1003 later in the week.

1-- labsdb1001.eqiad.wmnet
2
3| s51187 | 355 |
4Akoopal
5Alphos
6Cyberpower678
7MZMcBride
8Magnus Manske
9Nemo bis
10PiRSquared17
11Se4598
12TheDJ
13| p50380g50497 | 159 | --- PROBLEM: links to (not found) https://wikitech.wikimedia.org/wiki/User:liagent-php -- See also "tools.liangent-php (s51117)" on labsdb1003
14Liangent
15| p50380g50491 | 105 |
16Tb
17| u2815 | 79 |
18Dispenser
19| s51111 | 73 |
20Tb
21| p50380g50692 | 54 | --- PROBLEM: links to (not found) https://wikitech.wikimedia.org/wiki/User:Dplbot - should be https://tools.wmflabs.org/admin/tool/dplbot
22Dispenser
23JaGa
24Russell Blau
25| p50380g50512 | 49 |
26UNKNOWN
27| s52365 | 49 |
28Mjbmr
29| u3532 | 46 |
30DOESNT EXIST ANYMORE
31| u4817 | 35 |
32DOESNT EXIST ANYMORE
33| p50380g50816 | 30 |
34Kaldari
35Mr.Z-man
36Niharika29
37| s51454 | 26 |
38DOESNT EXIST ANYMORE
39| p50380g50728 | 21 |
40Jmorgan
41| p50380g50921 | 21 |
42Kolossos
43Retsam
44Simon04
45| u12219 | 18 |
46Kjschiroo
47| p50380g50494 | 17 |
48UNKNOWN
49| s51362 | 16 |
50Akoopal
51Alphos
52Cyberpower678
53MZMcBride
54Magnus Manske
55Nemo bis
56PiRSquared17
57Se4598
58TheDJ
59| s51127 | 14 |
60Merl
61| p50380g50551 | 12 | -- probably User:Hersfold User:Martijn_Hoekstra User:Thehelpfulone User:Tparis per https://tools.wmflabs.org/admin/tool/unblock ?
62UNKNOWN
63| s51892 | 11 |
64Jkroll
65Nosy
66Tobias Gritschacher
67| u3617 | 9 |
68Blahma
69| s52690 | 8 |
70dispenser
71| s53311 | 8 |
72Hall1467
73| s52467 | 7 |
74MahmoudHashemi
75Slaporte
76| u2041 | 7 |
77Halfak
78| s51755 | 6 |
79Blahma
80| s51154 | 6 |
81Zache-tool
82| s53381 | 6 |
83Tpt
84| s51698 | 5 |
85Superyetkin
86| p50380g50553 | 5 | --- PROBLEM: links to (not found) https://wikitech.wikimedia.org/wiki/User:suggestbot -- should be https://tools.wmflabs.org/admin/tool/suggestbot
87Bobo.03
88Estuempfig
89Halfak
90Kjschiroo
91Nettrom
92| s52490 | 5 |
93Madhuvishy
94MahmoudHashemi
95Ori.livneh
96Slaporte
97| u3533 | 4 |
98Hasteur
99| s51329 | 4 |
100BenKurtovic
101| p50380g50900 | 4 |
102UNKNOWN
103| s52037 | 4 |
104Anorange0409
105Denny Vrandecic
106Gwiki
107Pasleim
108| u2402 | 4 |
109Jmorgan
110| u2248 | 4 |
111DOESNT EXIST ANYMORE
112| u11049 | 3 |
113Ashwinpp
114| u3356 | 3 |
115Nutz
116| u3945 | 3 |
117Staeiou
118| u4301 | 2 |
119Robertb
120| u2043 | 2 |
121Giovanni Luca Ciampaglia
122| u3949 | 2 |
123Computermacgyver
124| s51306 | 2 |
125Eranroz
126Kaldari
127Lucas559
128MusikAnimal
129Niharika29
130Samwilson
131| u3290 | 2 |
132DOESNT EXIST ANYMORE
133| u3570 | 2 |
134DOESNT EXIST ANYMORE
135| p50380g40014 | 1 |
136UNKNOWN
137| s51058 | 1 |
138Betacommand
139| u2117 | 1 |
140DarTar
141| s51172 | 1 |
142DOESNT EXIST ANYMORE
143| s53012 | 1 |
144Lokal Profil
145Sebastian Berlin (WMSE)
146| u2843 | 1 |
147TTO
148| u10276 | 1 |
149Baku
150| s51055 | 1 |
151Anomie
152| s52744 | 1 |
153BenKurtovic
154| s52107 | 1 | -- PROBLEM: https://tools.wmflabs.org/admin/tool/ptbot lists the non-existent https://wikitech.wikimedia.org/wiki/User:Tools.ptbot as a maintainer
155Arash.pt
156| u2745 | 1 |
157Tb
158| s51223 | 1 |
159DOESNT EXIST ANYMORE
160| s51417 | 1 |
161Dominic
162Jeremyb
163| u2029 | 1 |
164Yuvipanda
165| u4974 | 1 |
166Awight
167| p50380g50592 | 1 |
168UNKNOWN
169| s51206 | 1 |
170Agx
171Alchimista
172Danilo
173Diego Queiroz
174He7d3r
175HenriqueCrang
176Raylton P. Sousa
177Rodrigo Padula
178Sir Lestaty de Lioncourt
179| s52524 | 1 |
180Andrew Bogott
181Coren
182Yuvipanda
183| s53151 | 1 |
184Samtar
185| u3214 | 1 |
186CBM
187| p50380g50838 | 1 | -- probably User:Mr.Z-man per https://tools.wmflabs.org/admin/tool/geophotoreq
188UNKNOWN
189| s51073 | 1 |
190Legoktm
191MZMcBride
192Naraht
193Niharika29
194Tim Landscheidt
195| s53024 | 1 |
196Soni
197| u3182 | 1 |
198Ladsgroup
199| p50380g50824 | 1 | -- probably User:Jalexander per https://tools.wmflabs.org/admin/tool/contributionsurveyor
200UNKNOWN
201| s51334 | 1 |
202MZMcBride

Is it possible to make a dump of the merl DB and put it into the tool dir? Merl has been inactive for some time now, but if case that he returns, that dump might be useful.

Is it possible to make a dump of the merl DB and put it into the tool dir? Merl has been inactive for some time now, but if case that he returns, that dump might be useful.

What is the database name? I cannot find anything if I look for merldb or merl on labsdb1001 and labsdb1003.
I would like to check a few things.

Is it possible to make a dump of the merl DB and put it into the tool dir? Merl has been inactive for some time now, but if case that he returns, that dump might be useful.

What is the database name? I cannot find anything if I look for merldb or merl on labsdb1001 and labsdb1003.
I would like to check a few things.

The match in my list above leads to s51127 and from there to https://tools.wmflabs.org/tool-db-usage/owner/s51127 if that helps?

Is it possible to make a dump of the merl DB and put it into the tool dir? Merl has been inactive for some time now, but if case that he returns, that dump might be useful.

What is the database name? I cannot find anything if I look for merldb or merl on labsdb1001 and labsdb1003.
I would like to check a few things.

The match in my list above leads to s51127 and from there to https://tools.wmflabs.org/tool-db-usage/owner/s51127 if that helps?

Yes, that are the DBs I meant, sorry.

I have made a backup of that directory and left it on:

[root@labsdb1001 06:23 /srv/tmp]
# pwd
/srv/tmp

[root@labsdb1001 06:24 /srv/tmp]
# ls -lh merlbot.tar.gz
-rw-r--r-- 1 root root 4.7M Oct 27 06:23 merlbot.tar.gz

Same place for labsdb1003 (as labsdb1001 will be rebooted).

As above, I've created the list for 1003 at

1-- labsdb1003.eqiad.wmnet
2
3| s51187 | 688 |https://tools.wmflabs.org/admin/tool/xtools
4Cyberpower678
5Matthewrbowker
6MusikAnimal
7Samwilson
8Technical 13
9| s51117 | 343 | --- PROBLEM: links to (not found) https://wikitech.wikimedia.org/wiki/User:liagent-php -- See also "tools.liangent-php (p50380g50497)" on labsdb1001
10| p50380g50497 | 261 | --- ibid
11| p50380g50921 | 138 | --- PROBLEM: links to (not found) https://tools.wmflabs.org/admin/tool/wikiminiatlas -- Probably https://tools.wmflabs.org/admin/tool/wiwosm but that leads to (404) https://tools.wmflabs.org/wiwosm/
12Kolossos
13Retsam
14Simon04
15| u2815 | 89 |
16Dispenser
17| s51362 | 86 |https://tools.wmflabs.org/admin/tool/erwin85
18Akoopal
19Alphos
20Cyberpower678
21MZMcBride
22Magnus Manske
23Nemo bis
24PiRSquared17
25Se4598
26TheDJ
27| p50380g50491 | 73 | https://tools.wmflabs.org/admin/tool/tb-dev
28Harej
29Tb
30Zhuyifei1999
31| s51111 | 72 |https://tools.wmflabs.org/admin/tool/tb-dev
32Harej
33Tb
34Zhuyifei1999
35| p50380g50769 | 39 |https://tools.wmflabs.org/admin/tool/wikiviewstats
36Betacommand
37Cyberpower678
38Hedonil
39Technical 13
40Tools.wikiviewstats2
41Tools.xtools
42| s52107 | 35 |https://tools.wmflabs.org/admin/tool/ptbot
43Arash.pt
44Tools.ptbot
45| u3710 | 33 |
46Hedonil
47| s51892 | 33 |https://tools.wmflabs.org/admin/tool/toolserverdb
48Jkroll
49Nosy
50Tobias Gritschacher
51| s52861 | 25 |https://tools.wmflabs.org/admin/tool/request
52FNDE
53| u4529 | 20 |
54Dungodung
55| p50380g50729 | 9 | --- PROBLEM: points to (non-registered, nor on metawiki) https://wikitech.wikimedia.org/wiki/User:Grantsbot -- ah, should be https://meta.wikimedia.org/wiki/User:GrantsBot
56I JethroBT
57Jmorgan
58| p50380g50553 | 8 | --- PROBLEM: points to (non-registered) https://wikitech.wikimedia.org/wiki/User:Suggestbot -- Probably should be https://en.wikipedia.org/wiki/User:SuggestBot or https://tools.wmflabs.org/admin/tool/suggestbot
59Bobo.03
60Estuempfig
61Halfak
62Kjschiroo
63Nettrom
64Tools.suggestbot
65| p50380g50692 | 7 | --- PROBLEM: points to (non-registered) https://wikitech.wikimedia.org/wiki/User:Dplbot
66Dispenser
67JaGa
68Russell Blau
69| s51072 | 5 |https://tools.wmflabs.org/admin/tool/giftbot
70Doc Taxon
71Gifti
72| s51580 | 4 |https://tools.wmflabs.org/admin/tool/hroest
73Hroest
74| s52957 | 3 |https://tools.wmflabs.org/admin/tool/wdcat
75Merlijn van Deen
76| p50380g50552 | 3 |
77UNKNOWN
78| s53311 | 3 |https://tools.wmflabs.org/admin/tool/tool-account
79Hall1467
80| s51206 | 2 |https://tools.wmflabs.org/admin/tool/ptwikis
81Agx
82Alchimista
83Danilo
84Diego Queiroz
85He7d3r
86HenriqueCrang
87Raylton P. Sousa
88Rodrigo Padula
89Sir Lestaty de Lioncourt
90| u2170 | 2 |
91Alex_Monk
92| s51447 | 2 |https://tools.wmflabs.org/admin/tool/cobain
93Alchimista
94Anskar
95Arnaugir
96Castor
97Coet
98Gerardduenas
99Joancreus
100KRLS
101Lluis tgn
102Nknudsen
103Unapersona
104| u3949 | 2 |
105Computermacgyver
106| s51430 | 1 | --- PROBLEM: links to (not found) https://tools.wmflabs.org/admin/tool/edinbot
107Edin
108| s53012 | 1 |https://tools.wmflabs.org/admin/tool/fikarummet
109Lokal Profil
110Sebastian Berlin (WMSE)
111| s51392 | 1 |https://tools.wmflabs.org/admin/tool/coord
112Myst
113| u3617 | 1 |
114Blahma
115| s51837 | 1 |https://tools.wmflabs.org/admin/tool/taxonbot
116Doc Taxon
117Gifti
118| u2718 | 1 |
119Edin
120| s52690 | 1 |https://tools.wmflabs.org/admin/tool/dispenser
121Dispenser
122| s52524 | 1 |https://tools.wmflabs.org/admin/tool/toolschecker
123Andrew Bogott
124Coren
125Yuvipanda
126| s51172 | 1 |
127MOVED ALREADY?

MassMessage sent to c3.labsdb database users (example).

TheDJ added a subscriber: TheDJ.EditedNov 17 2017, 12:57 AM

I checked the Tool-Erwin's-tools /s51362. The affected data seems to be used by 4 of its tools and is basically all related to category tree traversal caching. These 4 tools all do joins between the tool data and the replication data.

These tools will likely have to be deprecated, as I see no way to quickly convert them into tools that would perform sufficiently. It's been on life support for a long time now and I guess we just ran out.

bd808 closed this task as Resolved.Dec 11 2017, 7:03 AM
bd808 claimed this task.