Page MenuHomePhabricator

Create new blobs_clusterXX tables on new es4 and es5 sections
Closed, ResolvedPublic

Description

We are in process of setting up the new es4 and es5 sections and close as read-only the existing es2 and es3 due to space disk constraints.

While we are still setting up the new eqiad hosts, the masters are already up and ready to get the new tables created.

es4 -> es1020
es5 -> es1023

I would assume that, following the order we already have on db-eqiad.php:
The new tables would be
es4 -> blobs_cluster26
es5 -> blobs_cluster27

This is what we currently have:
es2 -> blobs_cluster24
es3 -> blobs_cluster25

'cluster24' => [ 'blobs table' => 'blobs_cluster24' ],
'cluster25' => [ 'blobs table' => 'blobs_cluster25' ],

I have been trying to look around phabricator and wikitech, but I haven't found the script (if there is one) to create these new tables on their respectively wikis.
While I can do that with a quick script, I guess there's a way to create them with a MW script?

Can someone with more MW knowledge help us here?

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 20 2020, 8:05 AM
Marostegui triaged this task as Medium priority.Feb 20 2020, 8:05 AM
Marostegui moved this task from Triage to Blocked external/Not db team on the DBA board.
Reedy added a comment.Feb 20 2020, 8:43 AM

Presumably it's https://github.com/wikimedia/mediawiki-extensions-WikimediaMaintenance/blob/master/storage/make-all-blobs and https://github.com/wikimedia/mediawiki/blob/master/maintenance/storage/blobs.sql

In the script in WikimediaMaintenance, I guess the path to blobs.sql is gonna be wrong, as that's in MW core. That's easily fixed though

Presumably it's https://github.com/wikimedia/mediawiki-extensions-WikimediaMaintenance/blob/master/storage/make-all-blobs and https://github.com/wikimedia/mediawiki/blob/master/maintenance/storage/blobs.sql

In the script in WikimediaMaintenance, I guess the path to blobs.sql is gonna be wrong, as that's in MW core. That's easily fixed though

Thanks @Reedy!
Indeed, looks like the path is different:

root@mwmaint1002:/srv# find . | grep make-all-blobs
./mediawiki/php-1.35.0-wmf.19/extensions/WikimediaMaintenance/storage/make-all-blobs
./mediawiki/php-1.35.0-wmf.20/extensions/WikimediaMaintenance/storage/make-all-blobs
root@mwmaint1002:/srv# vim ./mediawiki/php-1.35.0-wmf.19/extensions/WikimediaMaintenance/storage/make-all-blobs
root@mwmaint1002:/srv# find . -name blobs.sql
./mediawiki/php-1.35.0-wmf.19/maintenance/storage/blobs.sql
./mediawiki/php-1.35.0-wmf.20/maintenance/storage/blobs.sql

Do you want to give it a go for es4?
es4 master is es1020.eqiad.wmnet

root@es1020.eqiad.wmnet[(none)]> show databases;
+--------------------+
| Database           |
+--------------------+
| heartbeat          |
| information_schema |
| mysql              |
| ops                |
| performance_schema |
| sys                |
+--------------------+
6 rows in set (0.00 sec)
Reedy added a comment.Feb 20 2020, 8:54 AM

I'm guessing that when I moved most of the Wikimedia specific stuff (back in SVN days) from MW core, I probably never updated that path..

And it's possible we've not created a new blob cluster for years too? I'm not sure offhand when it might have last been created

I'm guessing that when I moved most of the Wikimedia specific stuff (back in SVN days) from MW core, I probably never updated that path..

And it's possible we've not created a new blob cluster for years too? I'm not sure offhand when it might have last been created

Around 5 years ago T105843

Change 573522 had a related patch set uploaded (by Reedy; owner: Reedy):
[mediawiki/extensions/WikimediaMaintenance@master] Fix path to blobs.sql in make-all-blobs

https://gerrit.wikimedia.org/r/573522

Change 573525 had a related patch set uploaded (by Reedy; owner: Reedy):
[mediawiki/extensions/WikimediaMaintenance@wmf/1.35.0-wmf.20] Fix path to blobs.sql in make-all-blobs

https://gerrit.wikimedia.org/r/573525

Change 573525 merged by jenkins-bot:
[mediawiki/extensions/WikimediaMaintenance@wmf/1.35.0-wmf.20] Fix path to blobs.sql in make-all-blobs

https://gerrit.wikimedia.org/r/573525

Change 573522 merged by jenkins-bot:
[mediawiki/extensions/WikimediaMaintenance@master] Fix path to blobs.sql in make-all-blobs

https://gerrit.wikimedia.org/r/573522

Mentioned in SAL (#wikimedia-operations) [2020-02-20T10:08:56Z] <Reedy> created $wikidb.blobs_cluster26 on es1020 - T245720

Marostegui added a comment.EditedFeb 20 2020, 10:11 AM

Looking good for es4 - thanks @Reedy!

root@cumin1001:/home/marostegui/git/tendril/bin# /home/marostegui/section es4 | while read host port; do echo "====$host====" ; mysql.py -h$host:$port enwiki -e "show tables";done
====es2022.codfw.wmnet====
Tables_in_enwiki
blobs_cluster26
====es2021.codfw.wmnet====
Tables_in_enwiki
blobs_cluster26
====es2020.codfw.wmnet====
Tables_in_enwiki
blobs_cluster26
====es1022.eqiad.wmnet====
Tables_in_enwiki
blobs_cluster26
====es1021.eqiad.wmnet====
Tables_in_enwiki
blobs_cluster26
====es1020.eqiad.wmnet====
Tables_in_enwiki
blobs_cluster26

Feel free to go for es5 as well on es1023 too (that would be blobs_cluster27 I guess)

Mentioned in SAL (#wikimedia-operations) [2020-02-20T10:12:26Z] <Reedy> created $wikidb.blobs_cluster27 on es1023 - T245720

I have compared the databases existing on es3 and the ones created on es4, there are some existing on es3 and not on the new es4, which I guess those are closed/deleted wikis or mistakes like wikidata instead of wikidatawiki:

@@ -1,3 +1,4 @@
+10wikipedia
@@ -15,6 +16,9 @@
+alswikibooks
+alswikiquote
+alswiktionary
@@ -67,6 +71,7 @@
+bawiktionary
@@ -136,6 +141,7 @@
+chwikimedia
@@ -173,6 +179,7 @@
+de_labswikimedia
@@ -203,6 +210,8 @@
+en_flaggedrevs_labswikimedia
+en_labswikimedia
@@ -260,6 +269,7 @@
+flaggedrevs_labswikimedia
@@ -458,8 +468,6 @@
@@ -476,6 +484,7 @@
+liquidthreads_labswikimedia
@@ -531,6 +540,8 @@
+mowiki
+mowiktionary
@@ -582,6 +593,7 @@
+nomcomwiki
@@ -659,6 +671,7 @@
+readerfeedback_labswikimedia
@@ -759,6 +772,8 @@
+strategyappswiki
+strategyappwiki
@@ -788,6 +803,7 @@
+tenwikipedia
@@ -813,11 +829,17 @@
+tlhwiki
+tlhwiktionary
+tokiponawiki
+tokiponawikibooks
+tokiponawikiquote
+tokiponawiktionary
@@ -849,6 +871,7 @@
+ukwikimedia
@@ -868,8 +891,10 @@
+vewikimedia
+viwikimedia
@@ -886,6 +911,7 @@
+wikidata
@@ -924,6 +950,7 @@
+zerowiki
Marostegui added a comment.EditedFeb 20 2020, 10:28 AM

From those 29 differences, 26 are on the deleted list:

root@cumin1001:/home/marostegui/git/mediawiki-config/dblists# for i in `cat /home/marostegui/diff`; do grep -iR $i deleted.dblist ;done
alswikibooks
alswikiquote
alswiktionary
bawiktionary
chwikimedia
de_labswikimedia
en_labswikimedia
flaggedrevs_labswikimedia
liquidthreads_labswikimedia
mowiki
mowiktionary
nomcomwiki
readerfeedback_labswikimedia
strategyappswiki
tlhwiki
tlhwiktionary
tokiponawiki
tokiponawikibooks
tokiponawikiquote
tokiponawikisource
tokiponawikibooks
tokiponawikiquote
tokiponawiktionary
ukwikimedia
vewikimedia
zerowiki

The other 3 are:
wikidata -> I guess mistake
10wikipedia -> I guess temporary test or something?
tenwikipedia -> I guess temporary test or something?

wikiuser can connect and select on the table:

root@mwmaint1002:~# mysql  -hes1023 enwiki -u wikiuser -p
Enter password:
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 7493
Server version: 10.1.43-MariaDB MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

wikiuser@es1023(enwiki)> select * from blobs_cluster27;
Empty set (0.00 sec)

wikiuser@es1023(enwiki)> Ctrl-C -- exit!
Aborted
root@mwmaint1002:~# mysql  -hes1020 enwiki -u wikiuser -p
Enter password:
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 36888
Server version: 10.1.43-MariaDB MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

wikiuser@es1020(enwiki)> select * from blobs_cluster26;
Empty set (0.00 sec)

wikiuser@es1020(enwiki)>

Same for wikiadmin:

root@mwmaint1002:~# mysql  -hes1020 enwiki -u wikiadmin -p
Enter password:
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 36900
Server version: 10.1.43-MariaDB MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

wikiadmin@es1020(enwiki)> select * from blobs_cluster26;
Empty set (0.00 sec)

wikiadmin@es1020(enwiki)> Ctrl-C -- exit!
Aborted
root@mwmaint1002:~# mysql  -hes1023 enwiki -u wikiadmin -p
Enter password:
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 7521
Server version: 10.1.43-MariaDB MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

wikiadmin@es1023(enwiki)> select * from blobs_cluster27;
Empty set (0.00 sec)

wikiadmin@es1023(enwiki)>

The other 3 are:
wikidata -> I guess mistake
10wikipedia -> I guess temporary test or something?
tenwikipedia -> I guess temporary test or something?

I know it's out of the scope for this ticket, and probably should have another filed.. But is there anything in those on any host? Can we drop them?

The other 3 are:
wikidata -> I guess mistake
10wikipedia -> I guess temporary test or something?
tenwikipedia -> I guess temporary test or something?

I know it's out of the scope for this ticket, and probably should have another filed.. But is there anything in those on any host? Can we drop them?

es1:
wikidata doesn't exist
1 row on blobs_cluster22
0 rows on blobs_cluster23

es2:
0 rows on the 3 databases

es3:
wikidata has 1 row
10wikipedia empty
tenwikipedia empty

Let's consider creating a lower prorirty, separate ticket to track "cleaning up cruft", closing this as soon as we are happy with the result. We can also include offtopic dbs like P10423, but it would be nice to count with @Reedy's archeology expertise. 0:-)

And it's possible we've not created a new blob cluster for years too

Indeed that hasn't happened since I got at WMF, last time there was a space concern, hosts got upgraded. There was some tests during the flowdb research to separate serialization of those, but it only got tested on beta, I think.

Marostegui closed this task as Resolved.Feb 20 2020, 1:39 PM
Marostegui assigned this task to Reedy.

Everything looks good.
Thanks @Reedy for getting this so fast!