Page MenuHomePhabricator

Move some wikis to s5
Closed, ResolvedPublic

Description

After wikidatawiki was moved to its own replica set (s8), s5 has very little load and disk usage.

We should move some wikis to it, probably from s3. Based on size (which is not the only indicator- a wiki could have a lot of load but be smaller):

enwikivoyage, cebwiki, shwiki, srwiki & mgwiktionary (labswiki at a later time)

dbstore1001:/srv/backups$ ls -lhS *wik*-*.sql.gz | head -n 100
-rw-r----- 1 root root  95G Jan 10 17:45 enwiki-201801100205.sql.gz
-rw-r----- 1 root root  95G Jan  3 17:46 enwiki-201801030205.sql.gz
-rw-r----- 1 root root  95G Dec 27 17:56 enwiki-201712270205.sql.gz
-rw-r----- 1 root root  83G Jan 10 17:59 commonswiki-201801100205.sql.gz
-rw-r----- 1 root root  83G Jan  3 16:22 commonswiki-201801030205.sql.gz
-rw-r----- 1 root root  83G Dec 27 16:19 commonswiki-201712270205.sql.gz
-rw-r----- 1 root root  68G Jan 11 06:09 wikidatawiki-201801100205.sql.gz
-rw-r----- 1 root root  68G Jan  4 02:04 wikidatawiki-201801030205.sql.gz
-rw-r----- 1 root root  68G Dec 28 01:34 wikidatawiki-201712270205.sql.gz
-rw-r----- 1 root root  46G Jan 10 07:54 dewiki-201801100205.sql.gz
-rw-r----- 1 root root  46G Jan  3 07:17 dewiki-201801030205.sql.gz
-rw-r----- 1 root root  46G Dec 27 07:17 dewiki-201712270205.sql.gz
-rw-r----- 1 root root  29G Jan 10 18:58 ruwiki-201801100205.sql.gz
-rw-r----- 1 root root  29G Jan  3 16:00 ruwiki-201801030205.sql.gz
-rw-r----- 1 root root  29G Dec 27 17:11 ruwiki-201712270205.sql.gz
-rw-r----- 1 root root  24G Jan 10 10:20 frwiki-201801100205.sql.gz
-rw-r----- 1 root root  24G Jan  3 08:36 frwiki-201801030205.sql.gz
-rw-r----- 1 root root  24G Dec 27 08:46 frwiki-201712270205.sql.gz
-rw-r----- 1 root root  16G Jan 11 09:26 plwiki-201801100205.sql.gz
-rw-r----- 1 root root  16G Jan  4 05:38 plwiki-201801030205.sql.gz
-rw-r----- 1 root root  16G Dec 28 06:37 plwiki-201712270205.sql.gz
-rw-r----- 1 root root  14G Jan 10 20:16 itwiki-201801100205.sql.gz
-rw-r----- 1 root root  14G Jan  3 16:07 itwiki-201801030205.sql.gz
-rw-r----- 1 root root  14G Dec 27 14:56 itwiki-201712270205.sql.gz
-rw-r----- 1 root root  13G Jan 10 04:51 enwikivoyage-201801100205.sql.gz
-rw-r----- 1 root root  13G Jan  3 04:36 enwikivoyage-201801030205.sql.gz
-rw-r----- 1 root root  13G Dec 27 04:51 enwikivoyage-201712270205.sql.gz
-rw-r----- 1 root root  12G Jan 10 07:56 eswiki-201801100205.sql.gz
-rw-r----- 1 root root  12G Jan  3 06:51 eswiki-201801030205.sql.gz
-rw-r----- 1 root root  12G Dec 27 07:08 eswiki-201712270205.sql.gz
-rw-r----- 1 root root  10G Jan 10 03:06 arwiki-201801100205.sql.gz
-rw-r----- 1 root root  10G Jan 10 20:12 nlwiki-201801100205.sql.gz
-rw-r----- 1 root root  10G Dec 27 17:19 nlwiki-201712270205.sql.gz
-rw-r----- 1 root root  10G Jan  3 18:09 nlwiki-201801030205.sql.gz
-rw-r----- 1 root root  10G Jan  3 03:25 arwiki-201801030205.sql.gz
-rw-r----- 1 root root 9.9G Dec 27 04:27 arwiki-201712270205.sql.gz
-rw-r----- 1 root root 8.2G Jan 11 14:54 zhwiki-201801100205.sql.gz
-rw-r----- 1 root root 8.1G Jan  4 07:28 zhwiki-201801030205.sql.gz
-rw-r----- 1 root root 8.1G Dec 28 08:31 zhwiki-201712270205.sql.gz
-rw-r----- 1 root root 8.1G Jan 10 06:00 cebwiki-201801100205.sql.gz
-rw-r----- 1 root root 8.1G Jan  3 05:58 cebwiki-201801030205.sql.gz
-rw-r----- 1 root root 8.1G Dec 27 06:13 cebwiki-201712270205.sql.gz
-rw-r----- 1 root root 7.6G Jan 11 08:22 ptwiki-201801100205.sql.gz
-rw-r----- 1 root root 7.6G Jan  4 02:09 ptwiki-201801030205.sql.gz
-rw-r----- 1 root root 7.6G Dec 28 02:16 ptwiki-201712270205.sql.gz
-rw-r----- 1 root root 7.6G Jan 10 10:37 jawiki-201801100205.sql.gz
-rw-r----- 1 root root 7.5G Jan  3 09:01 jawiki-201801030205.sql.gz
-rw-r----- 1 root root 7.5G Dec 27 09:07 jawiki-201712270205.sql.gz
-rw-r----- 1 root root 7.1G Jan 10 11:54 metawiki-201801100205.sql.gz
-rw-r----- 1 root root 7.0G Jan  3 10:00 metawiki-201801030205.sql.gz
-rw-r----- 1 root root 6.9G Jan 10 08:35 huwiki-201801100205.sql.gz
-rw-r----- 1 root root 6.9G Dec 27 10:16 metawiki-201712270205.sql.gz
-rw-r----- 1 root root 6.9G Jan  3 07:04 huwiki-201801030205.sql.gz
-rw-r----- 1 root root 6.9G Dec 27 07:45 huwiki-201712270205.sql.gz
-rw-r----- 1 root root 5.9G Jan 11 00:40 svwiki-201801100205.sql.gz
-rw-r----- 1 root root 5.9G Dec 27 22:55 svwiki-201712270205.sql.gz
-rw-r----- 1 root root 5.9G Jan  3 23:23 svwiki-201801030205.sql.gz
-rw-r----- 1 root root 5.6G Jan 10 22:28 ukwiki-201801100205.sql.gz
-rw-r----- 1 root root 5.5G Jan  3 18:50 ukwiki-201801030205.sql.gz
-rw-r----- 1 root root 5.5G Dec 27 18:59 ukwiki-201712270205.sql.gz
-rw-r----- 1 root root 5.4G Jan 10 18:30 enwiktionary-201801100205.sql.gz
-rw-r----- 1 root root 5.4G Jan  3 10:37 enwiktionary-201801030205.sql.gz
-rw-r----- 1 root root 5.4G Dec 27 13:43 enwiktionary-201712270205.sql.gz
-rw-r----- 1 root root 4.9G Jan 10 23:14 viwiki-201801100205.sql.gz
-rw-r----- 1 root root 4.9G Jan  3 18:53 viwiki-201801030205.sql.gz
-rw-r----- 1 root root 4.9G Dec 27 18:32 viwiki-201712270205.sql.gz
-rw-r----- 1 root root 4.5G Jan 10 07:32 hewiki-201801100205.sql.gz
-rw-r----- 1 root root 4.5G Jan  3 06:17 hewiki-201801030205.sql.gz
-rw-r----- 1 root root 4.5G Dec 27 06:30 hewiki-201712270205.sql.gz
-rw-r----- 1 root root 4.3G Jan 11 00:09 trwiki-201801100205.sql.gz
-rw-r----- 1 root root 4.3G Jan  3 21:20 trwiki-201801030205.sql.gz
-rw-r----- 1 root root 4.3G Dec 27 21:01 trwiki-201712270205.sql.gz
-rw-r----- 1 root root 3.8G Jan 11 01:09 wikishared-201801100205.sql.gz
-rw-r----- 1 root root 3.7G Jan  3 20:22 wikishared-201801030205.sql.gz
-rw-r----- 1 root root 3.7G Dec 27 20:09 wikishared-201712270205.sql.gz
-rw-r----- 1 root root 3.5G Jan 10 18:54 shwiki-201801100205.sql.gz
-rw-r----- 1 root root 3.5G Jan  3 16:18 shwiki-201801030205.sql.gz
-rw-r----- 1 root root 3.5G Dec 27 16:47 shwiki-201712270205.sql.gz
-rw-r----- 1 root root 3.3G Jan 10 11:53 mediawikiwiki-201801100205.sql.gz
-rw-r----- 1 root root 3.3G Jan  3 10:00 mediawikiwiki-201801030205.sql.gz
-rw-r----- 1 root root 3.3G Dec 27 10:13 mediawikiwiki-201712270205.sql.gz
-rw-r----- 1 root root 3.1G Jan 10 05:48 fawiki-201801100205.sql.gz
-rw-r----- 1 root root 3.1G Jan  3 05:03 fawiki-201801030205.sql.gz
-rw-r----- 1 root root 3.1G Dec 27 05:19 fawiki-201712270205.sql.gz
-rw-r----- 1 root root 3.1G Jan  3 15:08 nowiki-201801030205.sql.gz
-rw-r----- 1 root root 3.1G Jan 10 17:00 nowiki-201801100205.sql.gz
-rw-r----- 1 root root 3.1G Dec 27 14:05 nowiki-201712270205.sql.gz
-rw-r----- 1 root root 2.9G Jan 10 13:08 idwiki-201801100205.sql.gz
-rw-r----- 1 root root 2.9G Jan  3 11:35 idwiki-201801030205.sql.gz
-rw-r----- 1 root root 2.9G Dec 27 11:21 idwiki-201712270205.sql.gz
-rw-r----- 1 root root 2.9G Jan 10 09:01 fiwiki-201801100205.sql.gz
-rw-r----- 1 root root 2.9G Jan  3 07:40 fiwiki-201801030205.sql.gz
-rw-r----- 1 root root 2.9G Dec 27 07:22 fiwiki-201712270205.sql.gz
-rw-r----- 1 root root 2.9G Jan  3 02:57 cawiki-201801030205.sql.gz
-rw-r----- 1 root root 2.9G Dec 27 03:51 cawiki-201712270205.sql.gz
-rw-r----- 1 root root 2.8G Jan 10 03:01 cawiki-201801100205.sql.gz
-rw-r----- 1 root root 2.4G Jan 10 06:29 frwiktionary-201801100205.sql.gz
-rw-r----- 1 root root 2.4G Jan  3 05:27 frwiktionary-201801030205.sql.gz
-rw-r----- 1 root root 2.4G Dec 27 05:37 frwiktionary-201712270205.sql.gz
-rw-r----- 1 root root 2.4G Jan 10 21:25 srwiki-201801100205.sql.gz

The plan:

s3 -> s5 wiki movement planning

  • Disable alerts on s5 eqiad and on a slave on s3 eqiad
  • Stop replication on a s3 eqiad slave
  • From s3 codfw slave: mydumper enwikivoyage, cebwiki, shwiki, srwiki & mgwiktionary - it might be worth to do a dry run before that to see how many time it takes
  • On db1070: myloader all those wikis - and we should dry run the loading too
  • Set up multi-source on db1070 but DON'T START SLAVE: change master 's3' to master_host=db1075
  • Set up replication filters on db1070 for s3 replication channel
  • db1070: start slave
  • Prepare patch for db-eqiad.php
  • Once we are read only on both DCs
  • Rename tables on s3 codfw/eqiad master so if any writes arrive, they will fail
  • Once we are completely sure all is good, rename tables on eqiad slaves to make sure if something read from it, fails and we can check logs
  • Leave ^ for a few days before dropping tables on eqiad/codfw

Event Timeline

jcrespo created this task.Jan 12 2018, 3:00 PM
Restricted Application added subscribers: jeblad, Cosine02, Base, Aklapper. · View Herald TranscriptJan 12 2018, 3:00 PM

Only adding Release-Engineering-Team and #wmcs in case they can think of a reason not to move them (groupX reasons) or not to put labswiki there.

@Shizhao I'm sorry but Jcrespo hasn't any decision about zhwiki here, feel free to readd that tag if he really want to move zhwiki to s5 cluster, thank you for regarding.

Marostegui moved this task from Triage to Backlog on the DBA board.Jan 16 2018, 4:18 PM
Ottomata triaged this task as Normal priority.Jan 16 2018, 7:39 PM
Krinkle added a subscriber: Krinkle.Feb 3 2018, 2:18 AM
demon added a subscriber: demon.EditedFeb 21 2018, 9:46 PM

Only adding Release-Engineering-Team and #wmcs in case they can think of a reason not to move them (groupX reasons) or not to put labswiki there.

Group assignments are unrelated to DB cluster assignments from our end--they're mostly based on rough traffic estimates and other weird needs (eg: some Wikipedias like being early). The only caveat is schema changes--and when code rolling out uses them--but we already handle that by feature flags and such.

Thx for checking and the heads up, you should be good to go from our end.

jcrespo added a subscriber: Andrew.Sep 3 2018, 4:35 PM

Wikitech is current only m5- however, on switchover to codfw, it will point to db2037. However, m5-master will continue to be db1073 (read-write) to serve other cloud-related services. This is a problem. CC @Marostegui @Andrew

Restricted Application added a subscriber: Petar.petkovic. · View Herald TranscriptSep 3 2018, 4:35 PM
jcrespo added a comment.EditedOct 2 2018, 7:47 AM

As of 2018-10-02:

ls
-rw-r--r-- 1 dump dump  3.2G Sep 25 21:37 mgwiktionary.gz.tar
-rw-r--r-- 1 dump dump  3.8G Sep 25 21:36 mediawikiwiki.gz.tar
-rw-r--r-- 1 dump dump  3.9G Sep 25 21:39 srwiki.gz.tar
-rw-r--r-- 1 dump dump  4.3G Sep 25 21:39 shwiki.gz.tar
-rw-r--r-- 1 dump dump   11G Sep 25 21:36 cebwiki.gz.tar
-rw-r--r-- 1 dump dump   13G Sep 25 21:39 enwikivoyage.gz.tar

$ du -hs enwikivoyage cebwiki shwiki srwiki mediawikiwiki mgwiktionary
69G     enwikivoyage
500G    cebwiki
65G     shwiki
91G     srwiki
33G     mediawikiwiki
53G     mgwiktionary

From a compressed server:
$ du -hs enwikivoyage cebwiki shwiki srwiki mediawikiwiki mgwiktionary
28G     enwikivoyage
248G    cebwiki
30G     shwiki
43G     srwiki
16G     mediawikiwiki
25G     mgwiktionary

Change 463910 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Depool db1110 for testing s3 imports

https://gerrit.wikimedia.org/r/463910

Change 463910 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: Depool db1110 for testing s3 imports

https://gerrit.wikimedia.org/r/463910

jcrespo added a comment.EditedOct 2 2018, 8:20 AM

Running to check timing and correctness:

# time recover_section.py s3 --database enwikivoyage --host db1110.eqiad.wmnet --port 3306 --user <user> --password <pass>

real    97m55.831s
user    8m54.884s
sys     0m35.136s

# time recover_section.py s3 --database cebwiki --host db1110.eqiad.wmnet --port 3306 --user <user> --password <pass>
Attempting to recover "dump.s3.2018-09-25--18-57-33" ...
Running myloader...

real    200m29.433s
user    12m37.464s
sys     0m53.448s

root@dbstore1001:/srv/backups$ time recover_section.py s3 --database shwiki --host db1110.eqiad.wmnet --port 3306 --user <user> --password <pass>
Attempting to recover "dump.s3.2018-09-25--18-57-33" ...
Running myloader...

real    39m46.095s
user    2m44.484s
sys     0m13.492s

root@dbstore1001:/srv/backups$ time recover_section.py s3 --database srwiki --host db1110.eqiad.wmnet --port 3306 --user <user> --password <pass>
Attempting to recover "dump.s3.2018-09-25--18-57-33" ...
Running myloader...

real    54m52.389s
user    2m29.344s
sys     0m11.740s

root@dbstore1001:/srv/backups$ time recover_section.py s3 --database mgwiktionary --host db1110.eqiad.wmnet --port 3306 --user <user> --password <pass>
Attempting to recover "dump.s3.2018-09-25--18-57-33" ...
Running myloader...

real    38m20.734s
user    2m1.232s
sys     0m9.568s

Recovering at position:

Host: db1075.eqiad.wmnet
Log: db1075-bin.003275
Pos: 679559760

Change 463935 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Move some wikis from s3 to s5

https://gerrit.wikimedia.org/r/463935

jcrespo updated the task description. (Show Details)Oct 2 2018, 1:20 PM

The full import process took around 7-8 hours (although it was quite inefficient to re-import just one database at a time, as the last minutes are just blocked on a single table & connection.

There are now 5 extra databases at db1110 replicating from s3, I wonder if I should also allow the changes from heartbeat. They need to replicate 1 week of changes, so if something breaks we will know. Tomorrow, assuming everything is ok and there are not errors or data drifts, I will reimport everything into the master and prepare the definitive setup.

Command executed, for posterity:

root@db1110.eqiad.wmnet[(none)]> CHANGE MASTER 's3' TO ... <parameters seen above>

root@db1110.eqiad.wmnet[(none)]> set default_master_connection = 's3';
Query OK, 0 rows affected (0.00 sec)

root@db1110.eqiad.wmnet[(none)]> set global replicate_wild_do_table='enwikivoyage.%, cebwiki.%, shwiki.%, srwiki.%, mgwiktionary.%';

root@db1110.eqiad.wmnet[(none)]> start slave 's3';
Query OK, 0 rows affected (0.00 sec)
jcrespo added a comment.EditedOct 3 2018, 7:19 AM

So the plan is:

  • Remove or rename the db1110 tables and stop replicating from s3 there
  • Ignore the extra dbs for s5 connection on multisource host (dbstore1002, labsdb1009/10/11) with:
STOP SLAVE 's5';
set default_master_connection = 's5';
set global replicate_wild_ignore_table='enwikivoyage.%, cebwiki.%, shwiki.%, srwiki.%, mgwiktionary.%';
  • Import to s5 eqiad master (db1070) the databases as above from yesterday's backup, but with the --replicate flag so it is trasmitted to all hosts except the multisource ones
  • Start replicating from s3 eqiad master (at least for now, can be changed later)
  • When catched up, at some point, migrate the multisource to the s5 channel
  • Move replication to codfw s3 master?
  • Rename the s3 tables and deploy the new config to eqiad only

Mentioned in SAL (#wikimedia-operations) [2018-10-03T07:51:21Z] <jynus> deploying replication filtes to s5 at labsdb1009/10/11 and dbstore1002 T184805

Mentioned in SAL (#wikimedia-operations) [2018-10-03T08:18:43Z] <jynus> starting importing of certain s3 wikis into eqiad s5 master T184805

Mentioned in SAL (#wikimedia-operations) [2018-10-03T09:26:47Z] <jynus> reducing io overhead temporarilly in exchange for crash safety for s5 replicas T184805

Change 464164 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Update dblists to move some wikis from s3 to s5

https://gerrit.wikimedia.org/r/464164

jcrespo claimed this task.Oct 3 2018, 11:07 PM
jcrespo moved this task from Backlog to In progress on the DBA board.

Mentioned in SAL (#wikimedia-operations) [2018-10-04T14:09:50Z] <marostegui> Sanitize enwikivoyage cebwiki shwiki srwiki mgwiktionary on db1124:3315 T184805

Change 463935 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: Move some wikis from s3 to s5

https://gerrit.wikimedia.org/r/463935

So the wikis have been loaded into s5, and they are the primary place to read them (and eventually, write them), the only think pending is, some time before the switch, rename the original tables so they are not being written because missconfiguration anymore and the later cleanup.

Change 464797 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] site.pp: Comment fixes due to dewiki no longer being the only s5 wiki

https://gerrit.wikimedia.org/r/464797

This has to be done https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Wiki_Replica_DNS after the dblists are updated (without any puppet change) to change the dns entries on wikireplicas. CC @aborrero

Mentioned in SAL (#wikimedia-operations) [2018-10-08T14:03:43Z] <banyek@deploy1001> Synchronized wmf-config/db-eqiad.php: T184805: Revert 'mariadb: Depool db1110 for testing s3 imports' (duration: 00m 56s)

Mentioned in SAL (#wikimedia-operations) [2018-10-08T14:05:26Z] <banyek@deploy1001> Synchronized wmf-config/db-eqiad.php: T184805: Revert 'mariadb: Depool db1110 for testing s3 imports' (duration: 00m 57s)

Change 464164 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: Update dblists to move some wikis from s3 to s5

https://gerrit.wikimedia.org/r/464164

Mentioned in SAL (#wikimedia-operations) [2018-10-10T10:48:23Z] <marostegui@deploy1001> Synchronized dblists/s3.dblist: Update s3.dblist to reflect the wikis moved to s5 - T184805 (duration: 00m 58s)

Mentioned in SAL (#wikimedia-operations) [2018-10-10T10:49:34Z] <marostegui@deploy1001> Synchronized dblists/s5.dblist: Update s5.dblist to reflect the wikis moved from s3 - T184805 (duration: 00m 56s)

Mentioned in SAL (#wikimedia-operations) [2018-10-10T10:54:28Z] <marostegui> Set a replication filter on db1075 (s3 eqiad) to ignore enwikivoyage, cebwiki, shwiki, srwiki & mgwiktionary - T184805

Mentioned in SAL (#wikimedia-operations) [2018-10-10T11:41:26Z] <jynus> renaming some s3 wiki tables on eqiad master to prevent split brain T184805

Marostegui added a comment.EditedOct 10 2018, 2:43 PM

This was done successfully and new wikis are now live on eqiad.
What is pending now is:

  • Run the DNS changes for wikireplicas: T206623:
  • Stop replication codfw -> eqiad
  • Enable GTID on codfw hosts
  • Remove db2052 (s5 codfw master) filters
  • Re-import the moved wikis into s5 codfw
  • Remove s3 eqiad filters
  • Drop old wikis on s3 (make sure not to do this with replication as that will break dbstore1002 and labsdbXX, it should either be done host by host or with replication filters on labs).

Change 468936 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1100

https://gerrit.wikimedia.org/r/468936

Change 468936 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1100

https://gerrit.wikimedia.org/r/468936

Mentioned in SAL (#wikimedia-operations) [2018-10-22T08:32:59Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1100 - T184805 (duration: 00m 47s)

Mentioned in SAL (#wikimedia-operations) [2018-10-22T08:58:03Z] <marostegui> Stop replication in sync on db1100 and db2052 (codfw master) to reimport wikis - T184805

Mentioned in SAL (#wikimedia-operations) [2018-10-22T09:04:23Z] <marostegui> Run mydumper on db1100 for enwikivoyage cebwiki shwiki srwiki mgwiktionary - T184805

Mentioned in SAL (#wikimedia-operations) [2018-10-22T09:16:45Z] <marostegui> Remove replication filters from db2052 (s5 codfw master) - T184805

Marostegui added a comment.EditedOct 22 2018, 9:18 AM

This was done successfully and new wikis are now live on eqiad.
What is pending now is:

  • Run the DNS changes for wikireplicas: T206623:

Done

  • Stop replication codfw -> eqiad

Done

  • Enable GTID on codfw hosts

Pending and no need to be done as part of this ticket

  • Remove db2052 (s5 codfw master) filters

Done

  • Re-import the moved wikis into s5 codfw

Done

  • Remove s3 eqiad filters

Pending dbstore1002 and labs

  • Drop old wikis on s3 (make sure not to do this with replication as that will break dbstore1002 and labsdbXX, it should either be done host by host or with replication filters on labs).

Pending

Change 464797 merged by Marostegui:
[operations/puppet@production] site.pp: Comment fixes due to dewiki no longer being the only s5 wiki

https://gerrit.wikimedia.org/r/464797

jcrespo reassigned this task from jcrespo to Marostegui.Oct 22 2018, 9:46 AM

Mentioned in SAL (#wikimedia-operations) [2018-10-22T13:19:42Z] <marostegui> Run myloader for enwikivoyage cebwiki shwiki srwiki mgwiktionary on db2052 (s5 codfw master) - T184805

Mentioned in SAL (#wikimedia-operations) [2018-10-23T07:14:40Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1100 - T184805 (duration: 00m 48s)

Mentioned in SAL (#wikimedia-operations) [2018-10-24T10:28:47Z] <marostegui> Compare revision table on dewiki cebwiki shwiki srwiki mgwiktionary enwikivoyage on db1100 and db2075 - T184805

Marostegui reassigned this task from Marostegui to jcrespo.Oct 24 2018, 12:50 PM

I have checked revision and user table for all the wikis in s5:

dewiki cebwiki shwiki srwiki  mgwiktionary enwikivoyage

Between db1100 and db2075 and they are the same. No differences found. dbstore2001:3315 is still catching up.
I have updated the checklist of pending steps at: T184805#4684915

Assigning this to @jcrespo as I won't be doing the drops or filter removal today, as I will be gone for 2 weeks starting in a few hours :-)

Mentioned in SAL (#wikimedia-operations) [2018-10-30T08:24:41Z] <jynus> starting to delete moved to s5, s3 wikis T184805

Banyek added a subscriber: Banyek.Oct 30 2018, 9:26 AM

All those wikis sanitarium set up on db2094 too

All those wikis sanitarium set up on db2094 too

Thanks, I can see the triggers there now. Did you run the check script successfully, too?

yes, of course - I didn't saved the output. :(

Mentioned in SAL (#wikimedia-operations) [2018-10-30T12:01:42Z] <jynus> finishing deleting moved to s5, s3 wikis T184805

In theory the drops finished, but I need to do an additonal pass to check for missing hosts/dbs as well as check/remove filters.

Mentioned in SAL (#wikimedia-operations) [2018-10-30T12:06:40Z] <jynus> removing s3 replication filters on labsdb1009/10/11 T184805

Mentioned in SAL (#wikimedia-operations) [2018-10-30T12:10:14Z] <jynus> removing s3 replication filters on dbstore1002 T184805

No filters left that I can see:

./software/dbtools/section s5 | while read host; do echo $host; mysql.py -h $host -e "SHOW ALL SLAVES STATUS\G" | grep 'Wild' ; done
./software/dbtools/section s3 | while read host; do echo $host; mysql.py -h $host -e "SHOW ALL SLAVES STATUS\G" | grep 'Wild' ; done
jcrespo closed this task as Resolved.Oct 30 2018, 12:39 PM

Everthing at T184805#4654953 done, except the GTID handling, which has to be checked separately for other reasons.

Change 485158 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-codfw.php: Add new wikis to s5 codfw

https://gerrit.wikimedia.org/r/485158

Change 485158 merged by jenkins-bot:
[operations/mediawiki-config@master] db-codfw.php: Add new wikis to s5 codfw

https://gerrit.wikimedia.org/r/485158

Mentioned in SAL (#wikimedia-operations) [2019-01-18T09:23:49Z] <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Add migrated wikis from s3 to s5 to codfw config T184805 (duration: 00m 45s)