Page MenuHomePhabricator

Prepare and check storage layer for amwikimedia (including dropping s7 version of the wiki)
Closed, ResolvedPublic

Description

We cannot just drop the database on s7 because of multisource (it exist on s3, too). Or we can drop it there and reimport from another server, to make sure it is healthy on dbstore1001/2 and db1095 (sanitarium)

Event Timeline

Is this going to be a public or a private wiki?

fishbowl wiki. Public but not editable (unless with account) like wmfwiki

Ok, then this is not a blocker for the above ticket. We would thank a ping when the actual database is deployed.

I saw this database was created on today's SWAT - going to sanitize it on the sanitarium hosts before handing it over to cloud-services-team once SWAT is done

Marostegui triaged this task as Medium priority.Oct 5 2017, 12:46 PM

Mentioned in SAL (#wikimedia-operations) [2017-10-05T14:52:09Z] <marostegui> Sanitize db1095 for amwikimedia - T176043

Marostegui moved this task from Blocked external/Not db team to Done on the DBA board.

Sanitarium host was sanitized and check_private_data reported no issues. So this is ready for cloud-services-team to do their magic and resolve the ticket :-)

bd808 moved this task from Backlog to Wiki replicas on the Data-Services board.Oct 12 2017, 8:15 PM

Change 384545 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] data services: add amwikimedia to s3

https://gerrit.wikimedia.org/r/384545

Change 384545 merged by Andrew Bogott:
[operations/puppet@production] data services: add amwikimedia to s3

https://gerrit.wikimedia.org/r/384545

jcrespo claimed this task.Oct 17 2017, 2:39 PM
jcrespo raised the priority of this task from Medium to High.
jcrespo added a subscriber: Andrew.

@Ladsgroup There is a duplicate database on s7 called amwikimedia. I assume it is a duplicate, but we cannot just drop it, because it would drop the one on s3 on the multi-source hosts. Do you know how it ended up on s7? Having the same wiki on several shards can have unknown consequences, like replication breakage and backup failure (because the one on s7 overides the one on s3).

hmm, when at first I tried to make the wiki, due to lack of documentation I used "fawiki" instead of "aawiki" because the addWiki.php script should work with each wiki but it seems automagically it works with aawiki and breaks with other wikis which caused the breakage and we had to re-do lots of stuff. Since fawiki is in s7, I guess that's the reason. We added the note in the documentation after that this edit so this won't happen again.

Sorry, I phrased the "how" badly (I do not care much if there was a bug on the documentation/procedure). What I wanted to ask is- is the s7 database garbage? Can it be dropped (somehow)? Is the one on s3 the "real" one?

I was more into explaining that this was a one time thing and won't happen so we should not be worried about future cases, Yes, the s7 is garbage and should be thrown away. s3 is the correct shard (=default for new and small wikis).

jcrespo renamed this task from Prepare and check storage layer for amwikimedia to Prepare and check storage layer for amwikimedia (including dropping s7 version of the wiki).Oct 17 2017, 3:37 PM
jcrespo updated the task description. (Show Details)
Marostegui updated the task description. (Show Details)Oct 17 2017, 3:38 PM
jcrespo removed jcrespo as the assignee of this task.Nov 22 2017, 12:13 PM

Dropped amwikimedia from all s7 hosts (that were not multi-source hosts, as s3 still had it).

This is done from the DBA side @Andrew do you know if everything is done from the cloud team? If yes, this can be resolved, if not, this is fully on your side, nothing left to do from us.

jcrespo lowered the priority of this task from High to Medium.Nov 22 2017, 12:14 PM
bd808 closed this task as Resolved.Nov 30 2017, 10:31 PM
bd808 claimed this task.
$ sql amwikimedia
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 167746574
Server version: 10.0.22-MariaDB MariaDB Server

Copyright (c) 2000, 2017, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

(u3518@amwikimedia.labsdb) [amwikimedia_p]> show tables;
+-------------------------+
| Tables_in_amwikimedia_p |
+-------------------------+
| abuse_filter            |
| abuse_filter_action     |
| abuse_filter_log        |
| archive                 |
| archive_userindex       |
| babel                   |
| category                |
| categorylinks           |
| change_tag              |
| externallinks           |
| filearchive             |
| filearchive_userindex   |
| geo_tags                |
| global_block_whitelist  |
| globalblocks            |
| image                   |
| imagelinks              |
| interwiki               |
| ip_changes              |
| ipblocks                |
| ipblocks_ipindex        |
| iwlinks                 |
| l10n_cache              |
| langlinks               |
| linter                  |
| logging                 |
| logging_logindex        |
| logging_userindex       |
| math                    |
| module_deps             |
| oldimage                |
| oldimage_userindex      |
| page                    |
| page_props              |
| page_restrictions       |
| pagelinks               |
| protected_titles        |
| recentchanges           |
| recentchanges_userindex |
| redirect                |
| revision                |
| revision_userindex      |
| site_identifiers        |
| site_stats              |
| sites                   |
| tag_summary             |
| templatelinks           |
| transcode               |
| updatelog               |
| user                    |
| user_former_groups      |
| user_groups             |
| user_properties         |
| valid_tag               |
| wbc_entity_usage        |
+-------------------------+
55 rows in set (0.00 sec)

$ sql --cluster=web amwikimedia
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 38983010
Server version: 10.1.28-MariaDB MariaDB Server

Copyright (c) 2000, 2017, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

(u3518@amwikimedia.web.db.svc.eqiad.wmflabs) [amwikimedia_p]> show tables;
+-------------------------+
| Tables_in_amwikimedia_p |
+-------------------------+
| abuse_filter            |
| abuse_filter_action     |
| abuse_filter_log        |
| archive                 |
| archive_userindex       |
| babel                   |
| category                |
| categorylinks           |
| change_tag              |
| externallinks           |
| filearchive             |
| filearchive_userindex   |
| geo_tags                |
| global_block_whitelist  |
| globalblocks            |
| image                   |
| imagelinks              |
| interwiki               |
| ip_changes              |
| ipblocks                |
| ipblocks_ipindex        |
| iwlinks                 |
| l10n_cache              |
| langlinks               |
| linter                  |
| logging                 |
| logging_logindex        |
| logging_userindex       |
| math                    |
| module_deps             |
| oldimage                |
| oldimage_userindex      |
| page                    |
| page_props              |
| page_restrictions       |
| pagelinks               |
| protected_titles        |
| recentchanges           |
| recentchanges_userindex |
| redirect                |
| revision                |
| revision_userindex      |
| site_identifiers        |
| site_stats              |
| sites                   |
| tag_summary             |
| templatelinks           |
| transcode               |
| updatelog               |
| user                    |
| user_former_groups      |
| user_groups             |
| user_properties         |
| valid_tag               |
| wbc_entity_usage        |
+-------------------------+
55 rows in set (0.00 sec)

$ host amwikimedia.web.db.svc.eqiad.wmflabs
amwikimedia.web.db.svc.eqiad.wmflabs is an alias for s3.web.db.svc.eqiad.wmflabs.
s3.web.db.svc.eqiad.wmflabs has address 10.64.37.15
jcrespo reopened this task as Open.Jul 10 2018, 6:36 AM

amwikimedia is still on s7 in some places, at least dbstore2001. CC @Marostegui

jcrespo claimed this task.Jul 10 2018, 6:36 AM
jcrespo moved this task from Done to In progress on the DBA board.
jcrespo closed this task as Resolved.Jul 10 2018, 6:47 AM
jcrespo reassigned this task from jcrespo to bd808.

Only multisource hosts have now amwikimedia (because they also host s3):

 ./section s7 | while read host port; do ./mysql.py -BN -h $host:$port amwikimedia -e "SELECT @@GLOBAL.hostname, @@GLOBAL.port"; done
labsdb1011      3306
labsdb1010      3306
labsdb1009      3306
ERROR 1049 (42000): Unknown database 'amwikimedia'
dbstore1002     3306
ERROR 1049 (42000): Unknown database 'amwikimedia'
ERROR 1049 (42000): Unknown database 'amwikimedia'
ERROR 1049 (42000): Unknown database 'amwikimedia'
ERROR 1049 (42000): Unknown database 'amwikimedia'
ERROR 1049 (42000): Unknown database 'amwikimedia'
ERROR 1049 (42000): Unknown database 'amwikimedia'
ERROR 1049 (42000): Unknown database 'amwikimedia'
ERROR 1049 (42000): Unknown database 'amwikimedia'
ERROR 1049 (42000): Unknown database 'amwikimedia'
ERROR 1049 (42000): Unknown database 'amwikimedia'
ERROR 1049 (42000): Unknown database 'amwikimedia'
ERROR 1049 (42000): Unknown database 'amwikimedia'
ERROR 1049 (42000): Unknown database 'amwikimedia'
ERROR 1049 (42000): Unknown database 'amwikimedia'
ERROR 1049 (42000): Unknown database 'amwikimedia'
ERROR 1049 (42000): Unknown database 'amwikimedia'
ERROR 1049 (42000): Unknown database 'amwikimedia'