Page MenuHomePhabricator

Remove muswiki and mhwiktionary from s3
Closed, ResolvedPublic

Description

muswiki and mhwiktionary are two closed wikis that were moved from s3 to s5 at T259004.
We need to get them removed from s3.

Let's first rename the tables on the master and leave it like that for a few days before actually going ahead and dropping the databases.
It needs to be done without replication

Event Timeline

Mentioned in SAL (#wikimedia-operations) [2020-08-11T09:11:28Z] <marostegui> Rename tables on muswiki and mhwiktionary on s3 master (db1123) without replication T260112

I have renamed the tables on s3 master only (db1123) for muswiki and mhwiktionary.
If any writes attempts to happen, it will fail. Let's leave this for a few days while we monitor logstash for any issues.

This is how they look like now:

# mysql.py -hdb1123 -e "show tables" muswiki
+----------------------------------+
| Tables_in_muswiki                |
+----------------------------------+
| T260112_abuse_filter             |
| T260112_abuse_filter_action      |
| T260112_abuse_filter_history     |
| T260112_abuse_filter_log         |
| T260112_actor                    |
| T260112_archive                  |
| T260112_babel                    |
| T260112_betafeatures_user_counts |
| T260112_blobs                    |
| T260112_bv2009_edits             |
| T260112_bv2011_edits             |
| T260112_bv2013_edits             |
| T260112_bv2015_edits             |
| T260112_bv2017_edits             |
| T260112_category                 |
| T260112_categorylinks            |
| T260112_change_tag               |
| T260112_change_tag_def           |
| T260112_comment                  |
| T260112_content                  |
| T260112_content_models           |
| T260112_cu_changes               |
| T260112_cu_log                   |
| T260112_cur                      |
| T260112_externallinks            |
| T260112_filearchive              |
| T260112_geo_tags                 |
| T260112_global_block_whitelist   |
| T260112_hidden                   |
| T260112_image                    |
| T260112_imagelinks               |
| T260112_interwiki                |
| T260112_ip_changes               |
| T260112_ipblocks                 |
| T260112_ipblocks_restrictions    |
| T260112_iwlinks                  |
| T260112_job                      |
| T260112_l10n_cache               |
| T260112_langlinks                |
| T260112_linter                   |
| T260112_log_search               |
| T260112_logging                  |
| T260112_mathoid                  |
| T260112_module_deps              |
| T260112_objectcache              |
| T260112_oldimage                 |
| T260112_page                     |
| T260112_page_props               |
| T260112_page_restrictions        |
| T260112_pagelinks                |
| T260112_pif_edits                |
| T260112_povwatch_log             |
| T260112_povwatch_subscribers     |
| T260112_protected_titles         |
| T260112_querycache               |
| T260112_querycache_info          |
| T260112_querycachetwo            |
| T260112_recentchanges            |
| T260112_redirect                 |
| T260112_revision                 |
| T260112_revision_actor_temp      |
| T260112_revision_comment_temp    |
| T260112_searchindex              |
| T260112_securepoll_cookie_match  |
| T260112_securepoll_elections     |
| T260112_securepoll_entity        |
| T260112_securepoll_lists         |
| T260112_securepoll_msgs          |
| T260112_securepoll_options       |
| T260112_securepoll_properties    |
| T260112_securepoll_questions     |
| T260112_securepoll_strike        |
| T260112_securepoll_voters        |
| T260112_securepoll_votes         |
| T260112_site_identifiers         |
| T260112_site_stats               |
| T260112_sites                    |
| T260112_slot_roles               |
| T260112_slots                    |
| T260112_spoofuser                |
| T260112_templatelinks            |
| T260112_text                     |
| T260112_transcache               |
| T260112_transcode                |
| T260112_updatelog                |
| T260112_uploadstash              |
| T260112_user                     |
| T260112_user_former_groups       |
| T260112_user_groups              |
| T260112_user_newtalk             |
| T260112_user_properties          |
| T260112_watchlist                |
| T260112_watchlist_expiry         |
| T260112_wbc_entity_usage         |
+----------------------------------+

# mysql.py -hdb1123 -e "show tables" mhwiktionary
+----------------------------------+
| Tables_in_mhwiktionary           |
+----------------------------------+
| T260112_abuse_filter             |
| T260112_abuse_filter_action      |
| T260112_abuse_filter_history     |
| T260112_abuse_filter_log         |
| T260112_actor                    |
| T260112_archive                  |
| T260112_babel                    |
| T260112_betafeatures_user_counts |
| T260112_blobs                    |
| T260112_bv2009_edits             |
| T260112_bv2011_edits             |
| T260112_bv2013_edits             |
| T260112_bv2015_edits             |
| T260112_bv2017_edits             |
| T260112_category                 |
| T260112_categorylinks            |
| T260112_change_tag               |
| T260112_change_tag_def           |
| T260112_comment                  |
| T260112_content                  |
| T260112_content_models           |
| T260112_cu_changes               |
| T260112_cu_log                   |
| T260112_cur                      |
| T260112_externallinks            |
| T260112_filearchive              |
| T260112_geo_tags                 |
| T260112_global_block_whitelist   |
| T260112_hidden                   |
| T260112_image                    |
| T260112_imagelinks               |
| T260112_interwiki                |
| T260112_ip_changes               |
| T260112_ipblocks                 |
| T260112_ipblocks_restrictions    |
| T260112_iwlinks                  |
| T260112_job                      |
| T260112_l10n_cache               |
| T260112_langlinks                |
| T260112_linter                   |
| T260112_log_search               |
| T260112_logging                  |
| T260112_mathoid                  |
| T260112_module_deps              |
| T260112_objectcache              |
| T260112_oldimage                 |
| T260112_page                     |
| T260112_page_props               |
| T260112_page_restrictions        |
| T260112_pagelinks                |
| T260112_pif_edits                |
| T260112_povwatch_log             |
| T260112_povwatch_subscribers     |
| T260112_protected_titles         |
| T260112_querycache               |
| T260112_querycache_info          |
| T260112_querycachetwo            |
| T260112_recentchanges            |
| T260112_redirect                 |
| T260112_revision                 |
| T260112_revision_actor_temp      |
| T260112_revision_comment_temp    |
| T260112_searchindex              |
| T260112_securepoll_cookie_match  |
| T260112_securepoll_elections     |
| T260112_securepoll_entity        |
| T260112_securepoll_lists         |
| T260112_securepoll_msgs          |
| T260112_securepoll_options       |
| T260112_securepoll_properties    |
| T260112_securepoll_questions     |
| T260112_securepoll_strike        |
| T260112_securepoll_voters        |
| T260112_securepoll_votes         |
| T260112_site_identifiers         |
| T260112_site_stats               |
| T260112_sites                    |
| T260112_slot_roles               |
| T260112_slots                    |
| T260112_spoofuser                |
| T260112_templatelinks            |
| T260112_text                     |
| T260112_transcache               |
| T260112_transcode                |
| T260112_updatelog                |
| T260112_uploadstash              |
| T260112_user                     |
| T260112_user_former_groups       |
| T260112_user_groups              |
| T260112_user_newtalk             |
| T260112_user_properties          |
| T260112_watchlist                |
| T260112_watchlist_expiry         |
| T260112_wbc_entity_usage         |
+----------------------------------+

No errors so far, for the last 2 days.
If all continues like this, on Monday I will remove the tables and the database from s3.

No errors so far, for the last 2 days.
If all continues like this, on Monday I will remove the tables and the database from s3.

Did that happen? :-)

No, I am waiting for eqiad to be back as active

Just to confirm, this can now happen and wikis, as expected, have no writes on s3.

root@db1075:/srv/sqldata/muswiki# ls -lhrt *.ibd | tail -n 2
-rw-rw---- 1 mysql mysql 9.0M Aug 11 08:32 logging.ibd
-rw-rw---- 1 mysql mysql 512K Aug 11 09:49 user_properties.ibd

root@db1075:/srv/sqldata/mhwiktionary# ls -lhrt *.ibd | tail -n 2
-rw-rw---- 1 mysql mysql 176K Aug 11 09:49 cu_changes.ibd
-rw-rw---- 1 mysql mysql 9.0M Aug 11 09:49 logging.ibd

We need to drop the databases without replication on each host.

Marostegui moved this task from Ready to In progress on the DBA board.EditedNov 4 2020, 8:15 AM

Deletion progress:
NOT NEEDED labsdb1012
NOT NEEDED labsdb1011
NOT NEEDED labsdb1010
NOT NEEDED labsdb1009

  • dbstore1004:3313
  • db2127
  • db2109
  • db2105
  • db2098:3313
  • db2094:3313
  • db2074
  • db1124:3313
  • db1123
  • db1112
  • db1095
  • db1078
  • db1075

This has been cleaned up on codfw.
Waiting a few hours before going for eqiad.

This is done:

# ./section s3 | while read host port; do echo "$host:$port"; mysql.py -h$host:$port -e "show databases like 'muswiki'; show databases like 'mhwiktionary'";done
labsdb1012.eqiad.wmnet:3306
Database (muswiki)
muswiki
Database (mhwiktionary)
mhwiktionary
labsdb1011.eqiad.wmnet:3306
Database (muswiki)
muswiki
Database (mhwiktionary)
mhwiktionary
labsdb1010.eqiad.wmnet:3306
Database (muswiki)
muswiki
Database (mhwiktionary)
mhwiktionary
labsdb1009.eqiad.wmnet:3306
Database (muswiki)
muswiki
Database (mhwiktionary)
mhwiktionary
dbstore1004.eqiad.wmnet:3313
db2127.codfw.wmnet:3306
db2109.codfw.wmnet:3306
db2105.codfw.wmnet:3306
db2098.codfw.wmnet:3313
db2094.codfw.wmnet:3313
db2074.codfw.wmnet:3306
db1124.eqiad.wmnet:3313
db1123.eqiad.wmnet:3306
db1112.eqiad.wmnet:3306
db1095.eqiad.wmnet:3313
db1078.eqiad.wmnet:3306
db1075.eqiad.wmnet:3306