Page MenuHomePhabricator

Drop wb_terms in production from s4 (commonswiki, testcommonswiki), s3 (testwikidatawiki), s8 (wikidatawiki)
Closed, ResolvedPublic

Description

Creating this task to narrow the actionable for DBA instead of using T208425 which is the Epic

Following up the conversation on IRC with @Ladsgroup I have renamed wb_terms to T248086_wb_terms on the following hosts and wikis (all in codfw):

root@cumin1001:/home/marostegui# for i in db2106 db2109 db2081; do echo $i; mysql.py -h$i information_schema -e "select table_schema,table_name from tables where table_name like '%wb_terms';";done
db2106
+-----------------+------------------+
| table_schema    | table_name       |
+-----------------+------------------+
| commonswiki     | T248086_wb_terms |
| testcommonswiki | T248086_wb_terms |
+-----------------+------------------+
db2109
+------------------+------------------+
| table_schema     | table_name       |
+------------------+------------------+
| testwikidatawiki | T248086_wb_terms |
+------------------+------------------+
db2081
+--------------+------------------+
| table_schema | table_name       |
+--------------+------------------+
| wikidatawiki | T248086_wb_terms |
+--------------+------------------+

None of those hosts have slaves.
If there are any writes, replication will break and we'll notice.

If that happens, we just need to revert and restart replication with:

set session sql_log_bin=0; rename table  T248086_wb_terms to wb_terms; start slave;"

Let's give it 24h or so. Once we are fully sure nothing is writing, maybe we can move forward and rename the table on s3 (testwikidatawiki) eqiad host and monitor for errors.

The table must not be deleted on labs yet for s8 (wikidatawiki).

The table in s4: commonswiki, testcommonswiki is empty
The table in s3: wikidatawiki has around 500k rows

s4 (commonswiki, testcommonswiki). Empty table, to be dropped everywhere. To be run on the master with if exists ( (renamed everywhere)

  • master: db1138

s3 (wikidatawiki). Table with 500k rows, to be dropped everywhere, including labs and dbstore hosts

  • labsdb1012 (table renamed to: T248086_wb_terms)
  • labsdb1011 (table renamed to: T248086_wb_terms)
  • labsdb1010 (table renamed to: T248086_wb_terms)
  • labsdb1009 (table renamed to: T248086_wb_terms)
  • dbstore1004 (table renamed to: T248086_wb_terms)
  • db2127 (table renamed to: T248086_wb_terms)
  • db2109 (table renamed to: T248086_wb_terms)
  • db2105 (table renamed to: T248086_wb_terms)
  • db2098 (table renamed to: T248086_wb_terms)
  • db2094 (table renamed to: T248086_wb_terms)
  • db2074 (table renamed to: T248086_wb_terms)
  • db1124 (table renamed to: T248086_wb_terms)
  • db1112 (table renamed to: T248086_wb_terms)
  • db1095 (table renamed to: T248086_wb_terms)
  • db1078 (table renamed to: T248086_wb_terms)
  • db1075 (table renamed to: T248086_wb_terms)
  • db1123 (table renamed to: T248086_wb_terms)

s8 (wikidatawiki). To be dropped everywhere but NOT ON LABS/SANITARIUM/DBSTORE

  • labsdb1012 (table renamed to: T248086_wb_terms)
  • labsdb1011 (table renamed to: T248086_wb_terms)
  • labsdb1010 (table renamed to: T248086_wb_terms)
  • labsdb1009 (table renamed to: T248086_wb_terms)
  • dbstore1005 (table renamed to: T248086_wb_terms)
  • db2100 (table renamed to: T248086_wb_terms)
  • db2094 (table renamed to: T248086_wb_terms)
  • db2086 (table renamed to: T248086_wb_terms)
  • db2085 (table renamed to: T248086_wb_terms)
  • db2083 (table renamed to: T248086_wb_terms)
  • db2082 (table renamed to: T248086_wb_terms)
  • db2081 (table renamed to: T248086_wb_terms)
  • db2080 (table renamed to: T248086_wb_terms)
  • db2079 (table renamed to: T248086_wb_terms)
  • db1126 (table renamed to: T248086_wb_terms)
  • db1124 (table renamed to: T248086_wb_terms)
  • db1116 (table renamed to: T248086_wb_terms)
  • db1111 (table renamed to: T248086_wb_terms)
  • db1114 (table renamed to: T248086_wb_terms)
  • db1109 MASTER (table renamed to: T248086_wb_terms)
  • db1104 (table renamed to: T248086_wb_terms)
  • db1101 (table renamed to: T248086_wb_terms)
  • db1099 (table renamed to: T248086_wb_terms)
  • db1092 (table renamed to: T248086_wb_terms)
  • db1087 (table renamed to: T248086_wb_terms)

Related Objects

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 19 2020, 12:44 PM
Marostegui updated the task description. (Show Details)
Marostegui moved this task from Triage to In progress on the DBA board.

Mentioned in SAL (#wikimedia-operations) [2020-03-19T13:11:33Z] <marostegui> Rename testwikidatawiki.wb_terms on db1078 - T248086

Marostegui added a comment.EditedMar 19 2020, 1:12 PM

Table renamed on s3 (testwikidatawiki):

root@db1078.eqiad.wmnet[testwikidatawiki]> set session sql_log_bin=0; rename table wb_terms to T208425_wb_terms;
Query OK, 0 rows affected (0.00 sec)

Query OK, 0 rows affected (0.01 sec)

root@db1078.eqiad.wmnet[testwikidatawiki]>

To revert, just connect to that host and run:

use testwikidatawiki; set session sql_log_bin=0; rename table  T208425_wb_terms to wb_terms
Marostegui updated the task description. (Show Details)EditedMar 19 2020, 1:30 PM

Current stats

root@cumin1001:/home/marostegui# ./section s8 | grep eqiad | egrep -v "labs|dbstore" | while read host port; do echo $host:$port ; mysql.py -h$host:$port -e "SELECT rows_fetched FROM schema_table_statistics WHERE table_name='wb_terms';" sys; done
db1126.eqiad.wmnet:3306
rows_fetched
25585657798
db1124.eqiad.wmnet:3318
rows_fetched
2516079
db1116.eqiad.wmnet:3318
rows_fetched
5265689134
db1111.eqiad.wmnet:3306
rows_fetched
537159456
db1109.eqiad.wmnet:3306
rows_fetched
43745407490
db1104.eqiad.wmnet:3306
rows_fetched
51847033027
db1101.eqiad.wmnet:3318
rows_fetched
21146234326
db1099.eqiad.wmnet:3318
rows_fetched
20494170704
db1092.eqiad.wmnet:3306
rows_fetched
27418806752
db1087.eqiad.wmnet:3306
rows_fetched
3080068552

No errors for testwikidatawiki after yesterday's rename. Let's see if the whole weekend goes clean!

Current stats

root@cumin1001:/home/marostegui# ./section s8 | grep eqiad | egrep -v "labs|dbstore" | while read host port; do echo $host:$port ; mysql.py -h$host:$port -e "SELECT rows_fetched FROM schema_table_statistics WHERE table_name='wb_terms';" sys; done
db1126.eqiad.wmnet:3306
rows_fetched
25585657798
db1124.eqiad.wmnet:3318
rows_fetched
2516079
db1116.eqiad.wmnet:3318
rows_fetched
5265689134
db1111.eqiad.wmnet:3306
rows_fetched
537159456
db1109.eqiad.wmnet:3306
rows_fetched
43745407490
db1104.eqiad.wmnet:3306
rows_fetched
51847033027
db1101.eqiad.wmnet:3318
rows_fetched
21146234326
db1099.eqiad.wmnet:3318
rows_fetched
20494170704
db1092.eqiad.wmnet:3306
rows_fetched
27418806752
db1087.eqiad.wmnet:3306
rows_fetched
3080068552

Same results (db1109 got one row read, but I guess it could have been something manually triggered):

root@cumin1001:/home/marostegui# ./section s8 | grep eqiad | egrep -v "labs|dbstore" | while read host port; do echo $host:$port ; mysql.py -h$host:$port -e "SELECT rows_fetched FROM schema_table_statistics WHERE table_name='wb_terms';" sys; done
db1126.eqiad.wmnet:3306
rows_fetched
25585657798
db1124.eqiad.wmnet:3318
rows_fetched
2516079
db1116.eqiad.wmnet:3318
rows_fetched
5265689134
db1111.eqiad.wmnet:3306
rows_fetched
537159456
db1109.eqiad.wmnet:3306
rows_fetched
43745407491
db1104.eqiad.wmnet:3306
rows_fetched
51847033027
db1101.eqiad.wmnet:3318
rows_fetched
21146234326
db1099.eqiad.wmnet:3318
rows_fetched
20494170704
db1092.eqiad.wmnet:3306
rows_fetched
27418806752
db1087.eqiad.wmnet:3306
rows_fetched
3080068552

Also, no errors from testwikidatawiki on logstash for the last 4 days.
Should we go ahead and rename wikidatawiki.wb_terms on an eqiad host?

Marostegui updated the task description. (Show Details)Mar 23 2020, 9:44 AM

Mentioned in SAL (#wikimedia-operations) [2020-03-24T08:25:33Z] <marostegui> Rename wikidatawiki.wb_terms on db1104 - T248086

Table renamed on db1104 (main and api traffic)

root@db1104.eqiad.wmnet[wikidatawiki]> set session sql_log_bin=0; rename table wb_terms to T208425_wb_terms;
Query OK, 0 rows affected (0.00 sec)

Query OK, 0 rows affected (0.01 sec)

root@db1104.eqiad.wmnet[wikidatawiki]> show tables like 'T%';
+-----------------------------+
| Tables_in_wikidatawiki (T%) |
+-----------------------------+
| T208425_wb_terms            |
+-----------------------------+
1 row in set (0.00 sec)

To revert:

set session sql_log_bin=0; rename table  T208425_wb_terms to wb_terms
Marostegui updated the task description. (Show Details)Mar 24 2020, 8:27 AM

Overview of s8 in eqiad:

{
    "s8": {
      "api": {
        "db1092": 200,
        "db1104": 100
      },
      "contributions": {
        "db1099:3318": 100,
        "db1101:3318": 100
      },
      "dump": {
        "db1087": 100
      },
      "logpager": {
        "db1099:3318": 100,
        "db1101:3318": 100
      },
      "recentchanges": {
        "db1099:3318": 100,
        "db1101:3318": 100
      },
      "recentchangeslinked": {
        "db1099:3318": 100,
        "db1101:3318": 100
      },
      "vslow": {
        "db1087": 100
      },
      "watchlist": {
        "db1099:3318": 100,
        "db1101:3318": 100
      }
}

So the plan is to rename wb_terms in all replicas except the ones for labs (db1124 and anything it replicates to) on Monday. Does that sound good to you @WMDE-leszek ?

The generally sounds good. Labs should stay, as you said. What about the researcher replicas (i.e. dbstore1005) - this one will also be excluded, right?

What about the researcher replicas (i.e. dbstore1005) - this one will also be excluded, right?

Very valid question, I haven't heard from researchers since I posted it at March 19 (not even "I'm using it", total silence) so I guess it's fine to move forward with it as well.

For the record: @Ladsgroup is going to give a heads up to the researchers about the rename, and with that the renaming could also happen on dbstore1005

Mentioned in SAL (#wikimedia-operations) [2020-04-06T07:06:55Z] <marostegui> Rename wb_terms on codfw - T248086

The table has been renamed on codfw everywhere:
s3 - testwikidatawiki
s4 - commonswiki, testcommonswiki
s8 - wikidatawiki

I am going to proceed now with eqiad with the exception of labsdb1009, labsdb1010, labsdb1011 (wiki replicas), but including Analytics hosts: labsdb1012 and dbstore

Mentioned in SAL (#wikimedia-operations) [2020-04-06T07:35:26Z] <marostegui> Rename wb_terms on eqiad excluding labsdb1009, labdb1010, labsdb1011 - T248086

Marostegui updated the task description. (Show Details)Apr 6 2020, 7:41 AM

Tables renamed everywhere except: labsdb1009, labsdb1010, labsdb1011

s8:

root@cumin1001:/home/marostegui# ./section s8 | while read host port; do echo "$host:$port"; mysql.py -h $host:$port information_schema -e "select table_name, table_schema from tables where table_name like '%wb_terms%'";done
labsdb1012.eqiad.wmnet:3306
table_name	table_schema
T248086_wb_terms	commonswiki
wb_terms	commonswiki_p
T248086_wb_terms	testcommonswiki
wb_terms	testcommonswiki_p
T248086_wb_terms	testwikidatawiki
wb_terms	testwikidatawiki_p
T248086_wb_terms	wikidatawiki
wb_terms	wikidatawiki_p
wb_terms_no_longer_updated	wikidatawiki_p
labsdb1011.eqiad.wmnet:3306
table_name	table_schema
wb_terms	commonswiki
wb_terms	commonswiki_p
wb_terms	testcommonswiki
wb_terms	testcommonswiki_p
wb_terms	testwikidatawiki
wb_terms	testwikidatawiki_p
wb_terms	wikidatawiki
wb_terms	wikidatawiki_p
wb_terms_no_longer_updated	wikidatawiki_p
labsdb1010.eqiad.wmnet:3306
table_name	table_schema
wb_terms	commonswiki
wb_terms	commonswiki_p
wb_terms	testcommonswiki
wb_terms	testcommonswiki_p
wb_terms	testwikidatawiki
wb_terms	testwikidatawiki_p
wb_terms	wikidatawiki
wb_terms	wikidatawiki_p
wb_terms_no_longer_updated	wikidatawiki_p
labsdb1009.eqiad.wmnet:3306
table_name	table_schema
wb_terms	commonswiki
wb_terms	commonswiki_p
wb_terms	testcommonswiki
wb_terms	testcommonswiki_p
wb_terms	testwikidatawiki
wb_terms	testwikidatawiki_p
wb_terms	wikidatawiki
wb_terms	wikidatawiki_p
wb_terms_no_longer_updated	wikidatawiki_p
dbstore1005.eqiad.wmnet:3318
table_name	table_schema
T248086_wb_terms	wikidatawiki
db2100.codfw.wmnet:3318
table_name	table_schema
T248086_wb_terms	wikidatawiki
db2094.codfw.wmnet:3318
table_name	table_schema
T248086_wb_terms	wikidatawiki
db2086.codfw.wmnet:3318
table_name	table_schema
T248086_wb_terms	wikidatawiki
db2085.codfw.wmnet:3318
table_name	table_schema
T248086_wb_terms	wikidatawiki
db2083.codfw.wmnet:3306
table_name	table_schema
T248086_wb_terms	wikidatawiki
db2082.codfw.wmnet:3306
table_name	table_schema
T248086_wb_terms	wikidatawiki
db2081.codfw.wmnet:3306
table_name	table_schema
T248086_wb_terms	wikidatawiki
db2080.codfw.wmnet:3306
table_name	table_schema
T248086_wb_terms	wikidatawiki
db2079.codfw.wmnet:3306
table_name	table_schema
T248086_wb_terms	wikidatawiki
db1126.eqiad.wmnet:3306
table_name	table_schema
T248086_wb_terms	wikidatawiki
db1124.eqiad.wmnet:3318
table_name	table_schema
T248086_wb_terms	wikidatawiki
db1116.eqiad.wmnet:3318
table_name	table_schema
T248086_wb_terms	wikidatawiki
db1111.eqiad.wmnet:3306
table_name	table_schema
T248086_wb_terms	wikidatawiki
db1109.eqiad.wmnet:3306
table_name	table_schema
T248086_wb_terms	wikidatawiki
db1104.eqiad.wmnet:3306
table_name	table_schema
T248086_wb_terms	wikidatawiki
db1101.eqiad.wmnet:3318
table_name	table_schema
T248086_wb_terms	wikidatawiki
db1099.eqiad.wmnet:3318
table_name	table_schema
T248086_wb_terms	wikidatawiki
db1092.eqiad.wmnet:3306
table_name	table_schema
T248086_wb_terms	wikidatawiki
db1087.eqiad.wmnet:3306
table_name	table_schema
T248086_wb_terms	wikidatawiki

s4:

root@cumin1001:/home/marostegui# ./section s4 | while read host port; do echo "$host:$port"; mysql.py -h $host:$port information_schema -e "select table_name, table_schema from tables where table_name like '%wb_terms%'";done
labsdb1012.eqiad.wmnet:3306
table_name	table_schema
T248086_wb_terms	commonswiki
wb_terms	commonswiki_p
T248086_wb_terms	testcommonswiki
wb_terms	testcommonswiki_p
T248086_wb_terms	testwikidatawiki
wb_terms	testwikidatawiki_p
T248086_wb_terms	wikidatawiki
wb_terms	wikidatawiki_p
wb_terms_no_longer_updated	wikidatawiki_p
labsdb1011.eqiad.wmnet:3306
table_name	table_schema
wb_terms	commonswiki
wb_terms	commonswiki_p
wb_terms	testcommonswiki
wb_terms	testcommonswiki_p
wb_terms	testwikidatawiki
wb_terms	testwikidatawiki_p
wb_terms	wikidatawiki
wb_terms	wikidatawiki_p
wb_terms_no_longer_updated	wikidatawiki_p
labsdb1010.eqiad.wmnet:3306
table_name	table_schema
wb_terms	commonswiki
wb_terms	commonswiki_p
wb_terms	testcommonswiki
wb_terms	testcommonswiki_p
wb_terms	testwikidatawiki
wb_terms	testwikidatawiki_p
wb_terms	wikidatawiki
wb_terms	wikidatawiki_p
wb_terms_no_longer_updated	wikidatawiki_p
labsdb1009.eqiad.wmnet:3306
table_name	table_schema
wb_terms	commonswiki
wb_terms	commonswiki_p
wb_terms	testcommonswiki
wb_terms	testcommonswiki_p
wb_terms	testwikidatawiki
wb_terms	testwikidatawiki_p
wb_terms	wikidatawiki
wb_terms	wikidatawiki_p
wb_terms_no_longer_updated	wikidatawiki_p
dbstore1004.eqiad.wmnet:3314
table_name	table_schema
T248086_wb_terms	commonswiki
T248086_wb_terms	testcommonswiki
db2119.codfw.wmnet:3306
table_name	table_schema
T248086_wb_terms	commonswiki
T248086_wb_terms	testcommonswiki
db2110.codfw.wmnet:3306
table_name	table_schema
T248086_wb_terms	commonswiki
T248086_wb_terms	testcommonswiki
db2106.codfw.wmnet:3306
table_name	table_schema
T248086_wb_terms	commonswiki
T248086_wb_terms	testcommonswiki
db2099.codfw.wmnet:3314
table_name	table_schema
T248086_wb_terms	commonswiki
T248086_wb_terms	testcommonswiki
db2095.codfw.wmnet:3314
table_name	table_schema
T248086_wb_terms	commonswiki
T248086_wb_terms	testcommonswiki
db2091.codfw.wmnet:3314
table_name	table_schema
T248086_wb_terms	commonswiki
T248086_wb_terms	testcommonswiki
db2090.codfw.wmnet:3306
table_name	table_schema
T248086_wb_terms	commonswiki
T248086_wb_terms	testcommonswiki
db2084.codfw.wmnet:3314
table_name	table_schema
T248086_wb_terms	commonswiki
T248086_wb_terms	testcommonswiki
db2073.codfw.wmnet:3306
table_name	table_schema
T248086_wb_terms	commonswiki
T248086_wb_terms	testcommonswiki
db1138.eqiad.wmnet:3306
table_name	table_schema
T248086_wb_terms	commonswiki
T248086_wb_terms	testcommonswiki
db1125.eqiad.wmnet:3314
table_name	table_schema
T248086_wb_terms	commonswiki
T248086_wb_terms	testcommonswiki
db1121.eqiad.wmnet:3306
table_name	table_schema
T248086_wb_terms	commonswiki
T248086_wb_terms	testcommonswiki
db1103.eqiad.wmnet:3314
table_name	table_schema
T248086_wb_terms	commonswiki
T248086_wb_terms	testcommonswiki
db1102.eqiad.wmnet:3314
table_name	table_schema
T248086_wb_terms	commonswiki
T248086_wb_terms	testcommonswiki
db1097.eqiad.wmnet:3314
table_name	table_schema
T248086_wb_terms	commonswiki
T248086_wb_terms	testcommonswiki
db1091.eqiad.wmnet:3306
table_name	table_schema
T248086_wb_terms	commonswiki
T248086_wb_terms	testcommonswiki
db1084.eqiad.wmnet:3306
table_name	table_schema
T248086_wb_terms	commonswiki
T248086_wb_terms	testcommonswiki
db1081.eqiad.wmnet:3306
table_name	table_schema
T248086_wb_terms	commonswiki
T248086_wb_terms	testcommonswiki

s3:

root@cumin1001:/home/marostegui# ./section s3 | while read host port; do echo "$host:$port"; mysql.py -h $host:$port information_schema -e "select table_name, table_schema from tables where table_name like '%wb_terms%'";done
labsdb1012.eqiad.wmnet:3306
table_name	table_schema
T248086_wb_terms	commonswiki
wb_terms	commonswiki_p
T248086_wb_terms	testcommonswiki
wb_terms	testcommonswiki_p
T248086_wb_terms	testwikidatawiki
wb_terms	testwikidatawiki_p
T248086_wb_terms	wikidatawiki
wb_terms	wikidatawiki_p
wb_terms_no_longer_updated	wikidatawiki_p
labsdb1011.eqiad.wmnet:3306
table_name	table_schema
wb_terms	commonswiki
wb_terms	commonswiki_p
wb_terms	testcommonswiki
wb_terms	testcommonswiki_p
wb_terms	testwikidatawiki
wb_terms	testwikidatawiki_p
wb_terms	wikidatawiki
wb_terms	wikidatawiki_p
wb_terms_no_longer_updated	wikidatawiki_p
labsdb1010.eqiad.wmnet:3306
table_name	table_schema
wb_terms	commonswiki
wb_terms	commonswiki_p
wb_terms	testcommonswiki
wb_terms	testcommonswiki_p
wb_terms	testwikidatawiki
wb_terms	testwikidatawiki_p
wb_terms	wikidatawiki
wb_terms	wikidatawiki_p
wb_terms_no_longer_updated	wikidatawiki_p
labsdb1009.eqiad.wmnet:3306
table_name	table_schema
wb_terms	commonswiki
wb_terms	commonswiki_p
wb_terms	testcommonswiki
wb_terms	testcommonswiki_p
wb_terms	testwikidatawiki
wb_terms	testwikidatawiki_p
wb_terms	wikidatawiki
wb_terms	wikidatawiki_p
wb_terms_no_longer_updated	wikidatawiki_p
dbstore1004.eqiad.wmnet:3313
table_name	table_schema
T248086_wb_terms	testwikidatawiki
db2127.codfw.wmnet:3306
table_name	table_schema
T248086_wb_terms	testwikidatawiki
db2109.codfw.wmnet:3306
table_name	table_schema
T248086_wb_terms	testwikidatawiki
db2105.codfw.wmnet:3306
table_name	table_schema
T248086_wb_terms	testwikidatawiki
db2098.codfw.wmnet:3313
table_name	table_schema
T248086_wb_terms	testwikidatawiki
db2094.codfw.wmnet:3313
table_name	table_schema
T248086_wb_terms	testwikidatawiki
db2074.codfw.wmnet:3306
table_name	table_schema
T248086_wb_terms	testwikidatawiki
db1124.eqiad.wmnet:3313
table_name	table_schema
T248086_wb_terms	testwikidatawiki
db1123.eqiad.wmnet:3306
table_name	table_schema
T248086_wb_terms	testwikidatawiki
db1112.eqiad.wmnet:3306
table_name	table_schema
T248086_wb_terms	testwikidatawiki
db1095.eqiad.wmnet:3313
table_name	table_schema
T248086_wb_terms	testwikidatawiki
db1078.eqiad.wmnet:3306
table_name	table_schema
T248086_wb_terms	testwikidatawiki
db1075.eqiad.wmnet:3306
table_name	table_schema
T248086_wb_terms	testwikidatawiki
Marostegui updated the task description. (Show Details)Apr 6 2020, 7:58 AM

Mentioned in SAL (#wikimedia-operations) [2020-04-08T08:46:21Z] <marostegui> Rename wb_terms and recreate views on labsdb1009-labsdb1011 - T248592 T248086

Marostegui updated the task description. (Show Details)Apr 8 2020, 9:20 AM
Marostegui updated the task description. (Show Details)Apr 8 2020, 10:09 AM

@Ladsgroup @Addshore let's start this on Monday 27th?

@Ladsgroup @Addshore let's start this on Monday 27th?

Fine by me!

Also sounds good to me!

Mentioned in SAL (#wikimedia-operations) [2020-04-27T12:14:59Z] <marostegui> Remove empty table T248086_wb_terms from commonswiki and testcommonswiki on s4 master - T248086

Table gone in s4 (commonswiki and testcommonswiki):

root@cumin1001:/home/marostegui# ./section s4 | grep -v labs | while read host port; do echo "$host:$port"; mysql.py -h$host:$port information_schema -e "select table_schema from tables where table_name like '%wb_terms%'";done
dbstore1004.eqiad.wmnet:3314
db2119.codfw.wmnet:3306
db2110.codfw.wmnet:3306
db2106.codfw.wmnet:3306
db2099.codfw.wmnet:3314
db2095.codfw.wmnet:3314
db2091.codfw.wmnet:3314
db2090.codfw.wmnet:3306
db2084.codfw.wmnet:3314
db2073.codfw.wmnet:3306
db1138.eqiad.wmnet:3306
db1125.eqiad.wmnet:3314
db1121.eqiad.wmnet:3306
db1103.eqiad.wmnet:3314
db1102.eqiad.wmnet:3314
db1097.eqiad.wmnet:3314
db1091.eqiad.wmnet:3306
db1084.eqiad.wmnet:3306
db1081.eqiad.wmnet:3306
Marostegui updated the task description. (Show Details)Apr 27 2020, 12:17 PM

Mentioned in SAL (#wikimedia-operations) [2020-04-27T12:32:29Z] <marostegui> Remove empty table T248086_wb_terms from wikidatawiki on s8 codfw master - T248086

Marostegui updated the task description. (Show Details)Apr 27 2020, 12:36 PM

Mentioned in SAL (#wikimedia-operations) [2020-04-27T12:36:43Z] <marostegui> Remove empty table T248086_wb_terms from wikidatawiki on s3 codfw master - T248086

Marostegui updated the task description. (Show Details)Apr 27 2020, 12:37 PM

Mentioned in SAL (#wikimedia-operations) [2020-04-27T12:41:27Z] <marostegui> Remove empty table T248086_wb_terms from wikidatawiki on s3 eqiad - T248086

Table gone in s3 (testwikidatawiki):

root@cumin1001:/home/marostegui# ./section s3 | grep -v labs | while read host port; do echo "$host:$port"; mysql.py -h$host:$port information_schema -e "select table_schema from tables where table_name like '%wb_terms%'";done
dbstore1004.eqiad.wmnet:3313
db2127.codfw.wmnet:3306
db2109.codfw.wmnet:3306
db2105.codfw.wmnet:3306
db2098.codfw.wmnet:3313
db2094.codfw.wmnet:3313
db2074.codfw.wmnet:3306
db1124.eqiad.wmnet:3313
db1123.eqiad.wmnet:3306
db1112.eqiad.wmnet:3306
db1095.eqiad.wmnet:3313
db1078.eqiad.wmnet:3306
db1075.eqiad.wmnet:3306
Marostegui updated the task description. (Show Details)Apr 27 2020, 12:43 PM

Mentioned in SAL (#wikimedia-operations) [2020-04-27T12:53:43Z] <marostegui> Drop T248086_wb_terms from db1104 - T248086

The effect of dropping the table on s8 (wikidatawiki) master:

1root@db2079:/srv/sqldata# du -sh .
21.9T .
3root@db2079:/srv/sqldata# mysql -A wikidatawiki
4Welcome to the MariaDB monitor. Commands end with ; or \g.
5Your MariaDB connection id is 213591
6Server version: 10.1.44-MariaDB MariaDB Server
7
8Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
9
10Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
11
12root@PRODUCTION s8 slave[wikidatawiki]> drop table if exists T248086_wb_terms;
13Query OK, 0 rows affected (20.86 sec)
14
15root@PRODUCTION s8 slave[wikidatawiki]> show tables like '%wb_terms%';
16Empty set (0.00 sec)
17
18root@PRODUCTION s8 slave[wikidatawiki]> Ctrl-C -- exit!
19Aborted
20root@db2079:/srv/sqldata# du -sh .
211.3T .

Marostegui updated the task description. (Show Details)Apr 27 2020, 12:56 PM
Marostegui updated the task description. (Show Details)Apr 27 2020, 2:02 PM
Marostegui updated the task description. (Show Details)Apr 28 2020, 6:12 AM
Marostegui updated the task description. (Show Details)Apr 29 2020, 6:03 AM
Marostegui updated the task description. (Show Details)Apr 30 2020, 5:15 AM
Marostegui updated the task description. (Show Details)Apr 30 2020, 5:30 AM
Marostegui updated the task description. (Show Details)Apr 30 2020, 6:51 AM
Marostegui updated the task description. (Show Details)Apr 30 2020, 1:53 PM

Mentioned in SAL (#wikimedia-operations) [2020-05-05T10:44:42Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1126 T248086', diff saved to https://phabricator.wikimedia.org/P11139 and previous config saved to /var/cache/conftool/dbconfig/20200505-104441-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2020-05-05T10:45:41Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1126 T248086', diff saved to https://phabricator.wikimedia.org/P11140 and previous config saved to /var/cache/conftool/dbconfig/20200505-104540-marostegui.json

Marostegui updated the task description. (Show Details)Tue, May 5, 10:45 AM

Mentioned in SAL (#wikimedia-operations) [2020-05-05T11:30:45Z] <marostegui> Drop T248086_wb_terms table on labsdb hosts - T248086

Mentioned in SAL (#wikimedia-operations) [2020-05-05T11:31:01Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1087 T248086', diff saved to https://phabricator.wikimedia.org/P11143 and previous config saved to /var/cache/conftool/dbconfig/20200505-113100-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2020-05-05T11:31:53Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1087 T248086', diff saved to https://phabricator.wikimedia.org/P11144 and previous config saved to /var/cache/conftool/dbconfig/20200505-113152-marostegui.json

Marostegui updated the task description. (Show Details)Tue, May 5, 11:33 AM
Marostegui changed the status of subtask T251598: Clean up wb_terms related views from Stalled to Open.

As we have seen the drop creating contention on the slaves (to the point that they need to be depooled) I will drop this table during the read-only time that will happen with T251981

Marostegui triaged this task as Medium priority.Wed, May 6, 7:42 AM
Marostegui closed this task as Resolved.Tue, May 19, 5:07 AM
Marostegui updated the task description. (Show Details)

This has been dropped everywhere! \o/