Page MenuHomePhabricator

Drop revision_actor_temp in production
Closed, ResolvedPublic

Description

We have stopped writing to this table in most wikis and there is no fireworks so far. We should slowly start dropping the table.

I suggest first dropping it in mediawikiwiki (and one host in codfw only, then one host in eqiad, etc.).

I will write here once we stop writing to it everywhere (very likely tomorrow)

Progress

  • s1 (renamed - dropped)
  • s2 (renamed - dropped)
  • s3 (renamed - dropped)
  • s4 (renamed - dropped)
  • s5 (renamed - dropped)
  • s6 (renamed - dropped)
  • s7 (renamed - dropped)
  • s8 (renamed - dropped)
  • labtestwiki

Event Timeline

Marostegui triaged this task as High priority.
Marostegui moved this task from Triage to In progress on the DBA board.

I will take care of this dangerous task :)

Going to rename this on db2109 (mediawikiwiki) where the table hasn't been written for days:

root@db2109:~# ls -lh /srv/sqldata/mediawikiwiki/revision_actor_temp.ibd
-rw-r----- 1 mysql mysql 432M May  5 18:38 /srv/sqldata/mediawikiwiki/revision_actor_temp.ibd
mysql:root@localhost [mediawikiwiki]> set session sql_log_bin=0; rename table revision_actor_temp to T307906_revision_actor_temp;
Query OK, 0 rows affected (0.000 sec)

Query OK, 0 rows affected (0.006 sec)

mysql:root@localhost [mediawikiwiki]>

Mentioned in SAL (#wikimedia-operations) [2022-05-10T05:18:36Z] <marostegui> Rename revision_actor_temp on db2109 T307906

Update: We stopped writing to it everywhere now.

Excellent, I am going to rename it on an enwiki codfw host then

Mentioned in SAL (#wikimedia-operations) [2022-05-10T08:48:28Z] <marostegui> Rename revision_actor_temp on db2092 T307906

root@db2092.codfw.wmnet[enwiki]> set session sql_log_bin=0; rename table revision_actor_temp to T307906_revision_actor_temp;
Query OK, 0 rows affected (0.032 sec)

Query OK, 0 rows affected (0.035 sec)

root@db2092.codfw.wmnet[enwiki]>

Just checked all the masters:
s1 last write: May 10 08:19
s2 last write: May 10 08:22
s3 last write: May 10 08:09
s4 last write: May 9 04:50
s5 last write: May 10 10:37
s6 last write: May 10 08:57
s7 last write: May 10 09:02
s8 last write: May 9 05:48

Mentioned in SAL (#wikimedia-operations) [2022-05-11T08:12:57Z] <marostegui> Rename revision_actor_temp on db1132 (s1) and db1114 (s8) T307906

I have renamed the table on db1132 (s1) and db1114 (s8). If there are any reads we will see them failing on logstash:

root@db1114.eqiad.wmnet[wikidatawiki]> set session sql_log_bin=0; rename table revision_actor_temp to T307906_revision_actor_temp;
Query OK, 0 rows affected (0.000 sec)

Query OK, 0 rows affected (0.005 sec)
root@db1132.eqiad.wmnet[enwiki]> set session sql_log_bin=0; rename table revision_actor_temp to T307906_revision_actor_temp;
Query OK, 0 rows affected (0.000 sec)

Query OK, 0 rows affected (0.023 sec)
Marostegui updated the task description. (Show Details)
Marostegui updated the task description. (Show Details)

There were no errors on the hosts were the table was renamed, so I am going to go ahead and rename it everywhere in s6.

Mentioned in SAL (#wikimedia-operations) [2022-05-24T05:11:11Z] <marostegui> Rename revision_actor_temp on s6 T307906

Dropped in labtestwiki

mysql:root@localhost [labtestwiki]> drop table revision_actor_temp;
Query OK, 0 rows affected (0.076 sec)

Mentioned in SAL (#wikimedia-operations) [2022-05-24T06:08:09Z] <marostegui> Rename revision_actor_temp on s8 T307906

Mentioned in SAL (#wikimedia-operations) [2022-05-25T04:57:04Z] <marostegui> Rename revision_actor_temp on s4 T307906

Renamed on s4 (testcommonswiki and commonswiki)

Mentioned in SAL (#wikimedia-operations) [2022-05-25T05:03:12Z] <marostegui> Rename revision_actor_temp on s2 T307906

Mentioned in SAL (#wikimedia-operations) [2022-05-25T05:14:49Z] <marostegui> Rename revision_actor_temp on s7 T307906

Mentioned in SAL (#wikimedia-operations) [2022-05-25T05:18:41Z] <marostegui> Rename revision_actor_temp on s5 T307906

Mentioned in SAL (#wikimedia-operations) [2022-05-25T05:30:44Z] <marostegui> Rename revision_actor_temp on s1 T307906

Starting to rename it across s3, it will take around 8 hours to finish.

Starting to rename it across s3, it will take around 8 hours to finish.

This finished - I will drop it for good next week.

I have seen no errors on logstash so I am going to start proceeding with the deletion of the renamed table.

Mentioned in SAL (#wikimedia-operations) [2022-05-30T05:26:23Z] <marostegui> Drop renamed revision_actor_temp on s6 T307906

Mentioned in SAL (#wikimedia-operations) [2022-05-30T05:28:42Z] <marostegui> Drop renamed revision_actor_temp on s2 T307906

Mentioned in SAL (#wikimedia-operations) [2022-05-30T06:01:14Z] <marostegui> Drop renamed revision_actor_temp on s7 T307906

Mentioned in SAL (#wikimedia-operations) [2022-05-30T06:10:20Z] <marostegui> Drop renamed revision_actor_temp on s5 T307906

Mentioned in SAL (#wikimedia-operations) [2022-05-30T06:36:42Z] <marostegui> Drop renamed revision_actor_temp on s8 T307906

Mentioned in SAL (#wikimedia-operations) [2022-05-30T06:39:08Z] <marostegui> Drop renamed revision_actor_temp on s4 T307906

Mentioned in SAL (#wikimedia-operations) [2022-05-30T12:34:13Z] <marostegui> Drop renamed revision_actor_temp on s3 T307906

Started the drop on s3's master, it will take around 8h.

Started the drop on s3's master, it will take around 8h.

Done

Marostegui updated the task description. (Show Details)

This is all done