Drop m3 from dbstore servers
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Marostegui
	Jan 31 2017, 9:29 AM

Description

Hello,

Due to the phabricator update (T156373) we experienced a crash on all the m3 replicas because of a MariaDB bug (T156373#2973202 T156373#2973962).
We were lucky that the master and the eqiad dbstores were running an old non affected version of Mariadb, so they survived the crash.

However, we have started to wonder if it is really necessary to have m3 on the dbstores servers, specially on dbstore1002 (which is the non delayed replica).
We assume it is not needed at all on codfw and we are going to drop it from there, but we'd like to know if m3 in dbstore1002 is being used or it can be dropped there too.
Any thoughts from Analytics team?

Thanks!

Related Objects

Mentioned In: T168303: dbstore1002 /srv filling up
T156373: During Phabricator upgrade on 2017-01-26, all m3 replica dbs crashed at the same time
Mentioned Here: T156905: Phabricator master and slave crashed
T156373: During Phabricator upgrade on 2017-01-26, all m3 replica dbs crashed at the same time

Event Timeline

Marostegui created this task.Jan 31 2017, 9:29 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 31 2017, 9:29 AM

Marostegui moved this task from Triage to Backlog on the DBA board.Jan 31 2017, 9:29 AM

Marostegui mentioned this in T156373: During Phabricator upgrade on 2017-01-26, all m3 replica dbs crashed at the same time.Feb 1 2017, 6:47 AM

m3 has been removed from dbstore2001 as per: T156905#2991826

Paladox subscribed.Feb 2 2017, 12:23 PM

We don't know what m3 is. So far we don't know of a use for Phabricator databases on the analytics slaves.

Milimetric removed a project: Analytics.Feb 2 2017, 5:21 PM

Mentioned in SAL (#wikimedia-operations) [2017-08-09T20:51:38Z] <marostegui> Remove m3 replication from dbstore1002 - T156758

I have removed replication from 'm3' thread on dbstore1002 and these are the values:

root@DBSTORE[(none)]> select @@hostname;
+-------------+
| @@hostname  |
+-------------+
| dbstore1002 |
+-------------+
1 row in set (0.00 sec)

root@DBSTORE[(none)]> show slave status\G
*************************** 1. row ***************************
               Slave_IO_State:
                  Master_Host: db1043.eqiad.wmnet
                  Master_User: repl
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: db1043-bin.001752
          Read_Master_Log_Pos: 569268565
               Relay_Log_File: dbstore1002-relay-bin-m3.000744
                Relay_Log_Pos: 569268853
        Relay_Master_Log_File: db1043-bin.001752
             Slave_IO_Running: No
            Slave_SQL_Running: No
              Replicate_Do_DB:
          Replicate_Ignore_DB:
           Replicate_Do_Table:
       Replicate_Ignore_Table:
      Replicate_Wild_Do_Table: phab%.%,phlegal%.%,heartbeat.%
  Replicate_Wild_Ignore_Table:
                   Last_Errno: 0
                   Last_Error:
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 569268565
              Relay_Log_Space: 569269203
              Until_Condition: None
               Until_Log_File:
                Until_Log_Pos: 0
           Master_SSL_Allowed: Yes
           Master_SSL_CA_File:
           Master_SSL_CA_Path:
              Master_SSL_Cert:
            Master_SSL_Cipher:
               Master_SSL_Key:
        Seconds_Behind_Master: NULL
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error:
               Last_SQL_Errno: 0
               Last_SQL_Error:
  Replicate_Ignore_Server_Ids:
             Master_Server_Id: 171970592
               Master_SSL_Crl:
           Master_SSL_Crlpath:
                   Using_Gtid: No
                  Gtid_IO_Pos:
1 row in set (0.00 sec)

root@DBSTORE[(none)]> reset slave all;
Query OK, 0 rows affected (0.09 sec)

If there are no complains after this, I will remove all the m3 content from dbstore1002

Marostegui claimed this task.Aug 9 2017, 8:55 PM

Marostegui moved this task from Backlog to In progress on the DBA board.

Marostegui mentioned this in T168303: dbstore1002 /srv filling up.Aug 9 2017, 8:58 PM

Mentioned in SAL (#wikimedia-operations) [2017-08-10T13:17:49Z] <marostegui> Drop m3 databases from dbstore1002 - T156758

m3 is now gone from dbstore1002, dbstore2001 and dbstore2002 (those last two didn't have it a loooong time ago)

Drop m3 from dbstore serversClosed, ResolvedPublicActions

Description

Related Objects

Event Timeline

Drop m3 from dbstore servers
Closed, ResolvedPublic
Actions