Page MenuHomePhabricator

Drop m3 from dbstore servers
Closed, ResolvedPublic

Description

Hello,

Due to the phabricator update (T156373) we experienced a crash on all the m3 replicas because of a MariaDB bug (T156373#2973202 T156373#2973962).
We were lucky that the master and the eqiad dbstores were running an old non affected version of Mariadb, so they survived the crash.

However, we have started to wonder if it is really necessary to have m3 on the dbstores servers, specially on dbstore1002 (which is the non delayed replica).
We assume it is not needed at all on codfw and we are going to drop it from there, but we'd like to know if m3 in dbstore1002 is being used or it can be dropped there too.
Any thoughts from Analytics team?

Thanks!

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 31 2017, 9:29 AM
Marostegui moved this task from Triage to Backlog on the DBA board.Jan 31 2017, 9:29 AM

m3 has been removed from dbstore2001 as per: T156905#2991826

We don't know what m3 is. So far we don't know of a use for Phabricator databases on the analytics slaves.

Mentioned in SAL (#wikimedia-operations) [2017-08-09T20:51:38Z] <marostegui> Remove m3 replication from dbstore1002 - T156758

I have removed replication from 'm3' thread on dbstore1002 and these are the values:

root@DBSTORE[(none)]> select @@hostname;
+-------------+
| @@hostname  |
+-------------+
| dbstore1002 |
+-------------+
1 row in set (0.00 sec)

root@DBSTORE[(none)]> show slave status\G
*************************** 1. row ***************************
               Slave_IO_State:
                  Master_Host: db1043.eqiad.wmnet
                  Master_User: repl
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: db1043-bin.001752
          Read_Master_Log_Pos: 569268565
               Relay_Log_File: dbstore1002-relay-bin-m3.000744
                Relay_Log_Pos: 569268853
        Relay_Master_Log_File: db1043-bin.001752
             Slave_IO_Running: No
            Slave_SQL_Running: No
              Replicate_Do_DB:
          Replicate_Ignore_DB:
           Replicate_Do_Table:
       Replicate_Ignore_Table:
      Replicate_Wild_Do_Table: phab%.%,phlegal%.%,heartbeat.%
  Replicate_Wild_Ignore_Table:
                   Last_Errno: 0
                   Last_Error:
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 569268565
              Relay_Log_Space: 569269203
              Until_Condition: None
               Until_Log_File:
                Until_Log_Pos: 0
           Master_SSL_Allowed: Yes
           Master_SSL_CA_File:
           Master_SSL_CA_Path:
              Master_SSL_Cert:
            Master_SSL_Cipher:
               Master_SSL_Key:
        Seconds_Behind_Master: NULL
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error:
               Last_SQL_Errno: 0
               Last_SQL_Error:
  Replicate_Ignore_Server_Ids:
             Master_Server_Id: 171970592
               Master_SSL_Crl:
           Master_SSL_Crlpath:
                   Using_Gtid: No
                  Gtid_IO_Pos:
1 row in set (0.00 sec)

root@DBSTORE[(none)]> reset slave all;
Query OK, 0 rows affected (0.09 sec)

If there are no complains after this, I will remove all the m3 content from dbstore1002

Marostegui moved this task from Backlog to In progress on the DBA board.

Mentioned in SAL (#wikimedia-operations) [2017-08-10T13:17:49Z] <marostegui> Drop m3 databases from dbstore1002 - T156758

Marostegui closed this task as Resolved.Aug 10 2017, 1:20 PM

m3 is now gone from dbstore1002, dbstore2001 and dbstore2002 (those last two didn't have it a loooong time ago)