Page MenuHomePhabricator
Create Task
Maniphest T243373

Enable DB replication codfw -> eqiad before the switchover and some other checks
Closed, ResolvedPublic
Actions

Description

Date of the switchover still to be defined but before we switch over we have to make sure that:

  • Replication codfw -> eqiad is running
    • s1
    • s2
    • s3
    • s4
    • s5
    • s6
    • s7
    • s8
    • x1
    • es4
    • es5
    • pc1
    • pc2
    • pc3
  • Check and disable GTID on codfw masters for the above sections.
  • Check that all codfw slaves have GTID enabled
  • Check which notifications are disabled for codfw hosts
  • Check event scheduler is enabled on codfw hosts
  • Check that query killers are installed and enabled on codfw hosts
  • Review DB MW weights
    • s1
    • s2
    • s3
    • s4
    • s5
    • s6
    • s7
    • s8
    • x1
    • es4
    • es5
    • pc1
    • pc2
    • pc3

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 22 2020, 7:41 AM
• Marostegui renamed this task from Enable replication codfw -> eqiad before the switchover to Enable DB replication codfw -> eqiad before the switchover.Jan 22 2020, 7:41 AM
• Marostegui triaged this task as Medium priority.
• Marostegui moved this task from Triage to Backlog on the DBA board.
• Marostegui updated the task description. (Show Details)Jan 22 2020, 7:57 AM
RhinosF1 added a subscriber: RhinosF1.Aug 3 2020, 2:23 PM
• Marostegui added a comment.Aug 5 2020, 7:55 AM

Scheduled for 27th August.

• Marostegui moved this task from Backlog to In progress on the DBA board.Aug 25 2020, 12:27 PM
• Marostegui added subscribers: Kormat, jcrespo.EditedAug 27 2020, 7:23 AM

@jcrespo @Kormat thoughts on codfw -> eqiad replication on misc sections? Initially I didn't include them as we are not switching those services, but maybe for consistency we should

Stashbot added a comment.Aug 27 2020, 7:35 AM

Mentioned in SAL (#wikimedia-operations) [2020-08-27T07:35:06Z] <marostegui> Move pc2010 under pc2007 T243373

Stashbot added a comment.Aug 27 2020, 8:13 AM

Mentioned in SAL (#wikimedia-operations) [2020-08-27T08:13:22Z] <marostegui> Enable replication codfw -> eqiad on pc1 T243373

• Marostegui updated the task description. (Show Details)Aug 27 2020, 8:13 AM
Stashbot added a comment.Aug 27 2020, 8:44 AM

Mentioned in SAL (#wikimedia-operations) [2020-08-27T08:44:32Z] <kormat> enabling replication from pc2008 to pc1008 (pc2) T243373

Kormat added a project: User-Kormat.Aug 27 2020, 8:48 AM
Kormat updated the task description. (Show Details)
Stashbot added a comment.Aug 27 2020, 8:53 AM

Mentioned in SAL (#wikimedia-operations) [2020-08-27T08:53:40Z] <kormat> enabling replication from pc2009 to pc1009 (pc3) T243373

Kormat updated the task description. (Show Details)Aug 27 2020, 8:56 AM
Stashbot added a comment.Aug 27 2020, 9:07 AM

Mentioned in SAL (#wikimedia-operations) [2020-08-27T09:07:08Z] <kormat> enabling replication from db2090 to db1081 (s4) T243373

Kormat updated the task description. (Show Details)Aug 27 2020, 9:08 AM
Stashbot added a comment.Aug 27 2020, 9:15 AM

Mentioned in SAL (#wikimedia-operations) [2020-08-27T09:15:07Z] <kormat> enabling replication from db2079 to db1109 (s8) T243373

Stashbot added a comment.Aug 27 2020, 9:20 AM

Mentioned in SAL (#wikimedia-operations) [2020-08-27T09:20:54Z] <kormat> enabling replication from db2105 to db1123 (s3) T243373

Kormat updated the task description. (Show Details)Aug 27 2020, 9:21 AM
Stashbot added a comment.Aug 27 2020, 10:23 AM

Mentioned in SAL (#wikimedia-operations) [2020-08-27T10:23:57Z] <kormat> enabling replication from es2021 to es1021 (es4) T243373

Stashbot added a comment.Aug 27 2020, 10:30 AM

Mentioned in SAL (#wikimedia-operations) [2020-08-27T10:30:38Z] <kormat> enabling replication from es2023 to es1024 (es5) T243373

Kormat updated the task description. (Show Details)Aug 27 2020, 10:31 AM
Kormat updated the task description. (Show Details)
Stashbot added a comment.Aug 27 2020, 10:51 AM

Mentioned in SAL (#wikimedia-operations) [2020-08-27T10:51:18Z] <kormat> enabling replication from db2123 to db1100 (s5) T243373

Kormat updated the task description. (Show Details)Aug 27 2020, 10:51 AM
• Marostegui added a comment.Aug 27 2020, 10:53 AM

I am re-doing a compare on all those sections where cross replication is enabled to be double sure (and to keep warming up tables), and so far so good.

• Marostegui added a comment.Aug 27 2020, 11:06 AM

Checked all codfw hosts with notifications disabled.
Only found db2135 (m5)

• Marostegui updated the task description. (Show Details)Aug 27 2020, 11:06 AM
• Marostegui updated the task description. (Show Details)Aug 27 2020, 11:13 AM
• Marostegui updated the task description. (Show Details)Aug 27 2020, 11:18 AM
Stashbot added a comment.Aug 27 2020, 11:22 AM

Mentioned in SAL (#wikimedia-operations) [2020-08-27T11:22:13Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Adjust db2126 weight T243373', diff saved to https://phabricator.wikimedia.org/P12394 and previous config saved to /var/cache/conftool/dbconfig/20200827-112213-marostegui.json

• Marostegui updated the task description. (Show Details)Aug 27 2020, 11:22 AM
Stashbot added a comment.Aug 27 2020, 11:45 AM

Mentioned in SAL (#wikimedia-operations) [2020-08-27T11:45:10Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Adjust s1 codfw weights T243373', diff saved to https://phabricator.wikimedia.org/P12395 and previous config saved to /var/cache/conftool/dbconfig/20200827-114509-marostegui.json

• Marostegui updated the task description. (Show Details)Aug 27 2020, 11:45 AM
Stashbot added a comment.Aug 27 2020, 11:51 AM

Mentioned in SAL (#wikimedia-operations) [2020-08-27T11:51:11Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Adjust s4 codfw weights T243373', diff saved to https://phabricator.wikimedia.org/P12396 and previous config saved to /var/cache/conftool/dbconfig/20200827-115110-marostegui.json

• Marostegui updated the task description. (Show Details)Aug 27 2020, 11:51 AM
Stashbot added a comment.Aug 27 2020, 11:59 AM

Mentioned in SAL (#wikimedia-operations) [2020-08-27T11:59:35Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Adjust s5 eqiad weights T243373', diff saved to https://phabricator.wikimedia.org/P12397 and previous config saved to /var/cache/conftool/dbconfig/20200827-115934-marostegui.json

Stashbot added a comment.Aug 27 2020, 12:02 PM

Mentioned in SAL (#wikimedia-operations) [2020-08-27T12:02:11Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Adjust s5 codfw weights T243373', diff saved to https://phabricator.wikimedia.org/P12398 and previous config saved to /var/cache/conftool/dbconfig/20200827-120211-marostegui.json

• Marostegui updated the task description. (Show Details)Aug 27 2020, 12:02 PM
Stashbot added a comment.Aug 27 2020, 12:08 PM

Mentioned in SAL (#wikimedia-operations) [2020-08-27T12:08:16Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Adjust s6 weights T243373', diff saved to https://phabricator.wikimedia.org/P12399 and previous config saved to /var/cache/conftool/dbconfig/20200827-120816-marostegui.json

• Marostegui updated the task description. (Show Details)Aug 27 2020, 12:08 PM
Stashbot added a comment.Aug 27 2020, 12:14 PM

Mentioned in SAL (#wikimedia-operations) [2020-08-27T12:14:54Z] <kormat> enabling replication from db2129 to db1093 (s6) T243373

Stashbot added a comment.Aug 27 2020, 12:24 PM

Mentioned in SAL (#wikimedia-operations) [2020-08-27T12:24:10Z] <marostegui> Fix password format for in db2129 (s6 codfw master) T243373

Kormat updated the task description. (Show Details)Aug 27 2020, 12:24 PM
Stashbot added a comment.Aug 27 2020, 12:30 PM

Mentioned in SAL (#wikimedia-operations) [2020-08-27T12:30:03Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Adjust s7 weights T243373', diff saved to https://phabricator.wikimedia.org/P12400 and previous config saved to /var/cache/conftool/dbconfig/20200827-123003-marostegui.json

Stashbot added a comment.Aug 27 2020, 12:30 PM

Mentioned in SAL (#wikimedia-operations) [2020-08-27T12:30:28Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Adjust s7 weights T243373', diff saved to https://phabricator.wikimedia.org/P12401 and previous config saved to /var/cache/conftool/dbconfig/20200827-123028-marostegui.json

• Marostegui updated the task description. (Show Details)Aug 27 2020, 12:30 PM
Stashbot added a comment.Aug 27 2020, 12:43 PM

Mentioned in SAL (#wikimedia-operations) [2020-08-27T12:43:39Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Adjust s8 weights T243373', diff saved to https://phabricator.wikimedia.org/P12402 and previous config saved to /var/cache/conftool/dbconfig/20200827-124338-marostegui.json

• Marostegui updated the task description. (Show Details)Aug 27 2020, 12:44 PM

Weights are tackled, but I will give them another review tomorrow morning.

Stashbot added a comment.Aug 27 2020, 1:01 PM

Mentioned in SAL (#wikimedia-operations) [2020-08-27T13:01:31Z] <kormat> enabling replication from db2118 to db1086 (s7) T243373

Kormat updated the task description. (Show Details)Aug 27 2020, 1:02 PM
Stashbot added a comment.Aug 27 2020, 1:14 PM

Mentioned in SAL (#wikimedia-operations) [2020-08-27T13:14:02Z] <kormat> enabling replication from db2096 to db1103 (x1) T243373

Kormat updated the task description. (Show Details)Aug 27 2020, 1:14 PM
Stashbot added a comment.Aug 27 2020, 1:18 PM

Mentioned in SAL (#wikimedia-operations) [2020-08-27T13:18:44Z] <kormat> enabling replication from db2107 to db1122 (s2) T243373

Kormat updated the task description. (Show Details)Aug 27 2020, 1:19 PM
Kormat added a comment.EditedAug 27 2020, 1:22 PM
In T243373#6414969, @Marostegui wrote:

@jcrespo @Kormat thoughts on codfw -> eqiad replication on misc sections? Initially I didn't include them as we are not switching those services, but maybe for consistency we should

profile::mariadb::replication_lag assumes the 'master' is in mediawiki::state('primary_dc'). This will not be true once the dc switch happens.

(This is independent of codfw->eqiad replication.)

• Marostegui added a comment.EditedAug 27 2020, 1:48 PM

I have gone thru logs from past switchover and we indeed left mX hosts aside from reenabling replication codfw -> eqiad, so let's not enable it.

• Marostegui updated the task description. (Show Details)Aug 27 2020, 1:49 PM
Stashbot added a comment.Aug 27 2020, 1:50 PM

Mentioned in SAL (#wikimedia-operations) [2020-08-27T13:50:34Z] <kormat> disabling GTID on db2107 (s2) T243373

Stashbot added a comment.Aug 27 2020, 1:51 PM

Mentioned in SAL (#wikimedia-operations) [2020-08-27T13:51:24Z] <kormat> disabling GTID on db2105 (s3) T243373

Stashbot added a comment.Aug 27 2020, 1:52 PM

Mentioned in SAL (#wikimedia-operations) [2020-08-27T13:52:05Z] <kormat> disabling GTID on db2090 (s4) T243373

Stashbot added a comment.Aug 27 2020, 1:52 PM

Mentioned in SAL (#wikimedia-operations) [2020-08-27T13:52:46Z] <kormat> disabling GTID on db2123 (s5) T243373

Stashbot added a comment.Aug 27 2020, 1:53 PM

Mentioned in SAL (#wikimedia-operations) [2020-08-27T13:53:53Z] <kormat> disabling GTID on db2129 (s6), db2118 (s7), db2079 (s8) T243373

Stashbot added a comment.Aug 27 2020, 1:56 PM

Mentioned in SAL (#wikimedia-operations) [2020-08-27T13:56:31Z] <kormat> disabling GTID on db2096 (x1), es2021 (es4), es2023 (es5) T243373

Stashbot added a comment.Aug 27 2020, 1:58 PM

Mentioned in SAL (#wikimedia-operations) [2020-08-27T13:58:39Z] <kormat> disabling GTID on pc2007 (pc1), pc2008 (pc2), pc2009 (pc3) T243373

Kormat added a comment.Aug 27 2020, 1:59 PM

GTID disabled on all codfw masters except for s1.

Stashbot added a comment.Aug 28 2020, 8:22 AM

Mentioned in SAL (#wikimedia-operations) [2020-08-28T08:22:41Z] <kormat> enabling replication from db2112 to db1083 (s1) T243373

Kormat updated the task description. (Show Details)Aug 28 2020, 8:24 AM
Kormat updated the task description. (Show Details)Aug 28 2020, 8:27 AM

Confirmed that GTID is disabled on all codfw masters:

sudo cumin 'A:db-role-master and A:codfw' 'mysql -e "show slave status\G" | grep Using_Gtid'
15 hosts will be targeted:
db[2079,2090,2096,2105,2107,2112,2118,2123,2129].codfw.wmnet,es[2021,2023].codfw.wmnet,pc[2007-2010].codfw.wmnet
Confirm to continue [y/n]? y
===== NODE GROUP =====                                                                                                                                           
(1) pc2010.codfw.wmnet                                                                                                                                           
----- OUTPUT of 'mysql -e "show s... grep Using_Gtid' -----                                                                                                      
                    Using_Gtid: Slave_Pos                                                                                                                        
===== NODE GROUP =====                                                                                                                                           
(6) db2096.codfw.wmnet,es[2021,2023].codfw.wmnet,pc[2007-2009].codfw.wmnet                                                                                       
----- OUTPUT of 'mysql -e "show s... grep Using_Gtid' -----                                                                                                      
                    Using_Gtid: No                                                                                                                               
===== NODE GROUP =====                                                                                                                                           
(8) db[2079,2090,2105,2107,2112,2118,2123,2129].codfw.wmnet                                                                                                      
----- OUTPUT of 'mysql -e "show s... grep Using_Gtid' -----                                                                                                      
                   Using_Gtid: No                                                                                                                                
================                                                                                                                                                 
PASS |███████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (15/15) [00:01<00:00,  2.61hosts/s]          
FAIL |                                                                                                                |   0% (0/15) [00:01<?, ?hosts/s]
100.0% (15/15) success ratio (>= 100.0% threshold) for command: 'mysql -e "show s... grep Using_Gtid'.
100.0% (15/15) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.

(pc2010 is a false positive, as all parsercaches are 'masters')

• Marostegui closed this task as Resolved.Aug 28 2020, 8:43 AM

I have reviewed the weights again so this is all done.
As part of T260042 I am warming up all the hosts in codfw, including the masters, parsercache, es...

Thanks Stephen for taking care of the replication re-enablement.

• Marostegui updated the task description. (Show Details)Aug 31 2020, 4:53 AM
• Marostegui renamed this task from Enable DB replication codfw -> eqiad before the switchover to Enable DB replication codfw -> eqiad before the switchover and some other checks.Aug 31 2020, 5:02 AM
• Marostegui updated the task description. (Show Details)

All codfw have been checked for GTID enabled, they all had it (not codfw masters)

• Marostegui updated the task description. (Show Details)Sep 1 2020, 6:16 AM
Stashbot added a comment.Sep 1 2020, 6:20 AM

Mentioned in SAL (#wikimedia-operations) [2020-09-01T06:20:50Z] <marostegui> Install query killers on db2137:3314 T243373

• Marostegui added a comment.Sep 1 2020, 6:26 AM

I have checked that the event scheduler is enabled everywhere within codfw.
Same for query killers.
Query killer wasn't present on db2137:3314 db2137:3315, so I have installed them there.

On es3, es2017 only had the master ones, I have installed the slaves ones.

• Marostegui updated the task description. (Show Details)Sep 1 2020, 6:26 AM
• Marostegui added a comment.Sep 2 2020, 11:52 AM

I have checked that replication works fine from codfw to eqiad by checking a few tables, sections checked:

  • pc1,pc2, pc3
  • es4, es5
  • s1 enwiki
  • s2 itwiki
  • s3 ilowiki (created my own user to check)
  • s4 commons
  • s5 dewiki
  • s6 frwiki
  • s7 eswiki
  • s8 wikidatawiki
  • x1 enwiki for echo_event
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL