Page MenuHomePhabricator
Create Task
Maniphest T211973

Check GTID, consistency options, notifications across the fleet and db-eqiad.php weights
Closed, ResolvedPublic
Actions

Description

In order to prepare for the holidays season, let's make sure all the hosts in production have:

  • GTID enabled (excluding multi-source hosts) T211973#4823533
  • Consistency options are enforced (where they should be) T211973#4823549
  • Notifications are enabled (check both, puppet and icinga UI) for those hosts that are supposed to have them enabled. T211973#4823601
  • Check weights and possible depooled hosts on db-eqiad.php before holidays

Details

SubjectRepoBranchLines +/-
db-eqiad.php: Remove old commentsoperations/mediawiki-configmaster+2 -2
Customize query in gerrit

Event Timeline

Marostegui created this task.Dec 14 2018, 1:29 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 14 2018, 1:29 PM
Marostegui triaged this task as Medium priority.Dec 14 2018, 1:29 PM
Marostegui moved this task from Triage to In progress on the DBA board.
Marostegui added a comment.EditedDec 14 2018, 1:40 PM

The following hosts didn't have GTID enabled:

Most of those are codfw master that we left GTID without being enabled after the issue with s8.
What I will do is: run a compare on the revision table for them and an eqiad slave, enable it (one by one) let a few minutes pass by and then run the same compare.py was missed.

Marostegui updated the task description. (Show Details)Dec 14 2018, 1:41 PM
Stashbot subscribed.Dec 14 2018, 1:42 PM

Mentioned in SAL (#wikimedia-operations) [2018-12-14T13:42:25Z] <marostegui> Enable GTID on db1124:3318 - T211973

Marostegui added a comment.EditedDec 14 2018, 1:51 PM

The following hosts do not have sync_binlog and trx_commit enabled (and they should):

  • db2052.codfw.wmnet:3306 s5 master
  • db2051.codfw.wmnet:3306 s4 master
  • db2048.codfw.wmnet:3306 s1 master
  • db2040.codfw.wmnet:3306 s7 master
  • db2039.codfw.wmnet:3306 s6 master
  • db2035.codfw.wmnet:3306 s2 master
Marostegui updated the task description. (Show Details)Dec 14 2018, 1:51 PM
Stashbot added a comment.Dec 14 2018, 1:57 PM

Mentioned in SAL (#wikimedia-operations) [2018-12-14T13:57:31Z] <marostegui> Enable notifications for db2068 (s7 lag check)- T211973

Stashbot added a comment.Dec 14 2018, 1:59 PM

Mentioned in SAL (#wikimedia-operations) [2018-12-14T13:59:02Z] <marostegui> Enable notifications for db1095 (s3 lag check)- T211973

Marostegui added a comment.Dec 14 2018, 2:00 PM

db2068 had one check (s7 lag - only that one) with notifications disabled. I have reenabled it.
db1095 had one check (s3 lag - only that one) with notifications disabled. I have reenabled it.
The rest of hosts look good.

Marostegui updated the task description. (Show Details)Dec 14 2018, 2:01 PM
Stashbot added a comment.Dec 14 2018, 2:03 PM

Mentioned in SAL (#wikimedia-operations) [2018-12-14T14:03:36Z] <marostegui> Compare ruwiki.revision between db2039 (s6 master) and db1085 - T211973

Stashbot added a comment.Dec 14 2018, 2:38 PM

Mentioned in SAL (#wikimedia-operations) [2018-12-14T14:38:04Z] <marostegui> Enable GTID on db2039 (s6 codfw master) - T211973

Marostegui added a comment.Dec 14 2018, 4:14 PM

GTID enabled on db2039 (s6). After that a compare.py on frwiki.revision, jawiki.revision and ruwiki.revision revealed no differences.
I also checked the logs where the slave was stopped and started and it was the same position (the stop, GTID enablement and start happened within one second)

Dec 14 14:38:13 db2039 mysqld[1775]: 2018-12-14 14:38:13 140473564522240 [Note] Slave SQL thread exiting, replication stopped in log 'db1061-bin.002992' at position 715888744
Dec 14 14:38:13 db2039 mysqld[1775]: 2018-12-14 14:38:13 140612237440768 [Note] Slave I/O thread exiting, read up to log 'db1061-bin.002992', position 715888744
Dec 14 14:38:13 db2039 mysqld[1775]: 2018-12-14 14:38:13 140612234831616 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='db1061.eqiad.wmnet', master_port='3306', master_log_file='db1061-bin.002992', master_log_pos='715888744'. New state master_host='db1061
Dec 14 14:38:13 db2039 mysqld[1775]: 2018-12-14 14:38:13 140612234831616 [Note] Previous Using_Gtid=No. New Using_Gtid=Slave_Pos
Dec 14 14:38:13 db2039 mysqld[1775]: 2018-12-14 14:38:13 140473564522240 [Note] Slave I/O thread: Start asynchronous replication to master 'repl@db1061.eqiad.wmnet:3306' in log 'db1061-bin.002992' at position 715888744
Stashbot added a comment.Dec 17 2018, 6:29 AM

Mentioned in SAL (#wikimedia-operations) [2018-12-17T06:29:57Z] <marostegui> Enable replication consistency options on codfw masters - T211973

Marostegui added a comment.Dec 17 2018, 6:33 AM

I have enabled sync_binlog and trx_commit on db2052 db2051 db2048 db2040 db2039 db2035 - I'll keep an eye on the lag for those sections during the day

Marostegui updated the task description. (Show Details)Dec 17 2018, 6:34 AM
Stashbot added a comment.Dec 17 2018, 6:36 AM

Mentioned in SAL (#wikimedia-operations) [2018-12-17T06:36:34Z] <marostegui> Enable GTID on db2034 (x1 master) - T211973

Marostegui added a comment.Dec 17 2018, 6:38 AM

Enabled GTID on db2034 and position starts correctly:

Dec 17 06:36:50 db2034 mysqld[2632]: 2018-12-17  6:36:50 139848782440192 [Note] Slave SQL thread exiting, replication stopped in log 'db1069-bin.000287' at position 134076665
Dec 17 06:36:50 db2034 mysqld[2632]: 2018-12-17  6:36:50 139710424135424 [Note] Slave I/O thread exiting, read up to log 'db1069-bin.000287', position 134076665
Dec 17 06:36:50 db2034 mysqld[2632]: 2018-12-17  6:36:50 139850242307840 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='db1069.eqiad.wmnet', master_port='3306', master_log_file='db1069-bin.000287', master_log_pos='134076665'. New state master_host='db1069.eqiad.wmnet', master_port='3306', master_log_file='db1069-bin.000287', master_log_pos='134076665'.
Dec 17 06:36:50 db2034 mysqld[2632]: 2018-12-17  6:36:50 139850242307840 [Note] Previous Using_Gtid=No. New Using_Gtid=Slave_Pos
Dec 17 06:36:50 db2034 mysqld[2632]: 2018-12-17  6:36:50 139710424135424 [Note] Slave I/O thread: Start asynchronous replication to master 'repl@db1069.eqiad.wmnet:3306' in log 'db1069-bin.000287' at position 134076665
Dec 17 06:36:50 db2034 mysqld[2632]: 2018-12-17  6:36:50 139710424135424 [Note] Slave I/O thread: connected to master 'repl@db1069.eqiad.wmnet:3306',replication starts at GTID position '1-171970580-1,171970580-171970580-596994206,0-171970580-683331037,180363268-180363268-39992186,180355159-180355159-115369055,171966572-171966572-535580442,171974681-171974681-198565537'
Stashbot added a comment.Dec 17 2018, 6:48 AM

Mentioned in SAL (#wikimedia-operations) [2018-12-17T06:48:14Z] <marostegui> Enable GTID on db2035 (s2 master) - T211973

Marostegui added a comment.Dec 17 2018, 6:49 AM

Enabled GTID on db2035 and position starts correctly:

Dec 17 06:48:22 db2035 mysqld[64960]: 2018-12-17  6:48:22 139957454210816 [Note] Slave SQL thread exiting, replication stopped in log 'db1066-bin.000621' at position 306748668
Dec 17 06:48:22 db2035 mysqld[64960]: 2018-12-17  6:48:22 140096792909568 [Note] Slave I/O thread exiting, read up to log 'db1066-bin.000621', position 306748668
Dec 17 06:48:22 db2035 mysqld[64960]: 2018-12-17  6:48:22 139960099628800 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='db1066.eqiad.wmnet', master_port='3306', master_log_file='db1066-bin.000621', master_log_pos='306748668'. New state master_host='db1066.eqiad.wmnet', master_port='3306', master_log_file='db1066-bin.000621', master_log_pos='306748668'.
Dec 17 06:48:22 db2035 mysqld[64960]: 2018-12-17  6:48:22 139960099628800 [Note] Previous Using_Gtid=No. New Using_Gtid=Slave_Pos
Dec 17 06:48:22 db2035 mysqld[64960]: 2018-12-17  6:48:22 140096792909568 [Note] Slave I/O thread: Start asynchronous replication to master 'repl@db1066.eqiad.wmnet:3306' in log 'db1066-bin.000621' at position 306748668
Dec 17 06:48:22 db2035 mysqld[64960]: 2018-12-17  6:48:22 140096792909568 [Note] Slave I/O thread: connected to master 'repl@db1066.eqiad.wmnet:3306',replication starts at GTID position '171966574-171966574-941513258,0-180359173-4858865027,180359173-180359173-70817914,171966670-171966670-2410812544,171970567-171970567-390719906,180359241-180359241-121693516'
Marostegui added a comment.Dec 17 2018, 7:57 AM

A compare.py on x1 between codfw master and a eqiad slave for enwiki echo_notification table reveals no differences.
A compare.py on s2 between codfw master and eqiad slave for enwiktionary revision table reveals no differences.

Stashbot added a comment.Dec 17 2018, 7:58 AM

Mentioned in SAL (#wikimedia-operations) [2018-12-17T07:58:04Z] <marostegui> Enable GTID on db2052 (s5 master) - T211973

Marostegui added a comment.Dec 17 2018, 7:59 AM

Enabled GTID on db2052

Dec 17 07:58:08 db2052 mysqld[1565]: 2018-12-17  7:58:08 139774979925760 [Note] Slave SQL thread exiting, replication stopped in log 'db1070-bin.002351' at position 37077851
Dec 17 07:58:08 db2052 mysqld[1565]: 2018-12-17  7:58:08 139741735671552 [Note] Slave I/O thread exiting, read up to log 'db1070-bin.002351', position 37077851
Dec 17 07:58:08 db2052 mysqld[1565]: 2018-12-17  7:58:08 139809668024064 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='db1070.eqiad.wmnet', master_port='3306', master_log_file='db1070-bin.002351', master_log_pos='37077851'. New state master_host='db1070.eqiad.wmnet', master_port='3306', master_log_file='db1070-bin.002351', master_log_pos='37077851'.
Dec 17 07:58:08 db2052 mysqld[1565]: 2018-12-17  7:58:08 139809668024064 [Note] Previous Using_Gtid=No. New Using_Gtid=Slave_Pos
Dec 17 07:58:08 db2052 mysqld[1565]: 2018-12-17  7:58:08 139774979925760 [Note] Slave I/O thread: Start asynchronous replication to master 'repl@db1070.eqiad.wmnet:3306' in log 'db1070-bin.002351' at position 37077851
Dec 17 07:58:08 db2052 mysqld[1565]: 2018-12-17  7:58:08 139774979925760 [Note] Slave I/O thread: connected to master 'repl@db1070.eqiad.wmnet:3306',replication starts at GTID position '180359179-180359179-96523837,0-180359179-5734605861,180367364-180367364-67917352,171974884-171974884-1473084269,180363367-180363367-133158799,171970704-171970704-351094624,171978777-171978777-1338126799,171978768-171978768-202416'
Marostegui added a comment.Dec 17 2018, 9:43 AM

A compare.py on s5 between codfw master and a eqiad slave for dewiki revision table reveals no differences.

Stashbot added a comment.Dec 17 2018, 9:44 AM

Mentioned in SAL (#wikimedia-operations) [2018-12-17T09:44:10Z] <marostegui> Enable GTID on s4 codfw master - T211973

Marostegui added a comment.EditedDec 17 2018, 10:00 AM

s4 codfw master GTID enabled and SQL position looking fine before and after the stop:

Dec 17 09:44:22 db2051 mysqld[1407]: 2018-12-17  9:44:22 140456208512768 [Note] Slave SQL thread exiting, replication stopped in log 'db1068-bin.002309' at position 971302894
Dec 17 09:44:22 db2051 mysqld[1407]: 2018-12-17  9:44:22 140461864220416 [Note] Slave I/O thread exiting, read up to log 'db1068-bin.002309', position 971303723
Dec 17 09:44:22 db2051 mysqld[1407]: 2018-12-17  9:44:22 140459707184896 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='db1068.eqiad.wmnet', master_port='3306', master_log_file='db1068-bin.002309', master_log_pos='971303723'. New state master_host='db1068.eqiad.wmnet', master_port='3306', master_log_file='db1068-bin.002309', master_log_pos='971302894'.
Dec 17 09:44:22 db2051 mysqld[1407]: 2018-12-17  9:44:22 140459707184896 [Note] Previous Using_Gtid=No. New Using_Gtid=Slave_Pos
Dec 17 09:44:22 db2051 mysqld[1407]: 2018-12-17  9:44:22 140456208512768 [Note] Slave I/O thread: Start asynchronous replication to master 'repl@db1068.eqiad.wmnet:3306' in log 'db1068-bin.002309' at position 971302894
Dec 17 09:44:22 db2051 mysqld[1407]: 2018-12-17  9:44:22 140456208512768 [Note] Slave I/O thread: connected to master 'repl@db1068.eqiad.wmnet:3306',replication starts at GTID position '171978775-171978775-3839832693,0-180359175-3368394787,171970589-171970589-201132050,180359190-180359190-192195477,180359175-180359175-43143523'

I checked this:

Dec 17 09:44:22 db2051 mysqld[1407]: 2018-12-17  9:44:22 140459707184896 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='db1068.eqiad.wmnet', master_port='3306', master_log_file='db1068-bin.002309', master_log_pos='971303723'. New state master_host='db1068.eqiad.wmnet', master_port='3306', master_log_file='db1068-bin.002309', master_log_pos='971302894'.

I have also checked which transaction happened in between those positions (971303723 and 971302894) that time and I have done a diff for wbc_entity_usage for an eqiad slave and db2051 and it all looks good:

INSERT /* Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages  */ IGNORE INTO `wbc_entity_usage` (eu_page_id,eu_aspect,eu_entity_id) VALUES ('9825973','L.zh-mo','Q1028181'),('9825973','L.zh-hant','Q21488007'),('9825973','L.zh-hk','Q21488007'),('9825973','L.zh-mo','Q21488007'),('9825973','L.zh-hant','Q36524'),('9825973','L.zh-hk','Q36524'),('9825973','L.zh-mo','Q36524'),('9825973','L.zh-hant','Q3658608'),('9825973','L.zh-hk','Q3658608'),('9825973','L.zh-mo','Q3658608'),('9825973','L.zh-hant','Q4205426'),('9825973','L.zh-hk','Q4205426'),('9825973','L.zh-mo','Q4205426'),('9825973','L.zh-mo','Q82955'),('9825973','L.zh-mo','Q84'),('9825973','L.zh-mo','Q935877')
/*!*/;

I am also fully checking the whole wbc_entity_usage table, which will take 5 hours. Will update this comment once it is done.
Update: no differences found

Stashbot added a comment.Dec 17 2018, 1:49 PM

Mentioned in SAL (#wikimedia-operations) [2018-12-17T13:49:27Z] <marostegui> Enable GTID on s3 codfw master db2043 - T211973

Marostegui added a comment.Dec 17 2018, 1:51 PM

GTID enabled on s3 codfw master, db2043 and position before and after looking good:

Dec 17 13:49:47 db2043 mysqld[3352]: 2018-12-17 13:49:47 139902475896576 [Note] Slave SQL thread exiting, replication stopped in log 'db1075-bin.003543' at position 618453462
Dec 17 13:49:47 db2043 mysqld[3352]: 2018-12-17 13:49:47 139924462012160 [Note] Slave I/O thread exiting, read up to log 'db1075-bin.003543', position 618453462
Dec 17 13:49:47 db2043 mysqld[3352]: 2018-12-17 13:49:47 140062539745024 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='db1075.eqiad.wmnet', master_port='3306', master_log_file='db1075-bin.003543', master_log_pos='618453462'. New state master_host='db1075.eqiad.wmnet', master_port='3306', master_log_file='db1075-bin.003543', master_log_pos='618453462'.
Dec 17 13:49:47 db2043 mysqld[3352]: 2018-12-17 13:49:47 140062539745024 [Note] Previous Using_Gtid=No. New Using_Gtid=Slave_Pos
Dec 17 13:49:48 db2043 mysqld[3352]: 2018-12-17 13:49:48 139902475896576 [Note] Slave I/O thread: Start asynchronous replication to master 'repl@db1075.eqiad.wmnet:3306' in log 'db1075-bin.003543' at position 618453462
Dec 17 13:49:48 db2043 mysqld[3352]: 2018-12-17 13:49:48 139902475896576 [Note] Slave I/O thread: connected to master 'repl@db1075.eqiad.wmnet:3306',replication starts at GTID position '180363271-180363271-245332,0-171966669-4075108480,171966669-171966669-3427240963,180359174-180359174-94123433,180363367-180363367-134174373'
Marostegui renamed this task from Check GTID, consistency options and notifications across the fleet to Check GTID, consistency options, notifications across the fleet and db-eqiad.php weights.Dec 17 2018, 2:13 PM
Marostegui updated the task description. (Show Details)
Stashbot added a comment.Dec 18 2018, 8:23 AM

Mentioned in SAL (#wikimedia-operations) [2018-12-18T08:23:10Z] <marostegui> Enable GTID on es2 - T211973

Marostegui added a comment.Dec 18 2018, 8:24 AM

GTID enabled on es2 master (es2016) and all looks good

Dec 18 08:23:36 es2016 mysqld[5766]: 2018-12-18  8:23:36 140618110539520 [Note] Slave SQL thread exiting, replication stopped in log 'es1015-bin.002991' at position 745182040
Dec 18 08:23:36 es2016 mysqld[5766]: 2018-12-18  8:23:36 140507319916288 [Note] Slave I/O thread exiting, read up to log 'es1015-bin.002991', position 745182040
Dec 18 08:23:36 es2016 mysqld[5766]: 2018-12-18  8:23:36 140524179457792 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='es1015.eqiad.wmnet', master_port='3306', master_log_file='es1015-bin.002991', master_log_pos='745182040'. New state master_host='es1015.eqiad.wmnet', master_port='3306', master_log_file='es1015-bin.002991', master_log_pos='745182040'.
Dec 18 08:23:36 es2016 mysqld[5766]: 2018-12-18  8:23:36 140524179457792 [Note] Previous Using_Gtid=No. New Using_Gtid=Slave_Pos
Dec 18 08:23:36 es2016 mysqld[5766]: 2018-12-18  8:23:36 140507319916288 [Note] Slave I/O thread: Start asynchronous replication to master 'repl@es1015.eqiad.wmnet:3306' in log 'es1015-bin.002991' at position 745182040
Dec 18 08:23:37 es2016 mysqld[5766]: 2018-12-18  8:23:37 140507319916288 [Note] Slave I/O thread: connected to master 'repl@es1015.eqiad.wmnet:3306',replication starts at GTID position '0-171966470-411334248,171974840-171974840-75505599,180367401-180367401-77345258,171966470-171966470-356813944'
Stashbot added a comment.Dec 18 2018, 8:26 AM

Mentioned in SAL (#wikimedia-operations) [2018-12-18T08:26:05Z] <marostegui> Enable GTID on es3 - T211973

Marostegui added a comment.Dec 18 2018, 8:26 AM

GTID enabled on es3 master (es2017) and positions look good

Dec 18 08:26:09 es2017 mysqld[3886]: 2018-12-18  8:26:09 139599671289600 [Note] Slave SQL thread exiting, replication stopped in log 'es1017-bin.002989' at position 539807056
Dec 18 08:26:09 es2017 mysqld[3886]: 2018-12-18  8:26:09 139694367176448 [Note] Slave I/O thread exiting, read up to log 'es1017-bin.002989', position 539807056
Dec 18 08:26:10 es2017 mysqld[3886]: 2018-12-18  8:26:10 139667882735360 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='es1017.eqiad.wmnet', master_port='3306', master_log_file='es1017-bin.002989', master_log_pos='539807056'. New state master_host='es1017.eqiad.wmnet', master_port='3306', master_log_file='es1017-bin.002989', master_log_pos='539807056'.
Dec 18 08:26:10 es2017 mysqld[3886]: 2018-12-18  8:26:10 139667882735360 [Note] Previous Using_Gtid=No. New Using_Gtid=Slave_Pos
Dec 18 08:26:10 es2017 mysqld[3886]: 2018-12-18  8:26:10 139599671289600 [Note] Slave I/O thread: Start asynchronous replication to master 'repl@es1017.eqiad.wmnet:3306' in log 'es1017-bin.002989' at position 539807056
Dec 18 08:26:10 es2017 mysqld[3886]: 2018-12-18  8:26:10 139599671289600 [Note] Slave I/O thread: connected to master 'repl@es1017.eqiad.wmnet:3306',replication starts at GTID position '171970747-171970747-336240191,0-180359340-438187354,171974721-171974721-96102919,180355214-180355214-36897786,180359340-180359340-87'
Stashbot added a comment.Dec 18 2018, 8:44 AM

Mentioned in SAL (#wikimedia-operations) [2018-12-18T08:44:00Z] <marostegui> Enable GTID on s7 codfw master (db2040) - T211973

Marostegui added a comment.Dec 18 2018, 9:22 AM

GTID enabled on db2040 (s7 master)

Dec 18 08:44:07 db2040 mysqld[2167]: 2018-12-18  8:44:07 140468267357952 [Note] Slave SQL thread exiting, replication stopped in log 'db1062-bin.001823' at position 1022107387
Dec 18 08:44:07 db2040 mysqld[2167]: 2018-12-18  8:44:07 140609111860992 [Note] Slave I/O thread exiting, read up to log 'db1062-bin.001823', position 1022125212
Dec 18 08:44:07 db2040 mysqld[2167]: 2018-12-18  8:44:07 140465722799872 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='db1062.eqiad.wmnet', master_port='3306', master_log_file='db1062-bin.001823', master_log_pos='1022125212'. New state master_host='db1062.eqiad.wmnet', master_port='3306', master_log_file='db1062-bin.001823', master_log_pos='1022107387'.
Dec 18 08:44:07 db2040 mysqld[2167]: 2018-12-18  8:44:07 140465722799872 [Note] Previous Using_Gtid=No. New Using_Gtid=Slave_Pos
Dec 18 08:44:07 db2040 mysqld[2167]: 2018-12-18  8:44:07 140468267357952 [Note] Slave I/O thread: Start asynchronous replication to master 'repl@db1062.eqiad.wmnet:3306' in log 'db1062-bin.001823' at position 1022107387
Dec 18 08:44:07 db2040 mysqld[2167]: 2018-12-18  8:44:07 140468267357952 [Note] Slave I/O thread: connected to master 'repl@db1062.eqiad.wmnet:3306',replication starts at GTID position '171978767-171978767-3055755911,0-180359185-3359637071,180359185-180359185-71998080,171970590-171970590-196280066,180355111-180355111-131673159'

I have checked all the transactions that were supposed to run in between the BEGIN and the COMMIT and they are all there.

Stashbot added a comment.Dec 19 2018, 6:58 AM

Mentioned in SAL (#wikimedia-operations) [2018-12-19T06:58:18Z] <marostegui> Enable GTID on s1 codfw master (db2048) - T211973

Marostegui added a comment.Dec 19 2018, 6:59 AM

GTID enabled on s1 db2048 and it looks good:

Dec 19 06:58:26 db2048 mysqld[2665]: 2018-12-19  6:58:26 140190713382656 [Note] Slave SQL thread exiting, replication stopped in log 'db1067-bin.001933' at position 452800853
Dec 19 06:58:26 db2048 mysqld[2665]: 2018-12-19  6:58:26 140190721611520 [Note] Slave I/O thread exiting, read up to log 'db1067-bin.001933', position 452800853
Dec 19 06:58:26 db2048 mysqld[2665]: 2018-12-19  6:58:26 140190730041088 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='db1067.eqiad.wmnet', master_port='3306', master_log_file='db1067-bin.001933', master_log_pos='452800853'. New state master_host='db1067.eqiad.wmnet', master_port='3306', master_log_file='db1067-bin.001933', master_log_pos='452800853'.
Dec 19 06:58:26 db2048 mysqld[2665]: 2018-12-19  6:58:26 140190730041088 [Note] Previous Using_Gtid=No. New Using_Gtid=Slave_Pos
Dec 19 06:58:26 db2048 mysqld[2665]: 2018-12-19  6:58:26 140190713382656 [Note] Slave I/O thread: Start asynchronous replication to master 'repl@db1067.eqiad.wmnet:3306' in log 'db1067-bin.001933' at position 452800853
Dec 19 06:58:26 db2048 mysqld[2665]: 2018-12-19  6:58:26 140190713382656 [Note] Slave I/O thread: connected to master 'repl@db1067.eqiad.wmnet:3306',replication starts at GTID position '180355171-180355171-148310907,0-171970637-5484646134,171970637-171970637-2116621969,171978774-171978774-5,180359172-180359172-49702203,171974720-171974720-783079600'
Stashbot added a comment.Dec 19 2018, 6:59 AM

Mentioned in SAL (#wikimedia-operations) [2018-12-19T06:59:58Z] <marostegui> Enable GTID on s8 codfw master (db2045) - T211973

Marostegui added a comment.Dec 19 2018, 7:00 AM

GTID enabled on s8 db2045 and it looks good:

Dec 19 07:00:06 db2045 mysqld[3411]: 2018-12-19  7:00:06 140342229124864 [Note] Slave SQL thread exiting, replication stopped in log 'db1071-bin.007848' at position 641505655
Dec 19 07:00:06 db2045 mysqld[3411]: 2018-12-19  7:00:06 140481657648896 [Note] Slave I/O thread exiting, read up to log 'db1071-bin.007848', position 641505655
Dec 19 07:00:07 db2045 mysqld[3411]: 2018-12-19  7:00:07 140342935766784 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='db1071.eqiad.wmnet', master_port='3306', master_log_file='db1071-bin.007848', master_log_pos='641505655'. New state master_host='db1071.eqiad.wmnet', master_port='3306', master_log_file='db1071-bin.007848', master_log_pos='641505655'.
Dec 19 07:00:07 db2045 mysqld[3411]: 2018-12-19  7:00:07 140342935766784 [Note] Previous Using_Gtid=No. New Using_Gtid=Slave_Pos
Dec 19 07:00:07 db2045 mysqld[3411]: 2018-12-19  7:00:07 140481657648896 [Note] Slave I/O thread: Start asynchronous replication to master 'repl@db1071.eqiad.wmnet:3306' in log 'db1071-bin.007848' at position 641505655
Dec 19 07:00:07 db2045 mysqld[3411]: 2018-12-19  7:00:07 140481657648896 [Note] Slave I/O thread: connected to master 'repl@db1071.eqiad.wmnet:3306',replication starts at GTID position '180359179-180359179-96523837,0-180359179-5734605861,171974884-171974884-1473084269,171978778-171978778-1529781660,171970704-171970704-351094624,171978768-171978768-202416,171978777-171978777-329020349,180359242-180359242-170963125'
Marostegui updated the task description. (Show Details)Dec 19 2018, 7:01 AM
gerritbot subscribed.Dec 21 2018, 6:51 AM

Change 481118 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Remove old comments

https://gerrit.wikimedia.org/r/481118

gerritbot added a project: Patch-For-Review.Dec 21 2018, 6:51 AM
gerritbot added a comment.Dec 21 2018, 6:54 AM

Change 481118 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Remove old comments

https://gerrit.wikimedia.org/r/481118

Stashbot added a comment.Dec 21 2018, 6:55 AM

Mentioned in SAL (#wikimedia-operations) [2018-12-21T06:55:27Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Remove old comments T211973 (duration: 00m 46s)

Marostegui closed this task as Resolved.Dec 21 2018, 6:59 AM

I have reviewed db-eqiad.php and also db-codfw.php and all the servers that have to be pooled are pooled and the weights also look good to me.

Marostegui updated the task description. (Show Details)Dec 21 2018, 6:59 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL