Page MenuHomePhabricator

Check GTID, consistency options, notifications across the fleet and db-eqiad.php weights
Closed, ResolvedPublic

Description

In order to prepare for the holidays season, let's make sure all the hosts in production have:

  • GTID enabled (excluding multi-source hosts) T211973#4823533
  • Consistency options are enforced (where they should be) T211973#4823549
  • Notifications are enabled (check both, puppet and icinga UI) for those hosts that are supposed to have them enabled. T211973#4823601
  • Check weights and possible depooled hosts on db-eqiad.php before holidays

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 14 2018, 1:29 PM
Marostegui triaged this task as Medium priority.Dec 14 2018, 1:29 PM
Marostegui moved this task from Triage to In progress on the DBA board.
Marostegui added a comment.EditedDec 14 2018, 1:40 PM

The following hosts didn't have GTID enabled:

Most of those are codfw master that we left GTID without being enabled after the issue with s8.
What I will do is: run a compare on the revision table for them and an eqiad slave, enable it (one by one) let a few minutes pass by and then run the same compare.py was missed.

Marostegui updated the task description. (Show Details)Dec 14 2018, 1:41 PM

Mentioned in SAL (#wikimedia-operations) [2018-12-14T13:42:25Z] <marostegui> Enable GTID on db1124:3318 - T211973

Marostegui added a comment.EditedDec 14 2018, 1:51 PM

The following hosts do not have sync_binlog and trx_commit enabled (and they should):

  • db2052.codfw.wmnet:3306 s5 master
  • db2051.codfw.wmnet:3306 s4 master
  • db2048.codfw.wmnet:3306 s1 master
  • db2040.codfw.wmnet:3306 s7 master
  • db2039.codfw.wmnet:3306 s6 master
  • db2035.codfw.wmnet:3306 s2 master
Marostegui updated the task description. (Show Details)Dec 14 2018, 1:51 PM

Mentioned in SAL (#wikimedia-operations) [2018-12-14T13:57:31Z] <marostegui> Enable notifications for db2068 (s7 lag check)- T211973

Mentioned in SAL (#wikimedia-operations) [2018-12-14T13:59:02Z] <marostegui> Enable notifications for db1095 (s3 lag check)- T211973

db2068 had one check (s7 lag - only that one) with notifications disabled. I have reenabled it.
db1095 had one check (s3 lag - only that one) with notifications disabled. I have reenabled it.
The rest of hosts look good.

Marostegui updated the task description. (Show Details)Dec 14 2018, 2:01 PM

Mentioned in SAL (#wikimedia-operations) [2018-12-14T14:03:36Z] <marostegui> Compare ruwiki.revision between db2039 (s6 master) and db1085 - T211973

Mentioned in SAL (#wikimedia-operations) [2018-12-14T14:38:04Z] <marostegui> Enable GTID on db2039 (s6 codfw master) - T211973

GTID enabled on db2039 (s6). After that a compare.py on frwiki.revision, jawiki.revision and ruwiki.revision revealed no differences.
I also checked the logs where the slave was stopped and started and it was the same position (the stop, GTID enablement and start happened within one second)

Dec 14 14:38:13 db2039 mysqld[1775]: 2018-12-14 14:38:13 140473564522240 [Note] Slave SQL thread exiting, replication stopped in log 'db1061-bin.002992' at position 715888744
Dec 14 14:38:13 db2039 mysqld[1775]: 2018-12-14 14:38:13 140612237440768 [Note] Slave I/O thread exiting, read up to log 'db1061-bin.002992', position 715888744
Dec 14 14:38:13 db2039 mysqld[1775]: 2018-12-14 14:38:13 140612234831616 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='db1061.eqiad.wmnet', master_port='3306', master_log_file='db1061-bin.002992', master_log_pos='715888744'. New state master_host='db1061
Dec 14 14:38:13 db2039 mysqld[1775]: 2018-12-14 14:38:13 140612234831616 [Note] Previous Using_Gtid=No. New Using_Gtid=Slave_Pos
Dec 14 14:38:13 db2039 mysqld[1775]: 2018-12-14 14:38:13 140473564522240 [Note] Slave I/O thread: Start asynchronous replication to master 'repl@db1061.eqiad.wmnet:3306' in log 'db1061-bin.002992' at position 715888744

Mentioned in SAL (#wikimedia-operations) [2018-12-17T06:29:57Z] <marostegui> Enable replication consistency options on codfw masters - T211973

I have enabled sync_binlog and trx_commit on db2052 db2051 db2048 db2040 db2039 db2035 - I'll keep an eye on the lag for those sections during the day

Marostegui updated the task description. (Show Details)Dec 17 2018, 6:34 AM

Mentioned in SAL (#wikimedia-operations) [2018-12-17T06:36:34Z] <marostegui> Enable GTID on db2034 (x1 master) - T211973

Enabled GTID on db2034 and position starts correctly:

Dec 17 06:36:50 db2034 mysqld[2632]: 2018-12-17  6:36:50 139848782440192 [Note] Slave SQL thread exiting, replication stopped in log 'db1069-bin.000287' at position 134076665
Dec 17 06:36:50 db2034 mysqld[2632]: 2018-12-17  6:36:50 139710424135424 [Note] Slave I/O thread exiting, read up to log 'db1069-bin.000287', position 134076665
Dec 17 06:36:50 db2034 mysqld[2632]: 2018-12-17  6:36:50 139850242307840 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='db1069.eqiad.wmnet', master_port='3306', master_log_file='db1069-bin.000287', master_log_pos='134076665'. New state master_host='db1069.eqiad.wmnet', master_port='3306', master_log_file='db1069-bin.000287', master_log_pos='134076665'.
Dec 17 06:36:50 db2034 mysqld[2632]: 2018-12-17  6:36:50 139850242307840 [Note] Previous Using_Gtid=No. New Using_Gtid=Slave_Pos
Dec 17 06:36:50 db2034 mysqld[2632]: 2018-12-17  6:36:50 139710424135424 [Note] Slave I/O thread: Start asynchronous replication to master 'repl@db1069.eqiad.wmnet:3306' in log 'db1069-bin.000287' at position 134076665
Dec 17 06:36:50 db2034 mysqld[2632]: 2018-12-17  6:36:50 139710424135424 [Note] Slave I/O thread: connected to master 'repl@db1069.eqiad.wmnet:3306',replication starts at GTID position '1-171970580-1,171970580-171970580-596994206,0-171970580-683331037,180363268-180363268-39992186,180355159-180355159-115369055,171966572-171966572-535580442,171974681-171974681-198565537'

Mentioned in SAL (#wikimedia-operations) [2018-12-17T06:48:14Z] <marostegui> Enable GTID on db2035 (s2 master) - T211973

Enabled GTID on db2035 and position starts correctly:

Dec 17 06:48:22 db2035 mysqld[64960]: 2018-12-17  6:48:22 139957454210816 [Note] Slave SQL thread exiting, replication stopped in log 'db1066-bin.000621' at position 306748668
Dec 17 06:48:22 db2035 mysqld[64960]: 2018-12-17  6:48:22 140096792909568 [Note] Slave I/O thread exiting, read up to log 'db1066-bin.000621', position 306748668
Dec 17 06:48:22 db2035 mysqld[64960]: 2018-12-17  6:48:22 139960099628800 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='db1066.eqiad.wmnet', master_port='3306', master_log_file='db1066-bin.000621', master_log_pos='306748668'. New state master_host='db1066.eqiad.wmnet', master_port='3306', master_log_file='db1066-bin.000621', master_log_pos='306748668'.
Dec 17 06:48:22 db2035 mysqld[64960]: 2018-12-17  6:48:22 139960099628800 [Note] Previous Using_Gtid=No. New Using_Gtid=Slave_Pos
Dec 17 06:48:22 db2035 mysqld[64960]: 2018-12-17  6:48:22 140096792909568 [Note] Slave I/O thread: Start asynchronous replication to master 'repl@db1066.eqiad.wmnet:3306' in log 'db1066-bin.000621' at position 306748668
Dec 17 06:48:22 db2035 mysqld[64960]: 2018-12-17  6:48:22 140096792909568 [Note] Slave I/O thread: connected to master 'repl@db1066.eqiad.wmnet:3306',replication starts at GTID position '171966574-171966574-941513258,0-180359173-4858865027,180359173-180359173-70817914,171966670-171966670-2410812544,171970567-171970567-390719906,180359241-180359241-121693516'

A compare.py on x1 between codfw master and a eqiad slave for enwiki echo_notification table reveals no differences.
A compare.py on s2 between codfw master and eqiad slave for enwiktionary revision table reveals no differences.

Mentioned in SAL (#wikimedia-operations) [2018-12-17T07:58:04Z] <marostegui> Enable GTID on db2052 (s5 master) - T211973

Enabled GTID on db2052

Dec 17 07:58:08 db2052 mysqld[1565]: 2018-12-17  7:58:08 139774979925760 [Note] Slave SQL thread exiting, replication stopped in log 'db1070-bin.002351' at position 37077851
Dec 17 07:58:08 db2052 mysqld[1565]: 2018-12-17  7:58:08 139741735671552 [Note] Slave I/O thread exiting, read up to log 'db1070-bin.002351', position 37077851
Dec 17 07:58:08 db2052 mysqld[1565]: 2018-12-17  7:58:08 139809668024064 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='db1070.eqiad.wmnet', master_port='3306', master_log_file='db1070-bin.002351', master_log_pos='37077851'. New state master_host='db1070.eqiad.wmnet', master_port='3306', master_log_file='db1070-bin.002351', master_log_pos='37077851'.
Dec 17 07:58:08 db2052 mysqld[1565]: 2018-12-17  7:58:08 139809668024064 [Note] Previous Using_Gtid=No. New Using_Gtid=Slave_Pos
Dec 17 07:58:08 db2052 mysqld[1565]: 2018-12-17  7:58:08 139774979925760 [Note] Slave I/O thread: Start asynchronous replication to master 'repl@db1070.eqiad.wmnet:3306' in log 'db1070-bin.002351' at position 37077851
Dec 17 07:58:08 db2052 mysqld[1565]: 2018-12-17  7:58:08 139774979925760 [Note] Slave I/O thread: connected to master 'repl@db1070.eqiad.wmnet:3306',replication starts at GTID position '180359179-180359179-96523837,0-180359179-5734605861,180367364-180367364-67917352,171974884-171974884-1473084269,180363367-180363367-133158799,171970704-171970704-351094624,171978777-171978777-1338126799,171978768-171978768-202416'

A compare.py on s5 between codfw master and a eqiad slave for dewiki revision table reveals no differences.

Mentioned in SAL (#wikimedia-operations) [2018-12-17T09:44:10Z] <marostegui> Enable GTID on s4 codfw master - T211973

Marostegui added a comment.EditedDec 17 2018, 10:00 AM

s4 codfw master GTID enabled and SQL position looking fine before and after the stop:

Dec 17 09:44:22 db2051 mysqld[1407]: 2018-12-17  9:44:22 140456208512768 [Note] Slave SQL thread exiting, replication stopped in log 'db1068-bin.002309' at position 971302894
Dec 17 09:44:22 db2051 mysqld[1407]: 2018-12-17  9:44:22 140461864220416 [Note] Slave I/O thread exiting, read up to log 'db1068-bin.002309', position 971303723
Dec 17 09:44:22 db2051 mysqld[1407]: 2018-12-17  9:44:22 140459707184896 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='db1068.eqiad.wmnet', master_port='3306', master_log_file='db1068-bin.002309', master_log_pos='971303723'. New state master_host='db1068.eqiad.wmnet', master_port='3306', master_log_file='db1068-bin.002309', master_log_pos='971302894'.
Dec 17 09:44:22 db2051 mysqld[1407]: 2018-12-17  9:44:22 140459707184896 [Note] Previous Using_Gtid=No. New Using_Gtid=Slave_Pos
Dec 17 09:44:22 db2051 mysqld[1407]: 2018-12-17  9:44:22 140456208512768 [Note] Slave I/O thread: Start asynchronous replication to master 'repl@db1068.eqiad.wmnet:3306' in log 'db1068-bin.002309' at position 971302894
Dec 17 09:44:22 db2051 mysqld[1407]: 2018-12-17  9:44:22 140456208512768 [Note] Slave I/O thread: connected to master 'repl@db1068.eqiad.wmnet:3306',replication starts at GTID position '171978775-171978775-3839832693,0-180359175-3368394787,171970589-171970589-201132050,180359190-180359190-192195477,180359175-180359175-43143523'

I checked this:

Dec 17 09:44:22 db2051 mysqld[1407]: 2018-12-17  9:44:22 140459707184896 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='db1068.eqiad.wmnet', master_port='3306', master_log_file='db1068-bin.002309', master_log_pos='971303723'. New state master_host='db1068.eqiad.wmnet', master_port='3306', master_log_file='db1068-bin.002309', master_log_pos='971302894'.

I have also checked which transaction happened in between those positions (971303723 and 971302894) that time and I have done a diff for wbc_entity_usage for an eqiad slave and db2051 and it all looks good:

INSERT /* Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages  */ IGNORE INTO `wbc_entity_usage` (eu_page_id,eu_aspect,eu_entity_id) VALUES ('9825973','L.zh-mo','Q1028181'),('9825973','L.zh-hant','Q21488007'),('9825973','L.zh-hk','Q21488007'),('9825973','L.zh-mo','Q21488007'),('9825973','L.zh-hant','Q36524'),('9825973','L.zh-hk','Q36524'),('9825973','L.zh-mo','Q36524'),('9825973','L.zh-hant','Q3658608'),('9825973','L.zh-hk','Q3658608'),('9825973','L.zh-mo','Q3658608'),('9825973','L.zh-hant','Q4205426'),('9825973','L.zh-hk','Q4205426'),('9825973','L.zh-mo','Q4205426'),('9825973','L.zh-mo','Q82955'),('9825973','L.zh-mo','Q84'),('9825973','L.zh-mo','Q935877')
/*!*/;

I am also fully checking the whole wbc_entity_usage table, which will take 5 hours. Will update this comment once it is done.
Update: no differences found

Mentioned in SAL (#wikimedia-operations) [2018-12-17T13:49:27Z] <marostegui> Enable GTID on s3 codfw master db2043 - T211973

GTID enabled on s3 codfw master, db2043 and position before and after looking good:

Dec 17 13:49:47 db2043 mysqld[3352]: 2018-12-17 13:49:47 139902475896576 [Note] Slave SQL thread exiting, replication stopped in log 'db1075-bin.003543' at position 618453462
Dec 17 13:49:47 db2043 mysqld[3352]: 2018-12-17 13:49:47 139924462012160 [Note] Slave I/O thread exiting, read up to log 'db1075-bin.003543', position 618453462
Dec 17 13:49:47 db2043 mysqld[3352]: 2018-12-17 13:49:47 140062539745024 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='db1075.eqiad.wmnet', master_port='3306', master_log_file='db1075-bin.003543', master_log_pos='618453462'. New state master_host='db1075.eqiad.wmnet', master_port='3306', master_log_file='db1075-bin.003543', master_log_pos='618453462'.
Dec 17 13:49:47 db2043 mysqld[3352]: 2018-12-17 13:49:47 140062539745024 [Note] Previous Using_Gtid=No. New Using_Gtid=Slave_Pos
Dec 17 13:49:48 db2043 mysqld[3352]: 2018-12-17 13:49:48 139902475896576 [Note] Slave I/O thread: Start asynchronous replication to master 'repl@db1075.eqiad.wmnet:3306' in log 'db1075-bin.003543' at position 618453462
Dec 17 13:49:48 db2043 mysqld[3352]: 2018-12-17 13:49:48 139902475896576 [Note] Slave I/O thread: connected to master 'repl@db1075.eqiad.wmnet:3306',replication starts at GTID position '180363271-180363271-245332,0-171966669-4075108480,171966669-171966669-3427240963,180359174-180359174-94123433,180363367-180363367-134174373'
Marostegui renamed this task from Check GTID, consistency options and notifications across the fleet to Check GTID, consistency options, notifications across the fleet and db-eqiad.php weights.Dec 17 2018, 2:13 PM
Marostegui updated the task description. (Show Details)

Mentioned in SAL (#wikimedia-operations) [2018-12-18T08:23:10Z] <marostegui> Enable GTID on es2 - T211973

GTID enabled on es2 master (es2016) and all looks good

Dec 18 08:23:36 es2016 mysqld[5766]: 2018-12-18  8:23:36 140618110539520 [Note] Slave SQL thread exiting, replication stopped in log 'es1015-bin.002991' at position 745182040
Dec 18 08:23:36 es2016 mysqld[5766]: 2018-12-18  8:23:36 140507319916288 [Note] Slave I/O thread exiting, read up to log 'es1015-bin.002991', position 745182040
Dec 18 08:23:36 es2016 mysqld[5766]: 2018-12-18  8:23:36 140524179457792 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='es1015.eqiad.wmnet', master_port='3306', master_log_file='es1015-bin.002991', master_log_pos='745182040'. New state master_host='es1015.eqiad.wmnet', master_port='3306', master_log_file='es1015-bin.002991', master_log_pos='745182040'.
Dec 18 08:23:36 es2016 mysqld[5766]: 2018-12-18  8:23:36 140524179457792 [Note] Previous Using_Gtid=No. New Using_Gtid=Slave_Pos
Dec 18 08:23:36 es2016 mysqld[5766]: 2018-12-18  8:23:36 140507319916288 [Note] Slave I/O thread: Start asynchronous replication to master 'repl@es1015.eqiad.wmnet:3306' in log 'es1015-bin.002991' at position 745182040
Dec 18 08:23:37 es2016 mysqld[5766]: 2018-12-18  8:23:37 140507319916288 [Note] Slave I/O thread: connected to master 'repl@es1015.eqiad.wmnet:3306',replication starts at GTID position '0-171966470-411334248,171974840-171974840-75505599,180367401-180367401-77345258,171966470-171966470-356813944'

Mentioned in SAL (#wikimedia-operations) [2018-12-18T08:26:05Z] <marostegui> Enable GTID on es3 - T211973

GTID enabled on es3 master (es2017) and positions look good

Dec 18 08:26:09 es2017 mysqld[3886]: 2018-12-18  8:26:09 139599671289600 [Note] Slave SQL thread exiting, replication stopped in log 'es1017-bin.002989' at position 539807056
Dec 18 08:26:09 es2017 mysqld[3886]: 2018-12-18  8:26:09 139694367176448 [Note] Slave I/O thread exiting, read up to log 'es1017-bin.002989', position 539807056
Dec 18 08:26:10 es2017 mysqld[3886]: 2018-12-18  8:26:10 139667882735360 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='es1017.eqiad.wmnet', master_port='3306', master_log_file='es1017-bin.002989', master_log_pos='539807056'. New state master_host='es1017.eqiad.wmnet', master_port='3306', master_log_file='es1017-bin.002989', master_log_pos='539807056'.
Dec 18 08:26:10 es2017 mysqld[3886]: 2018-12-18  8:26:10 139667882735360 [Note] Previous Using_Gtid=No. New Using_Gtid=Slave_Pos
Dec 18 08:26:10 es2017 mysqld[3886]: 2018-12-18  8:26:10 139599671289600 [Note] Slave I/O thread: Start asynchronous replication to master 'repl@es1017.eqiad.wmnet:3306' in log 'es1017-bin.002989' at position 539807056
Dec 18 08:26:10 es2017 mysqld[3886]: 2018-12-18  8:26:10 139599671289600 [Note] Slave I/O thread: connected to master 'repl@es1017.eqiad.wmnet:3306',replication starts at GTID position '171970747-171970747-336240191,0-180359340-438187354,171974721-171974721-96102919,180355214-180355214-36897786,180359340-180359340-87'

Mentioned in SAL (#wikimedia-operations) [2018-12-18T08:44:00Z] <marostegui> Enable GTID on s7 codfw master (db2040) - T211973

GTID enabled on db2040 (s7 master)

Dec 18 08:44:07 db2040 mysqld[2167]: 2018-12-18  8:44:07 140468267357952 [Note] Slave SQL thread exiting, replication stopped in log 'db1062-bin.001823' at position 1022107387
Dec 18 08:44:07 db2040 mysqld[2167]: 2018-12-18  8:44:07 140609111860992 [Note] Slave I/O thread exiting, read up to log 'db1062-bin.001823', position 1022125212
Dec 18 08:44:07 db2040 mysqld[2167]: 2018-12-18  8:44:07 140465722799872 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='db1062.eqiad.wmnet', master_port='3306', master_log_file='db1062-bin.001823', master_log_pos='1022125212'. New state master_host='db1062.eqiad.wmnet', master_port='3306', master_log_file='db1062-bin.001823', master_log_pos='1022107387'.
Dec 18 08:44:07 db2040 mysqld[2167]: 2018-12-18  8:44:07 140465722799872 [Note] Previous Using_Gtid=No. New Using_Gtid=Slave_Pos
Dec 18 08:44:07 db2040 mysqld[2167]: 2018-12-18  8:44:07 140468267357952 [Note] Slave I/O thread: Start asynchronous replication to master 'repl@db1062.eqiad.wmnet:3306' in log 'db1062-bin.001823' at position 1022107387
Dec 18 08:44:07 db2040 mysqld[2167]: 2018-12-18  8:44:07 140468267357952 [Note] Slave I/O thread: connected to master 'repl@db1062.eqiad.wmnet:3306',replication starts at GTID position '171978767-171978767-3055755911,0-180359185-3359637071,180359185-180359185-71998080,171970590-171970590-196280066,180355111-180355111-131673159'

I have checked all the transactions that were supposed to run in between the BEGIN and the COMMIT and they are all there.

Mentioned in SAL (#wikimedia-operations) [2018-12-19T06:58:18Z] <marostegui> Enable GTID on s1 codfw master (db2048) - T211973

GTID enabled on s1 db2048 and it looks good:

Dec 19 06:58:26 db2048 mysqld[2665]: 2018-12-19  6:58:26 140190713382656 [Note] Slave SQL thread exiting, replication stopped in log 'db1067-bin.001933' at position 452800853
Dec 19 06:58:26 db2048 mysqld[2665]: 2018-12-19  6:58:26 140190721611520 [Note] Slave I/O thread exiting, read up to log 'db1067-bin.001933', position 452800853
Dec 19 06:58:26 db2048 mysqld[2665]: 2018-12-19  6:58:26 140190730041088 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='db1067.eqiad.wmnet', master_port='3306', master_log_file='db1067-bin.001933', master_log_pos='452800853'. New state master_host='db1067.eqiad.wmnet', master_port='3306', master_log_file='db1067-bin.001933', master_log_pos='452800853'.
Dec 19 06:58:26 db2048 mysqld[2665]: 2018-12-19  6:58:26 140190730041088 [Note] Previous Using_Gtid=No. New Using_Gtid=Slave_Pos
Dec 19 06:58:26 db2048 mysqld[2665]: 2018-12-19  6:58:26 140190713382656 [Note] Slave I/O thread: Start asynchronous replication to master 'repl@db1067.eqiad.wmnet:3306' in log 'db1067-bin.001933' at position 452800853
Dec 19 06:58:26 db2048 mysqld[2665]: 2018-12-19  6:58:26 140190713382656 [Note] Slave I/O thread: connected to master 'repl@db1067.eqiad.wmnet:3306',replication starts at GTID position '180355171-180355171-148310907,0-171970637-5484646134,171970637-171970637-2116621969,171978774-171978774-5,180359172-180359172-49702203,171974720-171974720-783079600'

Mentioned in SAL (#wikimedia-operations) [2018-12-19T06:59:58Z] <marostegui> Enable GTID on s8 codfw master (db2045) - T211973

GTID enabled on s8 db2045 and it looks good:

Dec 19 07:00:06 db2045 mysqld[3411]: 2018-12-19  7:00:06 140342229124864 [Note] Slave SQL thread exiting, replication stopped in log 'db1071-bin.007848' at position 641505655
Dec 19 07:00:06 db2045 mysqld[3411]: 2018-12-19  7:00:06 140481657648896 [Note] Slave I/O thread exiting, read up to log 'db1071-bin.007848', position 641505655
Dec 19 07:00:07 db2045 mysqld[3411]: 2018-12-19  7:00:07 140342935766784 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='db1071.eqiad.wmnet', master_port='3306', master_log_file='db1071-bin.007848', master_log_pos='641505655'. New state master_host='db1071.eqiad.wmnet', master_port='3306', master_log_file='db1071-bin.007848', master_log_pos='641505655'.
Dec 19 07:00:07 db2045 mysqld[3411]: 2018-12-19  7:00:07 140342935766784 [Note] Previous Using_Gtid=No. New Using_Gtid=Slave_Pos
Dec 19 07:00:07 db2045 mysqld[3411]: 2018-12-19  7:00:07 140481657648896 [Note] Slave I/O thread: Start asynchronous replication to master 'repl@db1071.eqiad.wmnet:3306' in log 'db1071-bin.007848' at position 641505655
Dec 19 07:00:07 db2045 mysqld[3411]: 2018-12-19  7:00:07 140481657648896 [Note] Slave I/O thread: connected to master 'repl@db1071.eqiad.wmnet:3306',replication starts at GTID position '180359179-180359179-96523837,0-180359179-5734605861,171974884-171974884-1473084269,171978778-171978778-1529781660,171970704-171970704-351094624,171978768-171978768-202416,171978777-171978777-329020349,180359242-180359242-170963125'
Marostegui updated the task description. (Show Details)Dec 19 2018, 7:01 AM

Change 481118 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Remove old comments

https://gerrit.wikimedia.org/r/481118

Change 481118 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Remove old comments

https://gerrit.wikimedia.org/r/481118

Mentioned in SAL (#wikimedia-operations) [2018-12-21T06:55:27Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Remove old comments T211973 (duration: 00m 46s)

Marostegui closed this task as Resolved.Dec 21 2018, 6:59 AM

I have reviewed db-eqiad.php and also db-codfw.php and all the servers that have to be pooled are pooled and the weights also look good to me.

Marostegui updated the task description. (Show Details)Dec 21 2018, 6:59 AM