Page MenuHomePhabricator
Create Task
Maniphest T161294

run pt-tablechecksum on s5/s8
Closed, ResolvedPublic
Actions

Description

s2 (T154485) and s6 (T160509) has been checksumed.
It is time to checksum s5 for the following hosts to be decommissioned: db1049,db1026,db1045

The __wmf_checksums is placed on every database of the shard. That means

dewiki.__wmf_checksums
wikidatawiki.__ wmf_checksums

Details

Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedNoneT134476 Decommission old coredb machines (<=db1050)
 Resolved• MarosteguiT161294 run pt-tablechecksum on s5/s8

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a comment.Dec 14 2017, 7:02 AM

Change 398216 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1109 and db1101:3318

https://gerrit.wikimedia.org/r/398216

gerritbot added a comment.Dec 14 2017, 7:06 AM

Change 398216 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1109 and db1101:3318

https://gerrit.wikimedia.org/r/398216

Stashbot added a comment.Dec 14 2017, 7:08 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-14T07:08:09Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1101:3318 and db1109 - T161294 (duration: 01m 07s)

Stashbot added a comment.Dec 14 2017, 7:08 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-14T07:08:49Z] <marostegui> Stop replication in sync on db1109 and db1101:3318 - T161294

• Marostegui added a comment.EditedDec 14 2017, 12:19 PM

I have fixed dewiki.archive across the board.
Next on the list:
dewiki.change_tag
dewiki.geo_tags
dewiki.tag_summary

Once those are done, I will do another checksum to verify.

• Marostegui added a comment.Dec 14 2017, 3:31 PM

The following tables have been fixed on dewiki:

dewiki.change_tag
dewiki.geo_tags

• Marostegui renamed this task from run pt-tablechecksum on s5 to run pt-tablechecksum on s5/s8.Dec 14 2017, 4:57 PM
• Marostegui added a comment.Dec 14 2017, 5:25 PM

I have fixed tag_summary too.

dewiki should be consistent now on s5 and s8 - but I will do some more checks tomorrow before starting with wikidatawiki.

Stashbot added a comment.Dec 15 2017, 6:26 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-15T06:26:17Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1101:3318 and db1109 - T161294 (duration: 01m 16s)

• Marostegui added a comment.Dec 15 2017, 7:48 AM

wikidatawiki.archive has been fixed.

jcrespo added a comment.Dec 16 2017, 3:21 PM

tag_summary broke on db1100 and on most or all of codfw- we need to reload that table almost everywhere; also check for existing code issues.

For the record, I did:

delete from tag_summary WHERE ts_id=17666046;
UPDATE tag_summary SET ts_id = 17666046 WHERE ts_rev_id = 182949546;

PKs are broken, and probably now missing a row.

• Marostegui added a comment.Dec 16 2017, 5:22 PM

I started with wikidatawiki.change_tag on Friday (and still fixing it - lots of differences across all the servers). Hopefully I will be done by tomorrow. Next on the list was actually wikidatawiki.tag_summary, which I was expecting to start with on Monday. Too late looks like :-(

• Marostegui added a comment.Dec 16 2017, 5:26 PM
In T161294#3842418, @jcrespo wrote:

tag_summary broke on db1100 and on most or all of codfw- we need to reload that table almost everywhere; also check for existing code issues.

For the record, I did:

delete from tag_summary WHERE ts_id=17666046;
UPDATE tag_summary SET ts_id = 17666046 WHERE ts_rev_id = 182949546;

PKs are broken, and probably now missing a row.

For the record, servers that crashed:

db1100
14:47 < icinga-wm> PROBLEM - MariaDB Slave Lag: s5 on db2089 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 605.33 seconds
14:48 < icinga-wm> PROBLEM - MariaDB Slave Lag: s5 on db2059 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 615.59 seconds
14:48 < icinga-wm> PROBLEM - MariaDB Slave Lag: s5 on db2075 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 618.47 seconds
14:48 < icinga-wm> PROBLEM - MariaDB Slave Lag: s8 on db2085 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 620.34 seconds
14:48 < icinga-wm> PROBLEM - MariaDB Slave Lag: s5 on db2084 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 620.51 seconds
14:48 < icinga-wm> PROBLEM - MariaDB Slave Lag: s8 on db2081 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 621.65 seconds
14:48 < icinga-wm> PROBLEM - MariaDB Slave Lag: s8 on db2086 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 621.67 seconds
14:48 < icinga-wm> PROBLEM - MariaDB Slave Lag: s5 on db2052 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 622.19 seconds
14:48 < icinga-wm> PROBLEM - MariaDB Slave Lag: s5 on db2066 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 638.74 seconds
14:48 < icinga-wm> PROBLEM - MariaDB Slave Lag: s8 on db2045 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 641.27 seconds
14:48 < icinga-wm> PROBLEM - MariaDB Slave Lag: s8 on db2080 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 647.47 seconds
• Marostegui added a comment.Dec 17 2017, 9:30 AM

wikidatawiki.change_tag is finished.

Tomorrow I will re-import tag_summary on all the hosts that failed yesterday and continue with the next wikidata tables. Once everything is done, I will run another diff to double check possible drifts on both, dewiki and wikidata

jcrespo added a comment.Dec 17 2017, 2:50 PM

Also db1100 and dbstore1001.

• Marostegui added a comment.Dec 17 2017, 2:53 PM
In T161294#3843160, @jcrespo wrote:

Also db1100 and dbstore1001.

Yeah db1100 I wrote it down above - right above the codfw hosts, dbstore1001 I was expecting it to crash today :(

• Marostegui added a comment.EditedDec 17 2017, 5:19 PM

I have fixed wikidatawiki.text

• Marostegui added a comment.Dec 17 2017, 5:43 PM

wikidatawiki.wb_items_per_site done.

gerritbot added a comment.Dec 18 2017, 6:11 AM

Change 398789 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1106

https://gerrit.wikimedia.org/r/398789

gerritbot added a comment.Dec 18 2017, 6:15 AM

Change 398789 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1106

https://gerrit.wikimedia.org/r/398789

Stashbot added a comment.Dec 18 2017, 6:17 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-18T06:17:21Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1106 - T161294 (duration: 00m 57s)

Stashbot added a comment.Dec 18 2017, 6:17 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-18T06:17:32Z] <marostegui> Stop replication in sync on db1106 and db1100 - T161294

Stashbot added a comment.Dec 18 2017, 6:28 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-18T06:28:59Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1106 - T161294 (duration: 00m 56s)

Stashbot added a comment.Dec 18 2017, 8:58 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-18T08:58:50Z] <marostegui> Stop replication in sync on db1100 and db2052 - T161294

gerritbot added a comment.Dec 18 2017, 10:25 AM

Change 398804 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1109

https://gerrit.wikimedia.org/r/398804

gerritbot added a comment.Dec 18 2017, 10:28 AM

Change 398804 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1109

https://gerrit.wikimedia.org/r/398804

Stashbot added a comment.Dec 18 2017, 10:30 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-18T10:30:07Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1109 - T161294 (duration: 00m 56s)

Stashbot added a comment.Dec 18 2017, 10:30 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-18T10:30:19Z] <marostegui> Stop replication on db1109 and db2045 in sync - T161294

Stashbot added a comment.Dec 18 2017, 10:46 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-18T10:46:44Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1109 - T161294 (duration: 00m 56s)

gerritbot added a comment.Dec 18 2017, 12:52 PM

Change 398827 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1106

https://gerrit.wikimedia.org/r/398827

gerritbot added a comment.Dec 18 2017, 12:57 PM

Change 398827 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1106

https://gerrit.wikimedia.org/r/398827

Stashbot added a comment.Dec 18 2017, 12:58 PM

Mentioned in SAL (#wikimedia-operations) [2017-12-18T12:58:03Z] <marostegui> Stop replication in sync on db1106 and db1100 - T161294

Stashbot added a comment.Dec 18 2017, 1:00 PM

Mentioned in SAL (#wikimedia-operations) [2017-12-18T13:00:47Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1106 - T161294 (duration: 03m 03s)

Stashbot added a comment.Dec 18 2017, 1:10 PM

Mentioned in SAL (#wikimedia-operations) [2017-12-18T13:10:13Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1106 - T161294 (duration: 03m 06s)

Stashbot added a comment.Dec 18 2017, 1:18 PM

Mentioned in SAL (#wikimedia-operations) [2017-12-18T13:18:36Z] <marostegui> Stop replication in sync on db1100 and db2052 - T161294

Stashbot added a comment.Dec 18 2017, 1:29 PM

Mentioned in SAL (#wikimedia-operations) [2017-12-18T13:29:56Z] <marostegui> Stop replicaiton in sync on db1109 and db2045 - T161294

Stashbot added a comment.Dec 18 2017, 3:37 PM

Mentioned in SAL (#wikimedia-operations) [2017-12-18T15:37:32Z] <marostegui> Stop db1100 and dbstore1002 in sync - T161294

gerritbot added a comment.Dec 19 2017, 6:21 AM

Change 399138 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1106

https://gerrit.wikimedia.org/r/399138

gerritbot added a comment.Dec 19 2017, 6:25 AM

Change 399138 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1106

https://gerrit.wikimedia.org/r/399138

Stashbot added a comment.Dec 19 2017, 6:26 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-19T06:26:51Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1106 - T161294 (duration: 00m 53s)

Stashbot added a comment.Dec 19 2017, 6:29 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-19T06:29:07Z] <marostegui> Stop replication in sync on db1100 and db1106 - T161294

Stashbot added a comment.Dec 19 2017, 6:40 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-19T06:40:21Z] <marostegui> Stop replication in sync on db1106 and dbstore1002 s5 - T161294

Stashbot added a comment.Dec 19 2017, 6:51 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-19T06:51:28Z] <marostegui> Stop replication in sync on db1106 and db2052 - T161294

Stashbot added a comment.Dec 19 2017, 7:00 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-19T07:00:03Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1106 - T161294 (duration: 00m 51s)

Stashbot added a comment.Dec 19 2017, 8:28 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-19T08:28:44Z] <marostegui> Stop replication in sync on db2045 and db1109 - T161294

• Marostegui added a comment.Dec 19 2017, 8:36 AM

I am fairly confident that dewiki is now really consistent.
Will keep doing a bit more tests to check it out.

Still fixing and checking wikidata

Stashbot added a comment.Dec 19 2017, 8:56 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-19T08:56:53Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1106 - T161294 (duration: 00m 51s)

gerritbot added a comment.Dec 19 2017, 9:03 AM

Change 399148 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1097:3315

https://gerrit.wikimedia.org/r/399148

gerritbot added a comment.Dec 19 2017, 9:11 AM

Change 399148 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1097:3315

https://gerrit.wikimedia.org/r/399148

Stashbot added a comment.Dec 19 2017, 9:12 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-19T09:12:38Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1097:3315 - T161294 (duration: 00m 51s)

Stashbot added a comment.Dec 19 2017, 3:01 PM

Mentioned in SAL (#wikimedia-operations) [2017-12-19T15:01:41Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1106 - T161294 (duration: 00m 52s)

gerritbot added a comment.Dec 19 2017, 3:04 PM

Change 399191 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1109, db1099:3318

https://gerrit.wikimedia.org/r/399191

gerritbot added a comment.Dec 19 2017, 3:09 PM

Change 399191 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1109, db1099:3318

https://gerrit.wikimedia.org/r/399191

Stashbot added a comment.Dec 19 2017, 3:10 PM

Mentioned in SAL (#wikimedia-operations) [2017-12-19T15:10:45Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1099:3318 and db1109 - T161294 (duration: 00m 51s)

Stashbot added a comment.Dec 19 2017, 3:10 PM

Mentioned in SAL (#wikimedia-operations) [2017-12-19T15:10:54Z] <marostegui> Stop replication in sync on db1109 and db1099:3318 - https://phabricator.wikimedia.org/T161294

Stashbot added a comment.Dec 19 2017, 4:54 PM

Mentioned in SAL (#wikimedia-operations) [2017-12-19T16:54:17Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1097:3315 - T161294 (duration: 00m 51s)

gerritbot added a comment.Dec 20 2017, 6:47 AM

Change 399342 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1101:3318, repool s8 db1099

https://gerrit.wikimedia.org/r/399342

gerritbot added a comment.Dec 20 2017, 6:52 AM

Change 399342 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1101:3318, repool s8 db1099

https://gerrit.wikimedia.org/r/399342

Stashbot added a comment.Dec 20 2017, 6:53 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-20T06:53:36Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1099:3318 depool db1101:3318 - T161294 (duration: 00m 51s)

Stashbot added a comment.Dec 20 2017, 6:53 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-20T06:53:47Z] <marostegui> Stop replication in sync on db1101:3318 and db1109 - T161294

gerritbot added a comment.Dec 20 2017, 7:01 AM

Change 399343 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1096:3315

https://gerrit.wikimedia.org/r/399343

gerritbot added a comment.Dec 20 2017, 7:05 AM

Change 399343 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1096:3315

https://gerrit.wikimedia.org/r/399343

Stashbot added a comment.Dec 20 2017, 7:08 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-20T07:08:22Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1096:3315 - T161294 (duration: 00m 51s)

Stashbot added a comment.Dec 20 2017, 7:09 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-20T07:09:01Z] <marostegui> Stop replication in sync on db1096:3315 and db1100 - T161294

Stashbot added a comment.Dec 20 2017, 8:11 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-20T08:11:37Z] <marostegui> Stop replication in sync on db1100 and dbstore1002 - T161294

Stashbot added a comment.Dec 20 2017, 11:11 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-20T11:11:19Z] <marostegui> Stop replication in sync on db1100 and db1071 - T161294

gerritbot added a comment.Dec 20 2017, 12:22 PM

Change 399387 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad: Repool db1101:3318, db1109

https://gerrit.wikimedia.org/r/399387

gerritbot added a comment.Dec 20 2017, 12:26 PM

Change 399387 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad: Repool db1101:3318, db1109

https://gerrit.wikimedia.org/r/399387

Stashbot added a comment.Dec 20 2017, 12:27 PM

Mentioned in SAL (#wikimedia-operations) [2017-12-20T12:27:44Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1101:3318 and db1109 - T161294 (duration: 00m 51s)

Stashbot added a comment.Dec 20 2017, 12:34 PM

Mentioned in SAL (#wikimedia-operations) [2017-12-20T12:34:50Z] <marostegui> Enable notifications for db1100 - T161294

• Marostegui mentioned this in T183341: New item fails (Special and WEF tool).Dec 20 2017, 12:55 PM
gerritbot added a comment.Dec 21 2017, 6:28 AM

Change 399564 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1109

https://gerrit.wikimedia.org/r/399564

gerritbot added a comment.Dec 21 2017, 6:32 AM

Change 399564 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1109

https://gerrit.wikimedia.org/r/399564

Stashbot added a comment.Dec 21 2017, 6:33 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-21T06:33:43Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1109 - T161294 (duration: 00m 51s)

Stashbot added a comment.Dec 21 2017, 6:35 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-21T06:35:37Z] <marostegui> Stop replication in sync db1100 and db1071 - T161294

Stashbot added a comment.Dec 21 2017, 6:42 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-21T06:42:33Z] <marostegui> Stop replication in sync on db1100 - db2052 - T161294

Stashbot added a comment.Dec 21 2017, 6:50 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-21T06:50:10Z] <marostegui> Stop replication in sync on dbstore1002 - db1100 - T161294

gerritbot added a comment.Dec 21 2017, 7:09 AM

Change 399566 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1109 and db1096:3315

https://gerrit.wikimedia.org/r/399566

gerritbot added a comment.Dec 21 2017, 7:15 AM

Change 399566 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Repool db1109 and db1096:3315

https://gerrit.wikimedia.org/r/399566

Stashbot added a comment.Dec 21 2017, 7:16 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-21T07:16:34Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1109 and db1096:3315 - T161294 (duration: 00m 51s)

Stashbot added a comment.Dec 21 2017, 7:37 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-21T07:37:23Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Slowly repool db1100- T161294 (duration: 00m 52s)

gerritbot added a comment.Dec 21 2017, 8:27 AM

Change 399581 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1100

https://gerrit.wikimedia.org/r/399581

gerritbot added a comment.Dec 21 2017, 8:34 AM

Change 399581 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1100

https://gerrit.wikimedia.org/r/399581

Stashbot added a comment.Dec 21 2017, 8:34 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-21T08:34:50Z] <marostegui> Stop replication in sync on db1100 and dbstore1002 - T161294

Stashbot added a comment.Dec 21 2017, 8:35 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-21T08:35:20Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1100 - T161294 (duration: 00m 51s)

Stashbot added a comment.Dec 21 2017, 9:22 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-21T09:22:34Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1100 - T161294 (duration: 00m 51s)

gerritbot added a comment.Dec 21 2017, 9:34 AM

Change 399590 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Increase traffic for db1100

https://gerrit.wikimedia.org/r/399590

gerritbot added a comment.Dec 21 2017, 9:38 AM

Change 399590 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Increase traffic for db1100

https://gerrit.wikimedia.org/r/399590

Stashbot added a comment.Dec 21 2017, 9:39 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-21T09:39:51Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Increase traffic for db1100 - T161294 (duration: 00m 51s)

gerritbot added a comment.Dec 21 2017, 9:58 AM

Change 399597 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Increase traffic for db1100

https://gerrit.wikimedia.org/r/399597

gerritbot added a comment.Dec 21 2017, 10:02 AM

Change 399597 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Increase traffic for db1100

https://gerrit.wikimedia.org/r/399597

Stashbot added a comment.Dec 21 2017, 10:03 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-21T10:03:20Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Increase traffic for db1100 and start restoring original weight for db1082 - T161294 (duration: 00m 48s)

gerritbot added a comment.Dec 21 2017, 10:24 AM

Change 399599 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Fully repool db1100,db1082

https://gerrit.wikimedia.org/r/399599

gerritbot added a comment.Dec 21 2017, 10:29 AM

Change 399599 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Fully repool db1100,db1082

https://gerrit.wikimedia.org/r/399599

Stashbot added a comment.Dec 21 2017, 10:30 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-21T10:30:48Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Restore db1082 and db1100 original weight - T161294 (duration: 00m 51s)

• Marostegui added a comment.Dec 21 2017, 1:07 PM

After fixing lots of differences on wikidata tables, it is impossible to say that it is fixed 100% but it is in a much better consistent state that it was before and I believe it is quite consistent now.
I will do some more checks around next week before closing this task.

• Marostegui closed this task as Resolved.Dec 27 2017, 9:23 AM

The last tests I have done look good, so I am going to close this as resolved.
It is obviously impossible to say that this is fixed 100%, but undoubtedly after fixing lots and lots of rows on numerous tables, we are in a lot better state that we were before.

• Marostegui mentioned this in T183735: Check data consistency across production shards.Dec 28 2017, 7:54 AM
Anomie mentioned this in T181731: Run maintenance/cleanupUsersWithNoId.php on all wikis.Jan 9 2018, 2:51 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL