centralauth database on dbstore1002 is out of date, replication stuck?
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Legoktm
	Apr 13 2015, 6:41 PM

Description

mysql:sul@dbstore1002 [centralauth]> select max(gu_id) from globaluser;
+------------+
| max(gu_id) |
+------------+
|   36083321 |
+------------+
1 row in set (0.02 sec)

versus current master:

mysql:wikiadmin@db1033 [centralauth]> select max(gu_id) from globaluser;
+------------+
| max(gu_id) |
+------------+
|   38547313 |
+------------+
1 row in set (0.00 sec)

I had been using that server to generate stats for SUL finalization since it can do joins with centralauth + local wikis and noticed it giving totally weird responses and started investigating further.

Event Timeline

Legoktm created this task.Apr 13 2015, 6:41 PM

Legoktm raised the priority of this task from to Needs Triage.

Legoktm updated the task description. (Show Details)

Legoktm added projects: acl*sre-team, SUL-Finalization.

Legoktm added subscribers: Legoktm, • Springle.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 13 2015, 6:41 PM

Krenair subscribed.Apr 13 2015, 6:46 PM

Nemo_bis subscribed.Apr 13 2015, 6:52 PM

I see dbstore1002 has the wrong replication rules for s7, so wikis are up to date but centralauth is not (likely 66+ days out of date based on uptime, ie, when config was last reloaded).

Related to https://gerrit.wikimedia.org/r/#/c/198292/ which has been fixed only on dbstore1001 so far. Will start a resync on dbstore1002 s7.

Keegan subscribed.Apr 13 2015, 8:19 PM

Resync is done and replication rules corrected.

thanks, looks good to me!

Legoktm reassigned this task from Legoktm to • Springle.Apr 14 2015, 5:29 AM

Legoktm set Security to None.

Keegan moved this task from It's complicated to Done on the SUL-Finalization board.Apr 14 2015, 6:35 AM

Happening again?

mysql:sul@dbstore1002 [centralauth]> select max(gu_id) from globaluser;
+------------+
| max(gu_id) |
+------------+
|   41826616 |
+------------+
1 row in set (0.00 sec)

mysql:wikiadmin@db1033 [centralauth]> select max(gu_id) from globaluser;
+------------+
| max(gu_id) |
+------------+
|   41984206 |
+------------+
1 row in set (0.00 sec)

Something weird is happening.

Replication for s7 is running, the replication rules (added earlier in this bug) still exist, the statements are appearing in dbstore1002 relay log, the centralauth tables seem intact ... but no new data for a week. Other shards seem ok.

A week ago dbstore1002 did have an outage due to analytics queries filling up /tmp mount, which required a mysqld restart. Perhaps that's connected somehow. No more useful information right now.

It's all of S7 affected:

arwiki
cawiki
eswiki
fawiki
frwiktionary
hewiki
huwiki
kowiki
metawiki
rowiki
ukwiki
viwiki

...and centralauth.

The other wikis all show up to date data.

Keegan moved this task from Done to It's complicated on the SUL-Finalization board.May 12 2015, 9:10 PM

Glaisher subscribed.Jun 18 2015, 5:17 AM

Oops, sorry, this was fixed a while back.

Keegan moved this task from It's complicated to Done on the SUL-Finalization board.Jul 16 2015, 12:04 AM

Restricted Application added subscribers: Steinsplitter, Matanya. · View Herald TranscriptJul 16 2015, 12:04 AM

centralauth database on dbstore1002 is out of date, replication stuck?Closed, ResolvedPublicActions

Description

Event Timeline

centralauth database on dbstore1002 is out of date, replication stuck?
Closed, ResolvedPublic
Actions