Data missing from June 11/12 on s3.labsdb
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	dungodung
	Oct 14 2015, 8:47 PM

Description

A fellow Wikipedian noticed while using this tool on Tool Labs that there are some articles of his missing in the list.

After some inspection, I found out that there's about 19 hours' worth of missing revisions in revision table on the Labs replica (s3.labsdb) around June 12, 2015 (from about 8:20pm on June 11 to about 3:20 pm on June 12).

MariaDB [srwiki_p]> select max(rev_timestamp) as max_rev from revision where rev_timestamp like '20150611%';
+----------------+
| max_rev        |
+----------------+
| 20150611202101 |
+----------------+

MariaDB [srwiki_p]> select min(rev_timestamp) as min_rev from revision where rev_timestamp like '20150612%';
+----------------+
| min_rev        |
+----------------+
| 20150612152251 |
+----------------+

I tested this with srwiki, but apparently, it's the case with all the databases on s3.labsdb. Other servers don't seem to be affected.

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		jcrespo	T50930 Database replication problems - production and labs (tracking)
		Resolved		jcrespo	T115517 Data missing from June 11/12 on s3.labsdb

Event Timeline

dungodung created this task.Oct 14 2015, 8:47 PM

dungodung raised the priority of this task from to Needs Triage.

dungodung updated the task description. (Show Details)

dungodung added a project: Cloud-Services.

dungodung subscribed.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 14 2015, 8:47 PM

Krenair added a project: DBA.Oct 14 2015, 8:49 PM

Krenair subscribed.

• chasemp triaged this task as Low priority.Nov 30 2015, 4:42 PM

• chasemp subscribed.

dungodung mentioned this in T129432: Lost database changes on s2 for 3 hours on labs replicas.Mar 17 2016, 4:47 PM

jcrespo subscribed.Mar 17 2016, 5:07 PM

jcrespo moved this task from Triage to Backlog on the DBA board.Mar 17 2016, 5:12 PM

Ricordisamoa subscribed.Mar 18 2016, 8:40 AM

jcrespo moved this task from Backlog to In progress on the DBA board.Mar 19 2016, 2:14 PM

jcrespo mentioned this in T131266: Querying the logging table on labs is slow.Mar 30 2016, 6:41 PM

jcrespo moved this task from In progress to Pending comment on the DBA board.Apr 1 2016, 4:17 PM

valhallasw mentioned this in T133469: Discrepancy between labsdb replicas of arwiki_p.user_groups.Apr 23 2016, 8:21 PM

T133469 concerns a very similar issue but on an s7 wiki - I know that the DNS setup in labs encourages use of the same underlying server for both of those:

krenair@tools-bastion-03:~$ host s3.labsdb
s3.labsdb has address 10.64.37.5
krenair@tools-bastion-03:~$ host s7.labsdb
s7.labsdb has address 10.64.37.5
krenair@tools-bastion-03:~$ host 10.64.37.5
5.37.64.10.in-addr.arpa domain name pointer labsdb1003.eqiad.wmnet.

So perhaps the issue is wider in scope than just s3, but also covering s7, and potentially also s5/s6 (which are also typically served by labsdb1003)?

So perhaps the issue is wider in scope than just s3, but also covering s7, and potentially also s5/s6 (which are also typically served by labsdb1003)?

No, this only affected s3 shard- it was a replication problem for that shard only. Now, I cannot guarantee the accuracy of other shards, that is why I am reimporting all, starting from enwiki/s1. As I said, it can take weeks (months?) to reimport all tables due to labs filtering, one by one, but it is happening now.

In T115517#2234279, @jcrespo wrote:

Now, I cannot guarantee the accuracy of other shards, that is why I am reimporting all, starting from enwiki/s1.

I can confirm there was some sort of similar issue on the enwiki replica.

tools.anomiebot@tools-bastion-03:~$ sql enwiki 'select * from page where page_id>=50271944 and page_id<=50275119;'
page_id	page_namespace	page_title	page_restrictions	page_counter	page_is_redirect	page_is_new	page_random	page_touched	page_links_updated	page_latest	page_len	page_content_model
50271944	10	2011-12_Football_League_Two_table		0	1	1	0.848858636718	20160422091535	20160422091539	716545485	141	wikitext
50275119	10	1968-69_in_English_football		0	1	1	0.567951926164	20160422124451	NULL	716566974	129	wikitext

That should have output 3069 lines, not just two. Also,

MariaDB [enwiki_p]> select rev_id, rev_page, rev_timestamp from revision where rev_id>=716545495 and rev_id<=716566961;
+-----------+----------+----------------+
| rev_id    | rev_page | rev_timestamp  |
+-----------+----------+----------------+
| 716545495 | 12264894 | 20160422091542 |
| 716561763 | 50201281 | 20160422115519 |
| 716563248 | 50201281 | 20160422121036 |
| 716566961 |  8368016 | 20160422124443 |
+-----------+----------+----------------+
4 rows in set (0.00 sec)

There should be 20997 rows there, not just four.

jcrespo moved this task from Pending comment to In progress on the DBA board.May 4 2016, 9:50 AM

valhallasw mentioned this in T138967: Labs database replica drift.Jun 29 2016, 5:22 PM

• AlexMonk-WMF added a parent task: T50930: Database replication problems - production and labs (tracking).Aug 26 2016, 8:10 AM

These are the results on the new replica service servers:

root@localhost[srwiki_p]> select max(rev_timestamp) as max_rev, min(rev_timestamp) as min_rev from revision where rev_timestamp like '20150611%';
+----------------+----------------+
| max_rev        | min_rev        |
+----------------+----------------+
| 20150611235958 | 20150611000406 |
+----------------+----------------+
1 row in set (0.00 sec)

For enwiki:

master:

root@localhost[enwiki]> select count(*) from page where page_id>=50271944 and page_id<=50275119;
+----------+
| count(*) |
+----------+
|     2999 |
+----------+
1 row in set (0.01 sec)

new replica servers:

root@localhost[enwiki_p]> select count(*) from page where page_id>=50271944 and page_id<=50275119;
+----------+
| count(*) |
+----------+
|     2999 |
+----------+
1 row in set (0.00 sec)

I consider this as solved, documentation and announcement on how the new system will work will arrive soon (please be patient). Please ask if you really need the fixed data sooner.

Aklapper removed a subscriber: Anomie.Oct 16 2020, 5:43 PM

Data missing from June 11/12 on s3.labsdbClosed, ResolvedPublicActions

Description

Related ObjectsSearch...

Event Timeline

Data missing from June 11/12 on s3.labsdb
Closed, ResolvedPublic
Actions

Related Objects
Search...