Page MenuHomePhabricator

Tool Labs enwiki_p replicated database missing rows
Closed, ResolvedPublic

Description

https://en.wikipedia.org/w/index.php?title=Talk:Stanley_Hauerwas&action=edit is using {{WikiProject Biography}}. You can see that in the text area and in the templates list below the edit interface.

However, Labs doesn't seem to know this:

MariaDB [enwiki_p]> SELECT
    ->   tl_title
    -> FROM page
    -> JOIN templatelinks
    -> ON tl_from = page_id
    -> WHERE page_namespace = 1
    -> AND page_title = 'Stanley_Hauerwas'
    -> AND tl_namespace = 10
    -> AND tl_title LIKE 'WikiProject%';
+---------------------------------+
| tl_title                        |
+---------------------------------+
| WikiProject_United_States       |
| WikiProject_United_States/class |
+---------------------------------+
2 rows in set (0.01 sec)

Compare with the query against the enwiki master (courtesy of @Legoktm):

mysql:wikiadmin@db1052 [enwiki]> SELECT
    ->   tl_title
    -> FROM page
    -> JOIN templatelinks
    -> ON tl_from = page_id
    -> WHERE page_namespace = 1
    -> AND page_title = 'Stanley_Hauerwas'
    -> AND tl_namespace = 10
    -> AND tl_title LIKE 'WikiProject%';
+---------------------------------+
| tl_title                        |
+---------------------------------+
| WikiProjectBannerShell          |
| WikiProject_Biography           |
| WikiProject_Biography/class     |
| WikiProject_Christianity        |
| WikiProject_Christianity/class  |
| WikiProject_Philosophy          |
| WikiProject_Philosophy/class    |
| WikiProject_Theology            |
| WikiProject_United_States       |
| WikiProject_United_States/class |
+---------------------------------+
10 rows in set (0.11 sec)

Yowza.

Event Timeline

MZMcBride raised the priority of this task from to High.
MZMcBride updated the task description. (Show Details)
MZMcBride added subscribers: MZMcBride, Springle, Legoktm.
jcrespo renamed this task from Possible Tool Labs database corruption in enwiki_p to Possible Tool Labs database missing rows in enwiki_p.Jul 22 2015, 1:04 PM
jcrespo added a subscriber: jcrespo.

Updating description, as "database corruption" would make the problem worse (data impossible to access). "Data integrity" issues are probably the way to better describe the problem.

I can confirm the issue happens in all labsdb* hosts, and in db1069:3311 and not in other production s1 master or its slaves. db1069 should be fixed, unless those were deleted due to privacy concerns (very unlikely).

MZMcBride renamed this task from Possible Tool Labs database missing rows in enwiki_p to Tool Labs enwiki_p replicated database missing rows.Jul 22 2015, 1:40 PM

Updating description, as "database corruption" would make the problem worse (data impossible to access). "Data integrity" issues are probably the way to better describe the problem.

Okay. I don't really care what we call it. :-)

I can confirm the issue happens in all labsdb* hosts, and in db1069:3311 and not in other production s1 master or its slaves. db1069 should be fixed, unless those were deleted due to privacy concerns (very unlikely).

How do we fix the labsdb* hosts? Is a re-import required?

How do we fix the labsdb* hosts? Is a re-import required?

I am doing an informal check to see how badly it is, but only counting the rows takes 20 minutes (it has almost 400M rows). A one time import could be done on the table, but it is complicated to do it without affecting current labs users and without stopping replication.

jcrespo moved this task from Backlog to In progress on the DBA board.

Sync is in progress. Looking better already.

MariaDB LABS localhost enwiki > SELECT
    -> tl_title
    -> FROM page
    -> JOIN templatelinks
    -> ON tl_from = page_id
    -> WHERE page_namespace = 1
    -> AND page_title = 'Stanley_Hauerwas'
    -> AND tl_namespace = 10
    -> AND tl_title LIKE 'WikiProject%';
+--------------------------------+
| tl_title                       |
+--------------------------------+
| WikiProjectBannerShell         |
| WikiProject_Biography          |
| WikiProject_Biography/class    |
| WikiProject_Christianity       |
| WikiProject_Christianity/class |
| WikiProject_Philosophy         |
| WikiProject_Philosophy/class   |
| WikiProject_Theology           |
+--------------------------------+
8 rows in set (0.02 sec)
jcrespo closed this task as Resolved.EditedJul 29 2015, 7:14 PM

s1-master:

mysql> SELECT
    -> tl_title
    -> FROM page
    -> JOIN templatelinks
    -> ON tl_from = page_id
    -> WHERE page_namespace = 1
    -> AND page_title = 'Stanley_Hauerwas'
    -> AND tl_namespace = 10
    -> AND tl_title LIKE 'WikiProject%';
+---------------------------------+
| tl_title                        |
+---------------------------------+
| WikiProjectBannerShell          |
| WikiProject_Biography           |
| WikiProject_Biography/class     |
| WikiProject_Christianity        |
| WikiProject_Christianity/class  |
| WikiProject_Philosophy          |
| WikiProject_Philosophy/class    |
| WikiProject_Theology            |
| WikiProject_United_States       |
| WikiProject_United_States/class |
+---------------------------------+
10 rows in set (0.00 sec)

labsdb1001:

mysql> SELECT
    -> tl_title
    -> FROM page
    -> JOIN templatelinks
    -> ON tl_from = page_id
    -> WHERE page_namespace = 1
    -> AND page_title = 'Stanley_Hauerwas'
    -> AND tl_namespace = 10
    -> AND tl_title LIKE 'WikiProject%';
+---------------------------------+
| tl_title                        |
+---------------------------------+
| WikiProjectBannerShell          |
| WikiProject_Biography           |
| WikiProject_Biography/class     |
| WikiProject_Christianity        |
| WikiProject_Christianity/class  |
| WikiProject_Philosophy          |
| WikiProject_Philosophy/class    |
| WikiProject_Theology            |
| WikiProject_United_States       |
| WikiProject_United_States/class |
+---------------------------------+
10 rows in set (0.02 sec)

I had to almost re-implement pt-table-sync on bash, but now templatelinks has the exact same contents than production without downtime. I cannot guarantee this will not happen again, due to labs particularities, but we are working on having something with automated checks very soon (T104459).

For the time being, if you see any discrepancy, please report it as a blocker of T50930. Thank you.