Page MenuHomePhabricator
Create Task
Maniphest T106470

Tool Labs enwiki_p replicated database missing rows
Closed, ResolvedPublic
Actions

Description

https://en.wikipedia.org/w/index.php?title=Talk:Stanley_Hauerwas&action=edit is using {{WikiProject Biography}}. You can see that in the text area and in the templates list below the edit interface.

However, Labs doesn't seem to know this:

MariaDB [enwiki_p]> SELECT
    ->   tl_title
    -> FROM page
    -> JOIN templatelinks
    -> ON tl_from = page_id
    -> WHERE page_namespace = 1
    -> AND page_title = 'Stanley_Hauerwas'
    -> AND tl_namespace = 10
    -> AND tl_title LIKE 'WikiProject%';
+---------------------------------+
| tl_title                        |
+---------------------------------+
| WikiProject_United_States       |
| WikiProject_United_States/class |
+---------------------------------+
2 rows in set (0.01 sec)

Compare with the query against the enwiki master (courtesy of @Legoktm):

mysql:wikiadmin@db1052 [enwiki]> SELECT
    ->   tl_title
    -> FROM page
    -> JOIN templatelinks
    -> ON tl_from = page_id
    -> WHERE page_namespace = 1
    -> AND page_title = 'Stanley_Hauerwas'
    -> AND tl_namespace = 10
    -> AND tl_title LIKE 'WikiProject%';
+---------------------------------+
| tl_title                        |
+---------------------------------+
| WikiProjectBannerShell          |
| WikiProject_Biography           |
| WikiProject_Biography/class     |
| WikiProject_Christianity        |
| WikiProject_Christianity/class  |
| WikiProject_Philosophy          |
| WikiProject_Philosophy/class    |
| WikiProject_Theology            |
| WikiProject_United_States       |
| WikiProject_United_States/class |
+---------------------------------+
10 rows in set (0.11 sec)

Yowza.

Related Objects
Search...

Event Timeline

• MZMcBride created this task.Jul 22 2015, 5:33 AM
• MZMcBride raised the priority of this task from to High.
• MZMcBride updated the task description. (Show Details)
• MZMcBride added projects: Toolforge, Cloud-Services.
• MZMcBride added subscribers: • MZMcBride, • Springle, Legoktm.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 22 2015, 5:33 AM
Legoktm added a project: DBA.Jul 22 2015, 5:38 AM
Legoktm set Security to None.
Ricordisamoa subscribed.Jul 22 2015, 8:28 AM
jcrespo renamed this task from Possible Tool Labs database corruption in enwiki_p to Possible Tool Labs database missing rows in enwiki_p.Jul 22 2015, 1:04 PM
jcrespo added a parent task: T50930: Database replication problems - production and labs (tracking).
jcrespo subscribed.
jcrespo added a comment.Jul 22 2015, 1:16 PM

Updating description, as "database corruption" would make the problem worse (data impossible to access). "Data integrity" issues are probably the way to better describe the problem.

I can confirm the issue happens in all labsdb* hosts, and in db1069:3311 and not in other production s1 master or its slaves. db1069 should be fixed, unless those were deleted due to privacy concerns (very unlikely).

• MZMcBride renamed this task from Possible Tool Labs database missing rows in enwiki_p to Tool Labs enwiki_p replicated database missing rows.Jul 22 2015, 1:40 PM
• MZMcBride added a comment.Jul 22 2015, 1:44 PM
In T106470#1470677, @jcrespo wrote:

Updating description, as "database corruption" would make the problem worse (data impossible to access). "Data integrity" issues are probably the way to better describe the problem.

Okay. I don't really care what we call it. :-)

I can confirm the issue happens in all labsdb* hosts, and in db1069:3311 and not in other production s1 master or its slaves. db1069 should be fixed, unless those were deleted due to privacy concerns (very unlikely).

How do we fix the labsdb* hosts? Is a re-import required?

jcrespo added a comment.Jul 22 2015, 2:03 PM

How do we fix the labsdb* hosts? Is a re-import required?

I am doing an informal check to see how badly it is, but only counting the rows takes 20 minutes (it has almost 400M rows). A one time import could be done on the table, but it is complicated to do it without affecting current labs users and without stopping replication.

jcrespo moved this task from Triage to Backlog on the DBA board.Jul 24 2015, 5:35 PM
jcrespo claimed this task.Jul 28 2015, 1:18 PM
jcrespo moved this task from Backlog to In progress on the DBA board.
jcrespo added a comment.Jul 28 2015, 3:43 PM

Sync is in progress. Looking better already.

MariaDB LABS localhost enwiki > SELECT
    -> tl_title
    -> FROM page
    -> JOIN templatelinks
    -> ON tl_from = page_id
    -> WHERE page_namespace = 1
    -> AND page_title = 'Stanley_Hauerwas'
    -> AND tl_namespace = 10
    -> AND tl_title LIKE 'WikiProject%';
+--------------------------------+
| tl_title                       |
+--------------------------------+
| WikiProjectBannerShell         |
| WikiProject_Biography          |
| WikiProject_Biography/class    |
| WikiProject_Christianity       |
| WikiProject_Christianity/class |
| WikiProject_Philosophy         |
| WikiProject_Philosophy/class   |
| WikiProject_Theology           |
+--------------------------------+
8 rows in set (0.02 sec)
jcrespo closed this task as Resolved.EditedJul 29 2015, 7:14 PM

s1-master:

mysql> SELECT
    -> tl_title
    -> FROM page
    -> JOIN templatelinks
    -> ON tl_from = page_id
    -> WHERE page_namespace = 1
    -> AND page_title = 'Stanley_Hauerwas'
    -> AND tl_namespace = 10
    -> AND tl_title LIKE 'WikiProject%';
+---------------------------------+
| tl_title                        |
+---------------------------------+
| WikiProjectBannerShell          |
| WikiProject_Biography           |
| WikiProject_Biography/class     |
| WikiProject_Christianity        |
| WikiProject_Christianity/class  |
| WikiProject_Philosophy          |
| WikiProject_Philosophy/class    |
| WikiProject_Theology            |
| WikiProject_United_States       |
| WikiProject_United_States/class |
+---------------------------------+
10 rows in set (0.00 sec)

labsdb1001:

mysql> SELECT
    -> tl_title
    -> FROM page
    -> JOIN templatelinks
    -> ON tl_from = page_id
    -> WHERE page_namespace = 1
    -> AND page_title = 'Stanley_Hauerwas'
    -> AND tl_namespace = 10
    -> AND tl_title LIKE 'WikiProject%';
+---------------------------------+
| tl_title                        |
+---------------------------------+
| WikiProjectBannerShell          |
| WikiProject_Biography           |
| WikiProject_Biography/class     |
| WikiProject_Christianity        |
| WikiProject_Christianity/class  |
| WikiProject_Philosophy          |
| WikiProject_Philosophy/class    |
| WikiProject_Theology            |
| WikiProject_United_States       |
| WikiProject_United_States/class |
+---------------------------------+
10 rows in set (0.02 sec)

I had to almost re-implement pt-table-sync on bash, but now templatelinks has the exact same contents than production without downtime. I cannot guarantee this will not happen again, due to labs particularities, but we are working on having something with automated checks very soon (T104459).

For the time being, if you see any discrepancy, please report it as a blocker of T50930. Thank you.

• MZMcBride added a comment.Jul 30 2015, 10:45 PM

Sweet, thanks!

• MZMcBride mentioned this in T111371: Potential templatelinks data integrity issue on Tool Labs' enwiki_p.Sep 3 2015, 9:07 PM
• MZMcBride mentioned this in T87716: Missing rows from categorylinks on production servers (dewiki).May 22 2016, 8:19 PM
valhallasw mentioned this in T138967: Labs database replica drift.Jun 29 2016, 5:22 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL