Page MenuHomePhabricator

Investigate why refreshMessageBlobs is so sloooow
Closed, ResolvedPublic

Description

It was never fast, but it's currently taking a few hours, so it seems to be taking a lot longer than it was before...

Event Timeline

Reedy created this task.Dec 3 2015, 5:17 PM
Reedy raised the priority of this task from to Needs Triage.
Reedy updated the task description. (Show Details)
Reedy added a subscriber: Reedy.
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald TranscriptDec 3 2015, 5:17 PM
Reedy added a comment.Dec 3 2015, 5:23 PM

TODO: Don't do wfWaitForSlaves() on every db row. There's 210003 rows on enwiki

Reedy added a comment.EditedDec 3 2015, 5:25 PM

Other stuff TODO:

  • foreach ( $wgConf->getLocalDatabases() as $wiki ) {} type script
  • Cache filemtime() calls too
Reedy added a comment.EditedDec 3 2015, 9:05 PM

See also T113092

Change 256756 had a related patch set uploaded (by Reedy):
Add maintenance script to remove bogus languages from msg_resource

https://gerrit.wikimedia.org/r/256756

Change 256780 had a related patch set uploaded (by Reedy):
Don't do a wfWaitForSlave() call on every row in refreshMessageBlobs.php

https://gerrit.wikimedia.org/r/256780

Change 256780 merged by jenkins-bot:
Don't do a wfWaitForSlave() call on every row in refreshMessageBlobs.php

https://gerrit.wikimedia.org/r/256780

Change 256864 had a related patch set uploaded (by Reedy):
Don't do a wfWaitForSlave() call on every row in refreshMessageBlobs.php

https://gerrit.wikimedia.org/r/256864

Reedy added a comment.EditedDec 4 2015, 7:53 AM

Another TODO if the table wasn't due to die somepoing soon... It should do batched reading. The only condition it uses is mr_blob != '{}', which I don't know how many rows it filters out... But there's still a lot on enwiki (among others)

mysql:wikiadmin@db1066 [enwiki]> select count(*) from msg_resource;
+----------+
| count(*) |
+----------+
|   155466 |
+----------+
1 row in set (5.00 sec)

mysql:wikiadmin@db1066 [enwiki]> select count(*) from msg_resource where mr_blob = '{}';
+----------+
| count(*) |
+----------+
|   125355 |
+----------+
1 row in set (4.53 sec)
Reedy added a comment.Dec 4 2015, 11:55 AM

Post cleanup

mysql:wikiadmin@db1073 [enwiki]> select count(*) from msg_resource;
+----------+
| count(*) |
+----------+
|    30586 |
+----------+
1 row in set (0.02 sec)

mysql:wikiadmin@db1073 [enwiki]> select count(*) from msg_resource where mr_blob = '{}';
+----------+
| count(*) |
+----------+
|      599 |
+----------+
1 row in set (0.03 sec)

Change 256864 merged by jenkins-bot:
Don't do a wfWaitForSlave() call on every row in refreshMessageBlobs.php

https://gerrit.wikimedia.org/r/256864

Change 256756 abandoned by Reedy:
Add maintenance script to remove bogus languages from msg_resource

Reason:
No longer needed

https://gerrit.wikimedia.org/r/256756

Krinkle closed this task as Resolved.Aug 5 2016, 2:35 AM
Krinkle claimed this task.
Krinkle added a subscriber: Krinkle.

Per T113092, this msg_resource table no longer exists. The maintenance script is much faster now.

T113092: Revise the design of ResourceLoader's MessageBlobStore