Page MenuHomePhabricator

timeouts and memory limits on translatewiki.net
Closed, ResolvedPublic

Description

We updated translatewiki.net to master branch yesterday and we started seeing timeouts and memory limits:

Maximum execution time of 15 seconds exceeded at PPNode_Hash_Tree.php:219
Maximum execution time of 15 seconds exceeded at Title.php:246
Maximum execution time of 15 seconds exceeded at Title.php:3038
Maximum execution time of 15 seconds exceeded at Title.php:3041
Maximum execution time of 15 seconds exceeded at Title.php:701
Maximum execution time of 15 seconds exceeded at Title.php:225
Maximum execution time of 15 seconds exceeded at Title.php:242
Maximum execution time of 15 seconds exceeded at Title.php:698
Maximum execution time of 15 seconds exceeded at Title.php:230
Maximum execution time of 15 seconds exceeded at Title.php:242
Maximum execution time of 15 seconds exceeded at Title.php:242
Maximum execution time of 15 seconds exceeded at Title.php:242
Maximum execution time of 15 seconds exceeded at Title.php:3038
Maximum execution time of 15 seconds exceeded at MessageGroupCache.php:72
Allowed memory size of 314572800 bytes exhausted (tried to allocate 548864 bytes) in DatabaseMysqli.php:49
Allowed memory size of 314572800 bytes exhausted (tried to allocate 4096 bytes) in MysqliResultWrapper.php:47
Allowed memory size of 314572800 bytes exhausted (tried to allocate 2621440 bytes) in MessageCollection.php:371
Allowed memory size of 314572800 bytes exhausted (tried to allocate 2621440 bytes) in MessageCollection.php:371

This leads me to believe the above patch doesn't fix the issue fully.

At least some of the translatewiki.net issues seem to be caused by rMW45c692eeeff3: Stop accessing members of the Title class publicly making Title object construction slower (it's used a lot, so it is very performance sensitive)

Event Timeline

Change 744881 had a related patch set uploaded (by Daniel Kinzler; author: Daniel Kinzler):

[mediawiki/core@master] DeprecationHelper: avoid closures

https://gerrit.wikimedia.org/r/744881

Benchmark wmf.9:

Title::makeTitle
   count: 1000000
    rate: 110993.2/s
   total:  9009.56ms
    mean:     0.01ms
     max:     1.97ms
  stddev:     0.00ms
Current memory usage: 16.00 MiB
   Peak memory usage: 16.00 MiB

Benchmark wmf.12:

Title::makeTitle
   count: 1000000
    rate:  45122.2/s
   total: 22162.03ms
    mean:     0.02ms
     max:     9.55ms
  stddev:     0.13ms
Current memory usage: 22.00 MiB
   Peak memory usage: 22.00 MiB

Benchmark after fix:

Title::makeTitle
   count: 1000000
    rate:  78865.1/s
   total: 12679.88ms
    mean:     0.01ms
     max:     0.15ms
  stddev:     0.00ms
Current memory usage: 16.00 MiB
   Peak memory usage: 16.00 MiB

So the memory is back to 16.00 MiB, rate is still down about 30%, but much better than before. Could be improved further by inlining some of the deprecation logic.

Note that I had to crank the benchmark to a million repetitions to really see a difference... not sure this is really what's causing problems on translatewiki.

Change 744881 merged by jenkins-bot:

[mediawiki/core@master] DeprecationHelper: avoid closures

https://gerrit.wikimedia.org/r/744881

Do you want a backport? Is it confirmed that this would solve the issue? Let me know if I can help with anything (including backporting the change)

Not sure who you are asking, but I'm confident this solves the issue for translatewiki.net. Tomorrow we are going to drop the old cherry-pick and update to latest master for translatewiki.net.

Would be good to have for WMF production as well.

Change 746909 had a related patch set uploaded (by Ladsgroup; author: Daniel Kinzler):

[mediawiki/core@wmf/1.38.0-wmf.12] DeprecationHelper: avoid closures

https://gerrit.wikimedia.org/r/746909

Change 746909 merged by jenkins-bot:

[mediawiki/core@wmf/1.38.0-wmf.12] DeprecationHelper: avoid closures

https://gerrit.wikimedia.org/r/746909

Mentioned in SAL (#wikimedia-operations) [2021-12-13T18:21:26Z] <ladsgroup@deploy1002> Synchronized php-1.38.0-wmf.12/includes/: Backport: [[gerrit:746909|DeprecationHelper: avoid closures (T297236)]] (duration: 01m 02s)

Nikerabbit lowered the priority of this task from Unbreak Now! to High.Jan 24 2022, 7:37 AM

I think this task can be closed. Doesn't seem to be UBN! for sure, given it's been idle for a month.

DAlangi_WMF subscribed.

Now making > 3 months. Also, coming from @Nikerabbit which the issue seemed to have happened on TW, it can be resolved. Please reopen if I'm wrong.