Page MenuHomePhabricator

~27% wt2html latency increase on Obama since April 2015
Closed, ResolvedPublic

Description

A recent re-run of the classic Obama benchmark piqued my interest by returning a higher-than-expected latency. Taking the historic code and content used in T89775 for comparison, I'm getting the following results:

Current master:

git checkout 6aae433135a94f9dbf25915629986bc111fad558
rm -r node_modules && npm install
node bin/server.js -n1 &
ab -n 10 http://localhost:8000/en.wikipedia.org/v3/page/html/Barack_Obama/647485918

Document Length:        1779006 bytes
Percentage of the requests served within a certain time (ms)
  50%  15173
  66%  15317
  75%  15345
  80%  15876
  90%  16500
  95%  16500
  98%  16500
  99%  16500
 100%  16500 (longest request)

Old code:

git checkout 5d8b4e05c33e0533651800ae26d3c2e59bae90b9
rm -r node_modules && npm install
node api/server.js -n1 &

ab -n 10 http://localhost:8000/v2/en.wikipedia.org/html/Barack_Obama/647485918

Document Length:        2183602 bytes
Percentage of the requests served within a certain time (ms)
  50%  11865
  66%  11933
  75%  12170
  80%  12660
  90%  12751
  95%  12751
  98%  12751
  99%  12751
 100%  12751 (longest request)

Event Timeline

I did a quick nodegrind bin/server.js -n0 run, and several core-js related methods stood out, especially the wrapped JSON.stringify and fix-re-wks.

Possible quick fixes:

  • npm install --save core-js@0.9
  • Alternatively, in core-upgrade.js:
if (process.version < 'v4.') {
    // Only shim for node < 4
    require('core-js/shim');
}

Over wifi, this gets me parse times of around 12.6s, compared to around 17.6s with plain master.

Change 276697 had a related patch set uploaded (by GWicke):
Performance: Only pull in the core-js shim for node < v4

https://gerrit.wikimedia.org/r/276697

I did a quick nodegrind bin/server.js -n0 run, and several core-js related methods stood out, especially the wrapped JSON.stringify and fix-re-wks.

Possible quick fixes:

  • npm install --save core-js@0.9
  • Alternatively, in core-upgrade.js:
if (process.version < 'v4.') {
    // Only shim for node < 4
    require('core-js/shim');
}

Over wifi, this gets me parse times of around 12.6s, compared to around 17.6s with plain master.

So, this regression on node v4 is from https://gerrit.wikimedia.org/r/#/c/254442/

Also, this explains why I wasn't seeing the perf regressions from time:total stats on ruthenium when I was looking there yesterday. The only bump I see across many pages is from the 2015-11-24 commit 587478842816f4e11edb688dcb77de20a38aa8cf which is the async WTS patch which we know caused a serious perf degradation for the serializer. time:total is the total rt-test run (we should update those stats to capture the individual components). Your comment explains this because ruthenium used to run node 0.10 till late Jan 2016.

|     485 | 26444 | c0287bcd9d3018d6ed32c2e239d607859ce2b029 | 2015-11-20 23:00:48 |
|     485 | 28961 | e1178d1486709e5285fe1876e64671940ab97845 | 2015-11-21 04:33:27 |
|     485 | 42691 | 587478842816f4e11edb688dcb77de20a38aa8cf | 2015-11-24 04:47:45 |
|     485 | 35990 | 6335800734c423727aea49d3c0c04d6209aa10b6 | 2015-11-27 20:11:02 |
|     485 | 40567 | cb01bb4afc7d6d1c358a632f860303e9455011e5 | 2015-12-02 05:11:45 |
|     485 | 36482 | 4119576fa87f8f24051d2589851fe2d81a4b85d5 | 2015-12-06 20:17:08 |

The issue should affect all node versions. The root cause is newer core-js degrading the performance of built-in functionality by wrapping it in order to better support Symbols. This is the case even on node 5.9.

The point of the version check is that you are not actually using any functionality that is not natively implemented in node 4, so do not need to use the shim at all with newer node versions.

You are right though that node 0.10 seems to be slow even with core-js 0.9. A possible explanation might be that some of the dependencies like prfun are now pulling in shims of their own, and need to shim more than usual on node 0.10.

Change 276697 merged by jenkins-bot:
Performance: Only pull in the core-js shim for node < v4

https://gerrit.wikimedia.org/r/276697

We should talk to the core-js guys to see if they can modularize the Symbol support. In general they are very open to an "include only what you need" coding style.

Perhaps this could be reevaluated after the Node.js 6 upgrade.

Arlolra claimed this task.

core-js shimming was limited to a few methods in https://gerrit.wikimedia.org/r/#/c/385238/ after rediscovering the slowdown from shimming json methods for symbol support.