Hacked up a temporary xhprof collector, removed blocker.

• 34% of wall clock time goes to Wikidata's SidebarHookHandlers, which does expensive things (mostly loading the site list) despite there being no sidebar.
• 28% goes to MWTidy::execExternalTidy
• 16% goes to MessageCache::transform() due to {{PLURAL}} usage in the limitreport-* messages

That leaves 22%, so it would be about 4x faster if the above three issues were fixed.

When I ran the same test another few times, MessageCache disappeared from the list, but SidebarHookHandlers and MWTidy::execExternalTidy were still very expensive: ~110ms and ~200ms respectively.

Wikidata's contribution should be minimized by these two patches, which the team hopes to ship this week:

https://gerrit.wikimedia.org/r/#/c/174113/
https://gerrit.wikimedia.org/r/#/c/174874/

Change 175940 had a related patch set uploaded (by Tim Starling):
Use HHVM EZC internal tidy

https://gerrit.wikimedia.org/r/175940

Patch-For-Review

Change 175940 merged by jenkins-bot:
Use HHVM EZC internal tidy

https://gerrit.wikimedia.org/r/175940

Change 176383 had a related patch set uploaded (by Ori.livneh):
Use HHVM EZC internal tidy

https://gerrit.wikimedia.org/r/176383

Patch-For-Review

Change 176383 merged by jenkins-bot:
Use HHVM EZC internal tidy

https://gerrit.wikimedia.org/r/176383

Change 176387 had a related patch set uploaded (by Ori.livneh):
Use HHVM EZC internal tidy

https://gerrit.wikimedia.org/r/176387

Patch-For-Review

Change 176387 merged by jenkins-bot:
Use HHVM EZC internal tidy

https://gerrit.wikimedia.org/r/176387

tidy.so still needs to be added to /etc/hhvm/fcgi.ini . Please deploy it with care -- test one server first, and watch the logs.

Change 176864 had a related patch set uploaded (by Ori.livneh):
Add https://github.com/facebook/hhvm/pull/4330

https://gerrit.wikimedia.org/r/176864

Patch-For-Review

Change 176864 merged by Ori.livneh:
Add https://github.com/facebook/hhvm/pull/4330

https://gerrit.wikimedia.org/r/176864

In T758#795816, @tstarling wrote:

tidy.so still needs to be added to /etc/hhvm/fcgi.ini . Please deploy it with care -- test one server first, and watch the logs.

@Joe: HHVM needs to be re-packaged, too -- we need https://github.com/facebook/hhvm/pull/4330. It backported cleanly: https://gerrit.wikimedia.org/r/#/c/176864/.

Order of operations for deployment:

1. Rebuild HHVM package with EZC fix.
2. Deploy HHVM package.
3. Merge and deploy change Ia69546e84: hhvm: load tidy.so extension.

(The extension will not be used yet, because I set $wgInternalTidy to false in CommonSettings.php.)

4. Merge and deploy change I376d064c5: Enable internal tidy if extension is loaded and hostname == mw1114.

If everything looks good, the final step is:

5. Merge and deploy change I24be75130: Unset $wgTidyInternal, so its default value is used.

This will cause MediaWiki to revert to its default behavior of using the tidy extension whenever it detects that the extension is available.

Ori's plan seems good to me, with the additional passage of testing every step in beta.

so, I performed step 1) and I deployed the package to beta, and a smoke test showed no crashes; so I decided to upload it to the repository.

We'll also have a few newly reimaged servers using the new package for the night, so that is everything is good tomorrow I'll deploy the new package everywhere - so step 2) will be done.

Next step is testing 3 and 4 in beta, then do the same in production. While in beta they can be tested together, the production deploy should be separated.

Sounds good to me.

I pushed this to Beta and we started getting crashes:

I wrote an HNI extension for just tidy_repair_string() and pushed it to the hni branch of the repository. Packaged it and deployed it to Labs. Looks good so far.

Extension deployed everywhere. New result for ab command:

Zend:

Requests per second:    63.46 [#/sec] (mean)
Time per request:       1575.710 [ms] (mean)
Time per request:       15.757 [ms] (mean, across all concurrent requests)

HHVM:

Requests per second:    121.21 [#/sec] (mean)
Time per request:       824.985 [ms] (mean)
Time per request:       8.250 [ms] (mean, across all concurrent requests)