Page MenuHomePhabricator

Enable compression for MW web responses in Jenkins jobs (e.g. Quibble, Fresnel)
Closed, ResolvedPublic

Description

MediaWiki does a lot of work to try and do this by default, yet it seems the particular combination of factors in CI make it not enable gzip for web responses.

This means that behaviour is not very representative. In particular, it means that for Fresnel, the metrics "raw code" (decodedBodySize) and "network transfer" (encodedBodySize) are the same, whereas the latter is meant to be smaller and behave differently.

Details

Related Gerrit Patches:

Event Timeline

Krinkle created this task.Mar 30 2019, 2:38 AM
Restricted Application added a project: Performance-Team. · View Herald TranscriptMar 30 2019, 2:38 AM
Krinkle updated the task description. (Show Details)Mar 30 2019, 2:39 AM

I thought maybe that our use of the PHP built-in webserver (php -S) is causing this, our the maintenance/dev/router.php script that we use.

However, I'm not able to reproduce uncompressed responses locally with these. They are compressed right out of the box. MediaWiki detects that zlib is installed, and detects that no default compression is happening elsewhere, and then rolls its own (handled in OutputHandler.php, called from WebStart.php).

This rules them out as the cause of the problem.

Using a Quibble container locally, I was able to reproduce the issue. Starting php -S from a new non-mediawiki directory in the container, I find that there is a pre-existing output buffer. Hence, MediaWiki would be unable to start its own top-level output buffer.

The problem is caused by the following IN setting for PHP:

quibble container /etc/php/7.0/cli/php.ini
; Default Value: Off
; http://php.net/output-buffering
output_buffering = 4096
ob_list_handlers.php
# ob_list_handlers()
array(
 0 => "default output handler"
)

Looks like the output_buffering feature is enabled. This means PHP starts an output buffer natively that flushes output directly to the user in chunks of 4KB.

By default in PHP this is disabled. I've also checked WMF production and it is disabled there as well, both for HHVM and for PHP 7.2.

When disabling it, the array from ob_list_handlers() becomes empty, and MediaWiki's output compression starts working.

hashar added a comment.Apr 9 2019, 5:58 PM

Quibble indeed uses php -S which saved me to have to deal with a frontend web server and have to configure it. Anyway, since the webserver is the PHP CLI it eventually loads /etc/php/7.0/cli/php.ini

Should we have Quibble to start the server hardcoding the setting? Eg:

php -d output_buffering=Off -S

Which might be recognized by HHVM as well (to be checked).

Krinkle added a comment.EditedApr 9 2019, 6:00 PM

@hashar Maybe, yeah!

I am also wondering where that INI file is coming from, because afaik it is not the PHP default. Perhaps that could be disabled somehow. But, I think it's nice to set explicitly on the php call, that way it will not depend on the environment.

Krinkle claimed this task.Apr 9 2019, 6:02 PM
Krinkle triaged this task as Normal priority.

I'll patch Quibble to try this out.

hashar added a comment.Apr 9 2019, 6:17 PM

The php package should be https://packages.debian.org/source/stretch/php7.0 , with the git repository supposedly on gitlab:

https://gitlab.com/deb.sury.org/php
https://gitlab.com/deb.sury.org/php.git

Which yield: You need to sign in or sign up before continuing.

I guess it is a case in favor of hosting on a third party.

Anyway I have instead downloaded the source package (apt-get source php7.0). To the best I can tell, php-7.0.33 upstream tarball comes with php.ini that have output_buffering = 4096.

So yeah, should be hardcoded in Quibble instead :]

Krinkle removed Krinkle as the assignee of this task.EditedMay 30 2019, 4:23 PM
Krinkle added a project: patch-welcome.

(Still interested in this as deliverable for a future quarter - as part of Fresnel maintenance and on-going improvements. But, if anyone else is interested in this before I get to it, that'd be great too.)

Change 534468 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[integration/quibble@master] Disable output buffering in running php

https://gerrit.wikimedia.org/r/534468

Krinkle assigned this task to Ladsgroup.Sep 4 2019, 4:03 PM
Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptSep 4 2019, 4:03 PM

Change 535938 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[mediawiki/core@master] DevelopmentSettings: Disable output buffering to be more like production

https://gerrit.wikimedia.org/r/535938

Ok I gave it a try creating a phpinfo file at the root of the mediawiki/core checkout and running something like:

rm src/LocalSettings.php
quibble --skip-zuul --skip-deps --db sqlite -c 'xdg-open http://127.0.0.1:9412/phpinfo.php'

output_buffering shows with no value.

And a basic script shows that it is no more buffered:

count.php
<?php
$i = 0;
while ( true ) {
    print $i++ . "\n";
    sleep(1);
    if ( $i === 1000 ) {
        exit;
    }
}

Change 534468 merged by jenkins-bot:
[integration/quibble@master] Disable output buffering in running php

https://gerrit.wikimedia.org/r/534468

Change 537469 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] docker: rebuild for quibble 0.0.35

https://gerrit.wikimedia.org/r/537469

Change 537469 merged by jenkins-bot:
[integration/config@master] docker: rebuild for quibble 0.0.35

https://gerrit.wikimedia.org/r/537469

Change 537485 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[integration/config@master] jjb: Upgrade all quibble jobs to 0.0.35

https://gerrit.wikimedia.org/r/537485

Change 537485 merged by jenkins-bot:
[integration/config@master] jjb: Upgrade all quibble jobs to 0.0.35

https://gerrit.wikimedia.org/r/537485

Jdforrester-WMF renamed this task from Enable compression for MW web responses in Jenkins Jenkins (e.g. Quibble, Fresnel) to Enable compression for MW web responses in Jenkins jobs (e.g. Quibble, Fresnel).Sep 17 2019, 6:41 PM
Krinkle closed this task as Resolved.EditedSep 17 2019, 6:58 PM

Confirmed to work.

Fresnel run from last night:

|   Total size of transfers during page load (pageWeight)  |   1.3 MB (± <0.4 B) |   1.3 MB (± <0.4 B) |   
|   Transfer size of HTML document (html)                  |  30.7 kB (± <0.4 B) |  30.7 kB (± <0.4 B) |  
|   Transfer size of CSS resources (css)                   |     51.6 kB (± 0 B) |     51.6 kB (± 0 B) |   
|   Transfer size of JavaScript resources (js)             |      1.2 MB (± 0 B) |      1.2 MB (± 0 B) |  
|   Transfer size of Image document (img)                  |     28.1 kB (± 0 B) |     28.1 kB (± 0 B) |

Fresnel run from just now

|
 |   Total size of transfers during page load (pageWeight)  |   271 kB (± <0.8 B) |   269 kB (± <0.7 B) | 
 |   Transfer size of HTML document (html)                  |   7.4 kB (± <0.8 B) |   7.4 kB (± <0.7 B) |  
 |   Transfer size of CSS resources (css)                   |     11.6 kB (± 0 B) |     11.6 kB (± 0 B) |    
 |   Transfer size of JavaScript resources (js)             |      224 kB (± 0 B) |      222 kB (± 0 B) | 
 |   Transfer size of Image document (img)                  |     28.1 kB (± 0 B) |     28.1 kB (± 0 B) |

Clearly much smaller, which can only be explained by gzip.

Well done, and thank you for the verification.

Released and deployed on September 17th with Quibble 0.0.35.

Also confirmed from mw-debug-www.log:

wfClientAcceptsGzip: client accepts gzip.
MediaWiki\OutputHandler::handleGzip() is compressing output
Maintenance_bot moved this task from Incoming to Done on the User-Ladsgroup board.Mon, Oct 28, 8:15 PM