Page MenuHomePhabricator

Consider httpd for quibble instead of php built-in server
Closed, ResolvedPublic

Description

I haven't done extensive tests, but after using httpd and php -S a fair amount for local development, httpd seems to be the winner in performance:

httpd:

❯ ab -n 500 -c 10 http://kafes.local/index.php/Special:BlankPage
This is ApacheBench, Version 2.3 <$Revision: 1843412 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking kafes.local (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Finished 500 requests


Server Software:        Apache/2.4.39
Server Hostname:        kafes.local
Server Port:            80

Document Path:          /index.php/Special:BlankPage
Document Length:        50625 bytes

Concurrency Level:      10
Time taken for tests:   45.375 seconds
Complete requests:      500
Failed requests:        342
   (Connect: 0, Receive: 0, Length: 342, Exceptions: 0)
Total transferred:      25449234 bytes
HTML transferred:       25242234 bytes
Requests per second:    11.02 [#/sec] (mean)
Time per request:       907.493 [ms] (mean)
Time per request:       90.749 [ms] (mean, across all concurrent requests)
Transfer rate:          547.72 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:   236  893 188.4    860    1420
Waiting:      205  804 178.2    778    1407
Total:        236  893 188.4    860    1420

Percentage of the requests served within a certain time (ms)
  50%    860
  66%    930
  75%    997
  80%   1054
  90%   1190
  95%   1258
  98%   1324
  99%   1416
 100%   1420 (longest request)

php -S:

❯ ab -n 500 -c 10 http://localhost:8888/index.php/Special:BlankPage
This is ApacheBench, Version 2.3 <$Revision: 1843412 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Finished 500 requests


Server Software:
Server Hostname:        localhost
Server Port:            8888

Document Path:          /index.php/Special:BlankPage
Document Length:        51357 bytes

Concurrency Level:      10
Time taken for tests:   203.485 seconds
Complete requests:      500
Failed requests:        499
   (Connect: 0, Receive: 0, Length: 499, Exceptions: 0)
Total transferred:      25502981 bytes
HTML transferred:       25320981 bytes
Requests per second:    2.46 [#/sec] (mean)
Time per request:       4069.707 [ms] (mean)
Time per request:       406.971 [ms] (mean, across all concurrent requests)
Transfer rate:          122.39 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       1
Processing:   395 4017 675.9   3881    7155
Waiting:      389 4004 674.6   3872    7150
Total:        395 4018 675.9   3881    7155

Percentage of the requests served within a certain time (ms)
  50%   3881
  66%   4060
  75%   4213
  80%   4369
  90%   4718
  95%   5180
  98%   5968
  99%   6691
 100%   7155 (longest request)

Related Objects

StatusSubtypeAssignedTask
ResolvedNone
OpenNone
OpenNone
ResolvedFeaturekostajh
OpenNone
OpenNone
OpenNone
ResolvedNone
ResolvedNone
Resolvedawight
ResolvedLucas_Werkmeister_WMDE
ResolvedNone
ResolvedDreamy_Jazz
OpenNone
Resolvedhashar
OpenNone
OpenNone
OpenNone
StalledNone
OpenNone
Resolved zeljkofilipin
ResolvedRickiJay-WMDE
ResolvedRickiJay-WMDE
OpenNone
Resolvedvaughnwalters
ResolvedEBernhardson
ResolvedEBernhardson
DeclinedNone
Resolvedhashar
DeclinedNone
OpenNone
DeclinedNone
ResolvedOsamaahmed17
ResolvedNone
ResolvedNone
DuplicateBUG REPORT zeljkofilipin
ResolvedOsamaahmed17
Resolved zeljkofilipin
ResolvedNone
ResolvedNone
ResolvedKrinkle
OpenNone
ResolvedOllie.Shotton_WMDE
ResolvedJakob_WMDE
DeclinedNone
OpenNone
ResolvedOsamaahmed17
Resolvedvaughnwalters
Resolved zeljkofilipin
Resolved zeljkofilipin
Resolved zeljkofilipin
Resolved zeljkofilipin
Resolvedkostajh
ResolvedOsamaahmed17
ResolvedOsamaahmed17
OpenNone
Resolvedpwangai
ResolvedWMDE-Fisch
ResolvedNone
ResolvedOsamaahmed17
ResolvedGehel
Resolvedpwangai
ResolvedNone
Invalidpwangai
ResolvedNone
InvalidNone
ResolvedOsamaahmed17
Resolvedvaughnwalters
Resolvedpwangai
ResolvedNone
InvalidNone
ResolvedOllie.Shotton_WMDE
ResolvedDreamy_Jazz
OpenNone
OpenNone
OpenNone
DuplicateNone
DeclinedNone
DeclinedNone
Resolved zeljkofilipin
Resolved zeljkofilipin
Resolved zeljkofilipin
ResolvedOsamaahmed17
OpenNone
OpenNone
ResolvedMhmohona
OpenNone
ResolvedAghaSaad04
ResolvedMhmohona
ResolvedLykarungi
Resolved zeljkofilipin
Resolved zeljkofilipin
Resolved zeljkofilipin
Resolved zeljkofilipin
OpenNone
OpenNone
Resolved zeljkofilipin
Resolved zeljkofilipin
OpenNone
ResolvedOsamaahmed17
OpenNone
ResolvedOsamaahmed17
OpenNone
ResolvedReedy
OpenNone
OpenNone
OpenNone
OpenNone
OpenNone
OpenNone

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Would you be able to run the same benchmark for hhvm as well?
hhvm -m server -p 8888 -d "hhvm.static_file.extensions[svg]=image/svg+xml"

Nice result, I was not expecting php -S to be THAT slower.


side note

On another task, Timo mentioned requests are not compressed, though I don't think it is a performance issue, that shows up that the built-in server uses the PHP options for CLI.

Serving a page through php -S I got:

PHP_SAPIcli-server
opcache.enableOn
opcache.enable_cliOff

But that does not seem to change anything from my loal testing.

end of side note


So yeah, looks like we need Quibble to learn how to start one of Apache/lighttpd/nginx and add that as a dependency.

Would you be able to run the same benchmark for hhvm as well?

I don't have hhvm set up on my machine, but if I get it configured, then sure :)

So yeah, looks like we need Quibble to learn how to start one of Apache/lighttpd/nginx and add that as a dependency.

I started experimenting with this; I started with Apache then abandoned it because of the difficulties of running as non-root, then tried nginx but that has its own issues with kick off php-fpm and nginx from within Quibble.

Would it work to do something like a CLI argument for --webserver=http://apache:8080 in which case Quibble routes browser requests to http://apache:8080. In integration/config we would then need to do docker run for an image containing Apache + PHP-(version), and after Quibble is done we'd need to stop and remove that container.

I started experimenting with this; I started with Apache then abandoned it because of the difficulties of running as non-root, then tried nginx but that has its own issues with kick off php-fpm and nginx from within Quibble.

Apache I could imagine you can not bind to tcp port 80 since that is only for root and I am not sure how that would work in an unprivileged container. That can be worked around with some linux capability, but Quibble / MediaWiki tests already support a port > 1024. So Apache should really be achievable.

Nginx, I have no idea, if you have hints I am willing to learn!

Thinking long, Quibble might well be able to detect/pick a httpd server depending on whats is available in the environment (eg Apache > nginx > lighttpd > php -s).

Would it work to do something like a CLI argument for --webserver=http://apache:8080 in which case Quibble routes browser requests to http://apache:8080. In integration/config we would then need to do docker run for an image containing Apache + PHP-(version), and after Quibble is done we'd need to stop and remove that container.

Then we will have to orchestrate containers and I am not really willing to do that with the current CI stack. The same issue I had with the patch to spawn Parsoid as a backend service :-\ Really that should be implemented with Helm and Kubernetes.

I am pretty sure we can hook Apache from Quibble :]

Change 516729 had a related patch set uploaded (by Kosta Harlan; owner: Kosta Harlan):
[integration/quibble@master] (wip) Add option for using Apache as server

https://gerrit.wikimedia.org/r/516729

Preliminary test run with Apache was not that thrilling: the selenium page.js test passed in 300 seconds instead of 345 with PHP built in server. I’ll poke at it some more though.

Some benchmarks with quibble + this patch + docker on macOS.

Starting the container

cd ~/quibble-builds && rm src/LocalSettings.php; docker run -it --rm \
  --name quibble \
  --entrypoint=bash \
    -e ZUUL_PROJECT=mediawiki/core \
    -v /Users/kharlan/quibble-builds/cache:/cache \
    -v /Users/kharlan/quibble-builds/log:/workspace/log \
    -v /Users/kharlan/quibble-builds/ref:/srv/git:ro \
    -v /Users/kharlan/quibble-builds/src:/workspace/src \
    quibble-apache

built-in PHP + sqlite

quibble --skip-deps --skip-zuul --db sqlite --db-dir /workspace/src/cache/ --run selenium

run 1

15 passing (318.30s)
5 skipped

run 2

15 passing (299.40s)
5 skipped

run 3 (disabled page undoable test):

14 passing (302.40s)
6 skipped

apache2 + sqlite

quibble --skip-deps --skip-zuul --db sqlite --db-dir /workspace/src/cache/ --run selenium --webserver=apache2

run 1

14 passing (246.50s)
5 skipped
1 failing

run 2

14 passing (208.10s)
5 skipped
1 failing

run 3 (disabled page undoable test)

14 passing (263.40s)
6 skipped

This error occurred on both runs with --webserver=apache2, so I disabled it for the third test run:

1) Page should be undoable:
'beforeEach-content-0.8998667415652057-Iñtërnâtiônàlizætiøn' === 'beforeEach-content-0.0023315024241947757-Iñtërnâtiônàlizætiøn'

built-in PHP + MySQL

quibble --skip-deps --skip-zuul --db mysql --db-dir /workspace/src/cache/ --run selenium

run 1

14 passing (225.20s)
6 skipped

run 2

14 passing (264.80s)
6 skipped

apache2 + MySQL

n.b. For MySQL, I had to change the db directory to /workspace/src/cache.

quibble --skip-deps --skip-zuul --db mysql --db-dir /workspace/src/cache/ --run selenium --webserver=apache2

run 1

14 passing (191.90s)
6 skipped

run 2

14 passing (195.10s)
6 skipped

Change 516729 abandoned by Kosta Harlan:
Add option for using Apache as server

Reason:
IMO let's wait for local-charts :) Although if someone wants to pick this up, feel free

https://gerrit.wikimedia.org/r/516729

Change 516729 restored by Kosta Harlan:
Add option for using Apache as server

https://gerrit.wikimedia.org/r/516729

Change 568931 had a related patch set uploaded (by Kosta Harlan; owner: Kosta Harlan):
[integration/quibble@master] (wip/poc/avert your eyes): quibble drives docker

https://gerrit.wikimedia.org/r/568931

I think this is a killer feature, in combination with T226869: Run browser tests in parallel. The built-in server is single-threaded, so I expect parallel browser testing to bottleneck on PHP.

However, I should mention this interesting changelog for PHP 7.4:

You can configure the built-in webserver to fork multiple workers in order to test code that requires multiple concurrent requests to the built-in webserver. Set the PHP_CLI_SERVER_WORKERS environment variable to the number of desired workers before starting the server.

@kostajh We started discussing in code review, but I wanted to move here for visibility. What do you think about heading towards --webserver=none, nor driving docker from Quibble, but instead just letting the CI job spin up an apache container pointing at the source directory? We can keep the code for a PHP standalone backend in Quibble in case of emergency. But the ideal would be to avoid adding any more complexity to Quibble, and decouple it completely from the details of how to serve pages.

My thought was already too naive--we would need to containerize the mysql server as well.

Change 590651 had a related patch set uploaded (by Awight; owner: Awight):
[integration/quibble@master] Flag for "none" webserver

https://gerrit.wikimedia.org/r/590651

Apache is definitely to be considered.

I also found php 7.4+ supports multiple workers and filed T259456 for that.

awight added a project: User-awight.
awight moved this task from Backlog to Prototype on the User-awight board.

Change 591826 had a related patch set uploaded (by Awight; owner: Awight):
[integration/quibble@master] Configurable web host and port

https://gerrit.wikimedia.org/r/591826

Change 591826 merged by jenkins-bot:
[integration/quibble@master] Configurable web host and port

https://gerrit.wikimedia.org/r/591826

Change 590651 merged by jenkins-bot:
[integration/quibble@master] Flag for "external" webserver

https://gerrit.wikimedia.org/r/590651

Change 625636 had a related patch set uploaded (by Awight; owner: Awight):
[integration/quibble@master] [WIP] Manage database backend in separate step

https://gerrit.wikimedia.org/r/625636

Change 625638 had a related patch set uploaded (by Awight; owner: Awight):
[integration/quibble@master] [WIP] Extract web backend management

https://gerrit.wikimedia.org/r/625638

Change 625636 merged by jenkins-bot:
[integration/quibble@master] Manage database backend outside of commands

https://gerrit.wikimedia.org/r/625636

Change 625638 merged by jenkins-bot:
[integration/quibble@master] Manage web backend outside of commands

https://gerrit.wikimedia.org/r/625638

Change 628138 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/quibble@master] Release Quibble 0.0.45

https://gerrit.wikimedia.org/r/628138

Change 628138 merged by jenkins-bot:
[integration/quibble@master] Release Quibble 0.0.45

https://gerrit.wikimedia.org/r/628138

Change 628759 had a related patch set uploaded (by Awight; owner: Awight):
[integration/quibble@master] Implement webserver engines as subclasses

https://gerrit.wikimedia.org/r/628759

Change 630164 had a related patch set uploaded (by Awight; owner: Awight):
[integration/config@master] [WIP] Apache service for Quibble

https://gerrit.wikimedia.org/r/630164

Change 628759 merged by jenkins-bot:
[integration/quibble@master] Implement webserver engines as subclasses

https://gerrit.wikimedia.org/r/628759

Change 640432 had a related patch set uploaded (by Awight; owner: Awight):
[integration/quibble@master] Adapt API URL to the current webserver

https://gerrit.wikimedia.org/r/640432

Change 640432 merged by jenkins-bot:
[integration/quibble@master] Adapt API URL to the current webserver

https://gerrit.wikimedia.org/r/640432

Change 516729 merged by jenkins-bot:
[integration/quibble@master] Dockerfile launches external apache

https://gerrit.wikimedia.org/r/516729

Change 644485 had a related patch set uploaded (by Awight; owner: Awight):
[integration/config@master] [WIP] CI image for quibble serving from apache

https://gerrit.wikimedia.org/r/644485

Change 630164 abandoned by Awight:
[integration/config@master] [WIP] Apache service for Quibble

Reason:
This was a stale fork.

https://gerrit.wikimedia.org/r/630164

Change 644485 merged by jenkins-bot:
[integration/config@master] Image for quibble serving from apache

https://gerrit.wikimedia.org/r/644485

Change 647202 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] dockerfiles: quibble-apache missed php.ini file

https://gerrit.wikimedia.org/r/647202

Change 647202 merged by jenkins-bot:
[integration/config@master] dockerfiles: quibble-apache missed php.ini file

https://gerrit.wikimedia.org/r/647202

Mentioned in SAL (#wikimedia-releng) [2020-12-09T10:28:17Z] <hashar> Successfully tagged docker-registry.discovery.wmnet/releng/quibble-apache:0.0.1 T225218

Change 647199 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Quibble full run with Apache

https://gerrit.wikimedia.org/r/647199

Change 647704 had a related patch set uploaded (by Awight; owner: Awight):
[integration/quibble@master] Installer hardcodes paths

https://gerrit.wikimedia.org/r/647704

Change 647199 merged by jenkins-bot:
[integration/config@master] Quibble full run with Apache

https://gerrit.wikimedia.org/r/647199

Change 647704 merged by jenkins-bot:
[integration/quibble@master] Installer hardcodes paths

https://gerrit.wikimedia.org/r/647704

@awight @hashar the deadlock error seen on https://integration.wikimedia.org/ci/view/CI/job/integration-quibble-apache-fullrun/9/console looks to be the same as T199393. We could mark those particular API tests (tests/api-testing/action/UserContribs.js) as skipped or add some pause in between account creations. Or we could fix T199393 :)

Nice catch. Given the test is subject to a race condition, I guess it should be disabled entirely or indeed fixed but that seems to not be a trivial task. :-(

Daniel had a patch https://gerrit.wikimedia.org/r/c/mediawiki/tools/api-testing/+/530343/ . Seems the account creation happens in parallel so the order is not guaranteed. Maybe php -S kind of forced the serialization of the requests, but with Apache they are processed in parallel and we hit the bug. Fun one!

Change 654443 had a related patch set uploaded (by Awight; owner: Awight):
[mediawiki/core@master] Wait until the recent changes are updated

https://gerrit.wikimedia.org/r/654443

Change 654443 merged by jenkins-bot:
[mediawiki/core@master] Wait until the recent changes are updated

https://gerrit.wikimedia.org/r/654443

@hashar The experimental quibble-apache job passes now, so the obstacles are cleared to deploying a non-voting job, at your leisure!

The tests are still running in sequence at the moment, so I would have expected the apache+php-fpm job to have similar or slightly better performance. The results so far are ambiguous, all I can say is that we don't seem to be incurring any huge extra costs. (Unfairly,) directly comparing timings for the test suites (browser tests for core and Vector; API tests; excluding install time for both suites) from two arbitrary quibble-fullrun jobs, one with apache and the other with standalone PHP:

browser testsAPI tests
PHP-standalone56s152s
apache+php-fpm72s118s

Total run time for the apache job was 8m50s. Cherry-picking a metric, total execution time for the server-limited suites is 9% lower for the apache job. Please take my garbage data with a grain of salt, obviously these are just ballpark numbers until we have enough equivalent runs to look at the average. My only interpretation so far is that it shows nothing is going horribly wrong.

@hashar The experimental quibble-apache job passes now, so the obstacles are cleared to deploying a non-voting job, at your leisure!

Just a caveat, there are going to be issues with api-testing and Selenium tests failing due to the absence of single-threaded processing. For example in this build https://integration.wikimedia.org/ci/view/CI/job/integration-quibble-apache-fullrun/12/console we can see some tests failing. I think it's OK to deploy as a non-voting job but we should probably create tasks (or a single tracking task?) to fix these as they come up.

The tests are still running in sequence at the moment, so I would have expected the apache+php-fpm job to have similar or slightly better performance. The results so far are ambiguous, all I can say is that we don't seem to be incurring any huge extra costs. (Unfairly,) directly comparing timings for the test suites (browser tests for core and Vector; API tests; excluding install time for both suites) from two arbitrary quibble-fullrun jobs, one with apache and the other with standalone PHP:

browser testsAPI tests
PHP-standalone56s152s
apache+php-fpm72s118s

Total run time for the apache job was 8m50s. Cherry-picking a metric, total execution time for the server-limited suites is 9% lower for the apache job. Please take my garbage data with a grain of salt, obviously these are just ballpark numbers until we have enough equivalent runs to look at the average. My only interpretation so far is that it shows nothing is going horribly wrong.

There is probably some optimization we could do for php-fpm/apache integration to speed things up. For example tweaking the opcache setting to cache PHP files indefinitely.

I thought I posted somewhere but I can't find it now; early on in this patch's history, I had experimented with benchmarking on a DigitalOcean box and didn't find any improvements with Apache over PHP's built-in server. Even so, the fact that we are surfacing race conditions in our tests, as well as triggering application errors like T199393, highlights the usefulness of using Apache with Quibble.

[...] the fact that we are surfacing race conditions in our tests, as well as triggering application errors like T199393, highlights the usefulness of using Apache with Quibble.

This about sums it up for me as well. I feel like we've reached a point where quibble-apache went from being an investigation, to a goal I'd like to commit to. Running UI test suites concurrently is a bonus, but the robustness benefits and similarity to production are more compelling.

Change 654804 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/quibble@master] Release Quibble 0.0.46

https://gerrit.wikimedia.org/r/654804

Change 654804 merged by jenkins-bot:
[integration/quibble@master] Release Quibble 0.0.46

https://gerrit.wikimedia.org/r/654804

@hashar The experimental quibble-apache job passes now, so the obstacles are cleared to deploying a non-voting job, at your leisure!

I assume the non-voting job would be just to make sure nothing melts down, and then we'd make it voting and transition away from the php built-in server jobs. Once the non-voting job is added, and again when it becomes voting, we should probably send a heads up to wikitech-l to let folks know about race condition failures with Selenium and API testing tests, and have a single phab task for tracking patches that disable those tests as well as patches for making them more resilient.

Quibble 0.0.46 has been released and the CI jobs switched to it.

The integration-quibble-apache-fullrun job which is running for integration/quibble is passing.

Change 568931 abandoned by Kosta Harlan:
[integration/quibble@master] (wip/poc/avert your eyes): quibble drives docker

Reason:

https://gerrit.wikimedia.org/r/568931

Just for the feeling of completion, I'll close this task and create a new one for next steps.

Change 884332 had a related patch set uploaded (by Umherirrender; author: Awight):

[mediawiki/core@REL1_35] Wait until the recent changes are updated

https://gerrit.wikimedia.org/r/884332

Change 884332 merged by jenkins-bot:

[mediawiki/core@REL1_35] Wait until the recent changes are updated

https://gerrit.wikimedia.org/r/884332