From the project page:
PHPBench is a benchmark runner for PHP analogous to PHPUnit but for performance rather than correctness.
https://github.com/phpbench/phpbench
I'm curious how/if it could fit into our existing profiling/benchmarking tools. It could be interesting to use it in CI to guard against performance regressions.
Further reading Blog Post: Benchmarking MediaWiki with PHPBench
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T291549 Evaluate phpbench for benchmarking and guarding against performance regressions | |||
| Resolved | kostajh | T306891 PhpBench fails for GrowthExperiments with UnexpectedValueException: Duplicate provider name "MediaWiki\Session\CookieSessionProvider" |
Change 722594 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):
[mediawiki/extensions/GrowthExperiments@master] Add phpbench test for LinkRecommendationFilter
Change 726681 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):
[integration/quibble@master] Add support for phpbench
Change 726681 merged by jenkins-bot:
[integration/quibble@master] Add support for phpbench
Change 726880 had a related patch set uploaded (by Jforrester; author: Kosta Harlan):
[integration/config@master] jjb: Bump Quibble jobs to 1.1.0
Mentioned in SAL (#wikimedia-releng) [2021-10-07T13:04:47Z] <hashar> Updating Jenkins jobs for Quibble 1.1.0 # T291549
Change 726880 merged by jenkins-bot:
[integration/config@master] jjb: Bump almost all Quibble jobs to 1.1.0
Change 727429 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):
[mediawiki/core@master] [WIP] Add phpbench to dev requirements
Change 727443 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):
[integration/quibble@master] phpbench: Run composer install first
Change 726653 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):
[mediawiki/extensions/GrowthExperiments@master] Add composer entry for phpbench
Change 727443 merged by jenkins-bot:
[integration/quibble@master] phpbench: Run composer install first
If this experiment turns out to be useful, Performance Team would definitely like to schedule some time to figure out how it fits in with existing tools.
Thanks @dpifke. The patches attached here are to facilitate experimentation (it helps to have CI be able to run your patch if you want to play around with the tool); I'm drafting a blog post that explains the tool and how to use it but doesn't make any suggestions about best practices or patterns to use around what type of code to test, types of assertions to make, etc. If you'd like to review the post or chat (here in phab or elsewhere) before I send that around, I'd be happy to. I was thinking that once the explanatory post is available, folks could experiment with building on top of the core patch or make their own extension patches, then we could have some evaluation to decide when/if/how to document and promote this as something in our toolkit. Does that sound OK?
https://phabricator.wikimedia.org/phame/post/view/257/benchmarking_mediawiki_with_phpbench/ if you'd like to review / comment on it (cc @Krinkle)
Change 741974 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):
[integration/quibble@master] phpbench: Support aggregate reports
Change 741976 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):
[integration/config@master] jjb: phpbench aggregate reports for MW core
https://phabricator.wikimedia.org/phame/post/view/257/benchmarking_mediawiki_with_phpbench/ it's published now.
I also added a patch for quibble implementing the before/after aggregate report, and an example config to run that for mediawiki/core, like Fresnel is run currently. Review/opinions/thoughts wanted. :)
Change 741974 merged by jenkins-bot:
[integration/quibble@master] phpbench: Support aggregate reports
Change 758952 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):
[integration/quibble@master] release: Quibble 1.4.0
Change 758952 merged by jenkins-bot:
[integration/quibble@master] release: Quibble 1.4.0
Mentioned in SAL (#wikimedia-releng) [2022-02-03T13:51:27Z] <kostajh> Tag Quibble 1.4.0 @ 4231bc2832395d94e29a332fe8d863301a0cd441 # T300340 T291549 T225730
Change 763559 had a related patch set uploaded (by Hashar; author: Hashar):
[integration/config@master] jjb: update Quibble jobs from 1.3.0 to 1.4.0
Mentioned in SAL (#wikimedia-releng) [2022-02-21T07:31:03Z] <hashar> Switching Quibble jobs from Quibble 1.3.0 to 1.4.0 # T300340 T291549 T225730
Change 763559 merged by jenkins-bot:
[integration/config@master] jjb: update Quibble jobs from 1.3.0 to 1.4.0
Change 767749 had a related patch set uploaded (by Hashar; author: Hashar):
[integration/config@master] jjb: update Quibble jobs from 1.3.0 to 1.4.3
Change 767749 merged by jenkins-bot:
[integration/config@master] jjb: update Quibble jobs from 1.3.0 to 1.4.3
Change 726653 abandoned by Kosta Harlan:
[mediawiki/extensions/GrowthExperiments@master] Add composer entry for phpbench
Reason:
squashed
Change 741976 merged by jenkins-bot:
[integration/config@master] jjb: Add mediawiki-phpbench-patch-docker, add GrowthExperiments as first test
Change 773807 had a related patch set uploaded (by Hashar; author: Hashar):
[integration/config@master] jjb: drop THING_SUBNAME in mediawiki-phpbench-patch-docker
Change 773807 merged by jenkins-bot:
[integration/config@master] jjb: drop THING_SUBNAME in mediawiki-phpbench-patch-docker
Change 773869 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):
[integration/config@master] jjb: Adjust phpbench args
Change 774379 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):
[integration/quibble@master] phpbench: Use git-checkout instead of git-switch
Change 773869 merged by jenkins-bot:
[integration/config@master] jjb: Adjust phpbench args
Change 774379 merged by jenkins-bot:
[integration/quibble@master] phpbench: Use git-checkout instead of git-switch
Change 774410 had a related patch set uploaded (by Hashar; author: Hashar):
[integration/quibble@master] release: Quibble 1.4.5
Change 774415 had a related patch set uploaded (by Hashar; author: Hashar):
[integration/config@master] dockerfiles update to Quibble 1.4.5
Change 774410 merged by jenkins-bot:
[integration/quibble@master] release: Quibble 1.4.5
Mentioned in SAL (#wikimedia-releng) [2022-03-28T09:53:26Z] <hashar> Tag Quibble 1.4.5 @ abe16d574 | T291549
Change 774415 merged by jenkins-bot:
[integration/config@master] dockerfiles update to Quibble 1.4.5
Change 774427 had a related patch set uploaded (by Hashar; author: Hashar):
[integration/config@master] jjb: mediawiki-phpbench-patch-docker to Quibble 1.4.5
Change 774427 merged by jenkins-bot:
[integration/config@master] jjb: mediawiki-phpbench-patch-docker to Quibble 1.4.5
Update: here's an example patch implementing aggregate reporting for GrowthExperiments https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GrowthExperiments/+/722594
Change 722594 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@master] phpbench: tests for LinkRecommendationFilter and Special:Homepage
Change 813194 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):
[integration/config@master] zuul: Run non-voting phpbench job for MW core patches
@kostajh there are two patches pending to add phpbench to mediawiki/core:
| https://gerrit.wikimedia.org/r/c/mediawiki/core/+/727429 | testing: Add phpbench support for performance regression testing |
| https://gerrit.wikimedia.org/r/c/integration/config/+/813194 | zuul: Run non-voting phpbench job for MW core patches |
That would adds a bit of overhead and complexity for a single test case and I am not sure whether it is worth pursuing?
I don't think there's any enthusiasm from the developer community about this, so we can probably abandon these.
Change 727429 abandoned by Kosta Harlan:
[mediawiki/core@master] testing: Add phpbench support for performance regression testing
Reason:
Change 813194 abandoned by Kosta Harlan:
[integration/config@master] zuul: Run non-voting phpbench job for MW core patches
Reason: