Page MenuHomePhabricator

Evaluate phpbench for benchmarking and guarding against performance regressions
Open, Needs TriagePublic

Description

From the project page:

PHPBench is a benchmark runner for PHP analogous to PHPUnit but for performance rather than correctness.

https://github.com/phpbench/phpbench

I'm curious how/if it could fit into our existing profiling/benchmarking tools. It could be interesting to use it in CI to guard against performance regressions.

Further reading Blog Post: Benchmarking MediaWiki with PHPBench

Event Timeline

Change 722594 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/GrowthExperiments@master] Add phpbench test for LinkRecommendationFilter

https://gerrit.wikimedia.org/r/722594

Change 726681 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[integration/quibble@master] Add support for phpbench

https://gerrit.wikimedia.org/r/726681

Change 726681 merged by jenkins-bot:

[integration/quibble@master] Add support for phpbench

https://gerrit.wikimedia.org/r/726681

Change 726880 had a related patch set uploaded (by Jforrester; author: Kosta Harlan):

[integration/config@master] jjb: Bump Quibble jobs to 1.1.0

https://gerrit.wikimedia.org/r/726880

Mentioned in SAL (#wikimedia-releng) [2021-10-07T13:04:47Z] <hashar> Updating Jenkins jobs for Quibble 1.1.0 # T291549

Change 726880 merged by jenkins-bot:

[integration/config@master] jjb: Bump almost all Quibble jobs to 1.1.0

https://gerrit.wikimedia.org/r/726880

Change 727429 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/core@master] [WIP] Add phpbench to dev requirements

https://gerrit.wikimedia.org/r/727429

Change 727443 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[integration/quibble@master] phpbench: Run composer install first

https://gerrit.wikimedia.org/r/727443

Change 726653 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/GrowthExperiments@master] Add composer entry for phpbench

https://gerrit.wikimedia.org/r/726653

Change 727443 merged by jenkins-bot:

[integration/quibble@master] phpbench: Run composer install first

https://gerrit.wikimedia.org/r/727443

dpifke added a subscriber: dpifke.

If this experiment turns out to be useful, Performance Team would definitely like to schedule some time to figure out how it fits in with existing tools.

Thanks @dpifke. The patches attached here are to facilitate experimentation (it helps to have CI be able to run your patch if you want to play around with the tool); I'm drafting a blog post that explains the tool and how to use it but doesn't make any suggestions about best practices or patterns to use around what type of code to test, types of assertions to make, etc. If you'd like to review the post or chat (here in phab or elsewhere) before I send that around, I'd be happy to. I was thinking that once the explanatory post is available, folks could experiment with building on top of the core patch or make their own extension patches, then we could have some evaluation to decide when/if/how to document and promote this as something in our toolkit. Does that sound OK?

Thanks @dpifke. The patches attached here are to facilitate experimentation (it helps to have CI be able to run your patch if you want to play around with the tool); I'm drafting a blog post that explains the tool and how to use it but doesn't make any suggestions about best practices or patterns to use around what type of code to test, types of assertions to make, etc. If you'd like to review the post or chat (here in phab or elsewhere) before I send that around, I'd be happy to. I was thinking that once the explanatory post is available, folks could experiment with building on top of the core patch or make their own extension patches, then we could have some evaluation to decide when/if/how to document and promote this as something in our toolkit. Does that sound OK?

https://phabricator.wikimedia.org/phame/post/view/257/benchmarking_mediawiki_with_phpbench/ if you'd like to review / comment on it (cc @Krinkle)

I get a permission error trying to view the linked draft.

I get a permission error trying to view the linked draft.

I can't figure out how to share the phab link so here's the PDF of that post.

Change 741974 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[integration/quibble@master] phpbench: Support aggregate reports

https://gerrit.wikimedia.org/r/741974

Change 741976 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[integration/config@master] jjb: phpbench aggregate reports for MW core

https://gerrit.wikimedia.org/r/741976

I get a permission error trying to view the linked draft.

I can't figure out how to share the phab link so here's the PDF of that post.

https://phabricator.wikimedia.org/phame/post/view/257/benchmarking_mediawiki_with_phpbench/ it's published now.

I also added a patch for quibble implementing the before/after aggregate report, and an example config to run that for mediawiki/core, like Fresnel is run currently. Review/opinions/thoughts wanted. :)

Change 741974 merged by jenkins-bot:

[integration/quibble@master] phpbench: Support aggregate reports

https://gerrit.wikimedia.org/r/741974

Change 758952 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[integration/quibble@master] release: Quibble 1.4.0

https://gerrit.wikimedia.org/r/758952

Change 758952 merged by jenkins-bot:

[integration/quibble@master] release: Quibble 1.4.0

https://gerrit.wikimedia.org/r/758952

Change 763559 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] jjb: update Quibble jobs from 1.3.0 to 1.4.0

https://gerrit.wikimedia.org/r/763559

Mentioned in SAL (#wikimedia-releng) [2022-02-21T07:31:03Z] <hashar> Switching Quibble jobs from Quibble 1.3.0 to 1.4.0 # T300340 T291549 T225730

Change 763559 merged by jenkins-bot:

[integration/config@master] jjb: update Quibble jobs from 1.3.0 to 1.4.0

https://gerrit.wikimedia.org/r/763559

Change 767749 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] jjb: update Quibble jobs from 1.3.0 to 1.4.3

https://gerrit.wikimedia.org/r/767749

Change 767749 merged by jenkins-bot:

[integration/config@master] jjb: update Quibble jobs from 1.3.0 to 1.4.3

https://gerrit.wikimedia.org/r/767749

Change 726653 abandoned by Kosta Harlan:

[mediawiki/extensions/GrowthExperiments@master] Add composer entry for phpbench

Reason:

squashed

https://gerrit.wikimedia.org/r/726653

Change 741976 merged by jenkins-bot:

[integration/config@master] jjb: Add mediawiki-phpbench-patch-docker, add GrowthExperiments as first test

https://gerrit.wikimedia.org/r/741976

Change 773807 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] jjb: drop THING_SUBNAME in mediawiki-phpbench-patch-docker

https://gerrit.wikimedia.org/r/773807

Change 773807 merged by jenkins-bot:

[integration/config@master] jjb: drop THING_SUBNAME in mediawiki-phpbench-patch-docker

https://gerrit.wikimedia.org/r/773807

Change 773869 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[integration/config@master] jjb: Adjust phpbench args

https://gerrit.wikimedia.org/r/773869

Change 774379 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[integration/quibble@master] phpbench: Use git-checkout instead of git-switch

https://gerrit.wikimedia.org/r/774379

Change 773869 merged by jenkins-bot:

[integration/config@master] jjb: Adjust phpbench args

https://gerrit.wikimedia.org/r/773869

Change 774379 merged by jenkins-bot:

[integration/quibble@master] phpbench: Use git-checkout instead of git-switch

https://gerrit.wikimedia.org/r/774379

Change 774410 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/quibble@master] release: Quibble 1.4.5

https://gerrit.wikimedia.org/r/774410

Change 774415 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] dockerfiles update to Quibble 1.4.5

https://gerrit.wikimedia.org/r/774415

Change 774410 merged by jenkins-bot:

[integration/quibble@master] release: Quibble 1.4.5

https://gerrit.wikimedia.org/r/774410

Change 774415 merged by jenkins-bot:

[integration/config@master] dockerfiles update to Quibble 1.4.5

https://gerrit.wikimedia.org/r/774415

Change 774427 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] jjb: mediawiki-phpbench-patch-docker to Quibble 1.4.5

https://gerrit.wikimedia.org/r/774427

Change 774427 merged by jenkins-bot:

[integration/config@master] jjb: mediawiki-phpbench-patch-docker to Quibble 1.4.5

https://gerrit.wikimedia.org/r/774427

I get a permission error trying to view the linked draft.

I can't figure out how to share the phab link so here's the PDF of that post.

https://phabricator.wikimedia.org/phame/post/view/257/benchmarking_mediawiki_with_phpbench/ it's published now.

I also added a patch for quibble implementing the before/after aggregate report, and an example config to run that for mediawiki/core, like Fresnel is run currently. Review/opinions/thoughts wanted. :)

Update: here's an example patch implementing aggregate reporting for GrowthExperiments https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GrowthExperiments/+/722594

Change 722594 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] phpbench: tests for LinkRecommendationFilter and Special:Homepage

https://gerrit.wikimedia.org/r/722594

Change 813194 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[integration/config@master] zuul: Run non-voting phpbench job for MW core patches

https://gerrit.wikimedia.org/r/813194

@kostajh there are two patches pending to add phpbench to mediawiki/core:

https://gerrit.wikimedia.org/r/c/mediawiki/core/+/727429testing: Add phpbench support for performance regression testing
https://gerrit.wikimedia.org/r/c/integration/config/+/813194zuul: Run non-voting phpbench job for MW core patches

That would adds a bit of overhead and complexity for a single test case and I am not sure whether it is worth pursuing?

@kostajh there are two patches pending to add phpbench to mediawiki/core:

https://gerrit.wikimedia.org/r/c/mediawiki/core/+/727429testing: Add phpbench support for performance regression testing
https://gerrit.wikimedia.org/r/c/integration/config/+/813194zuul: Run non-voting phpbench job for MW core patches

That would adds a bit of overhead and complexity for a single test case and I am not sure whether it is worth pursuing?

I don't think there's any enthusiasm from the developer community about this, so we can probably abandon these.

Change 727429 abandoned by Kosta Harlan:

[mediawiki/core@master] testing: Add phpbench support for performance regression testing

Reason:

https://gerrit.wikimedia.org/r/727429

Change 813194 abandoned by Kosta Harlan:

[integration/config@master] zuul: Run non-voting phpbench job for MW core patches

Reason:

https://gerrit.wikimedia.org/r/813194