Consider httpd for quibble instead of php built-in server
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	kostajh
	Jun 6 2019, 2:23 PM

Description

I haven't done extensive tests, but after using httpd and php -S a fair amount for local development, httpd seems to be the winner in performance:

httpd:

❯ ab -n 500 -c 10 http://kafes.local/index.php/Special:BlankPage
This is ApacheBench, Version 2.3 <$Revision: 1843412 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking kafes.local (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Finished 500 requests


Server Software:        Apache/2.4.39
Server Hostname:        kafes.local
Server Port:            80

Document Path:          /index.php/Special:BlankPage
Document Length:        50625 bytes

Concurrency Level:      10
Time taken for tests:   45.375 seconds
Complete requests:      500
Failed requests:        342
   (Connect: 0, Receive: 0, Length: 342, Exceptions: 0)
Total transferred:      25449234 bytes
HTML transferred:       25242234 bytes
Requests per second:    11.02 [#/sec] (mean)
Time per request:       907.493 [ms] (mean)
Time per request:       90.749 [ms] (mean, across all concurrent requests)
Transfer rate:          547.72 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:   236  893 188.4    860    1420
Waiting:      205  804 178.2    778    1407
Total:        236  893 188.4    860    1420

Percentage of the requests served within a certain time (ms)
  50%    860
  66%    930
  75%    997
  80%   1054
  90%   1190
  95%   1258
  98%   1324
  99%   1416
 100%   1420 (longest request)

php -S:

❯ ab -n 500 -c 10 http://localhost:8888/index.php/Special:BlankPage
This is ApacheBench, Version 2.3 <$Revision: 1843412 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Finished 500 requests


Server Software:
Server Hostname:        localhost
Server Port:            8888

Document Path:          /index.php/Special:BlankPage
Document Length:        51357 bytes

Concurrency Level:      10
Time taken for tests:   203.485 seconds
Complete requests:      500
Failed requests:        499
   (Connect: 0, Receive: 0, Length: 499, Exceptions: 0)
Total transferred:      25502981 bytes
HTML transferred:       25320981 bytes
Requests per second:    2.46 [#/sec] (mean)
Time per request:       4069.707 [ms] (mean)
Time per request:       406.971 [ms] (mean, across all concurrent requests)
Transfer rate:          122.39 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       1
Processing:   395 4017 675.9   3881    7155
Waiting:      389 4004 674.6   3872    7150
Total:        395 4018 675.9   3881    7155

Percentage of the requests served within a certain time (ms)
  50%   3881
  66%   4060
  75%   4213
  80%   4369
  90%   4718
  95%   5180
  98%   5968
  99%   6691
 100%   7155 (longest request)

Details

Related Changes in Gerrit:

Subject	Repo	Branch	Lines +/-
Wait until the recent changes are updated	mediawiki/core	REL1_35	+9 -0
(wip/poc/avert your eyes): quibble drives docker	integration/quibble	master	+354 -94
Release Quibble 0.0.46	integration/quibble	master	+74 -2
Wait until the recent changes are updated	mediawiki/core	master	+9 -0
Installer hardcodes paths	integration/quibble	master	+48 -41
Quibble full run with Apache	integration/config	master	+31 -0
dockerfiles: quibble-apache missed php.ini file	integration/config	master	+10 -1
Image for quibble serving from apache	integration/config	master	+185 -0
[WIP] Apache service for Quibble	integration/config	master	+828 -2
Dockerfile launches external apache	integration/quibble	master	+177 -2
Adapt API URL to the current webserver	integration/quibble	master	+34 -11
Implement webserver engines as subclasses	integration/quibble	master	+71 -41
Release Quibble 0.0.45	integration/quibble	master	+54 -2
Manage web backend outside of commands	integration/quibble	master	+55 -88
Manage database backend outside of commands	integration/quibble	master	+78 -12
Flag for "external" webserver	integration/quibble	master	+56 -30
Configurable web host and port	integration/quibble	master	+73 -60

Related Objects
Search...

Status	Subtype	Assigned	Task
Open		None	T76586 K3. Run Flow tests with Parsoid on Jenkins
Resolved	Feature	kostajh	T218534 Run Parsoid service in quibble
Open		None	T225730 Reduce runtime of MW shared gate Jenkins jobs to 5 min
Open		None	T226869 Run browser tests in parallel
Declined		None	T291476 Migrate Wikibase selenium tests to Quibble+Apache, enable concurrency
Resolved		None	T285649 Switch all Quibble Selenium and api-testing jobs to use apache
Resolved		None	T276428 Introduce non-voting jobs with quibble+apache
Resolved		awight	T225218 Consider httpd for quibble instead of php built-in server
Resolved		Lucas_Werkmeister_WMDE	T298682 Wikibase\Repo\Store\Sql\SqlIdGenerator::generateNewId deadlock on wb_id_counters when running selenium tests in parallel
Resolved		None	T337862 Try to run browser tests of WMDE-TechWish projects in parallel
Resolved		Dreamy_Jazz	T340823 Fix CheckUser selenium tests so they can run concurrently
Resolved		Krinkle	T199393 Selenium tests sometimes fail due to deadlock in User::addToDatabase from api.php?action=createaccount
Resolved		hashar	T342088 Run Quibble Mariadb server with innodb_print_all_deadlocks
Resolved		None	T340943 TwoColConflict selenium tests run in parallel trigger db errors
Open		Peter	T344754 Browser tests video capture is shared between tests
Open		None	T345149 Special:RecentChanges Selenium test has a race condition when running concurrently
Resolved	Spike	zeljkofilipin	T402397 Increase maxInstances for WebdriverIO test in core
Open	Spike	zeljkofilipin	T402927 Investigate why increasing maxInstances for WebdriverIO causes tests to fail

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

kostajh mentioned this in T225248: Consider moving browser based tests (Selenium and QUnit) to a non-voting pipeline.Jun 7 2019, 2:02 AM

hashar mentioned this in T225496: Improve caching in CI tests.Jun 12 2019, 8:00 AM

Would you be able to run the same benchmark for hhvm as well?
hhvm -m server -p 8888 -d "hhvm.static_file.extensions[svg]=image/svg+xml"

Nice result, I was not expecting php -S to be THAT slower.

side note

On another task, Timo mentioned requests are not compressed, though I don't think it is a performance issue, that shows up that the built-in server uses the PHP options for CLI.

Serving a page through php -S I got:

PHP_SAPI	cli-server
opcache.enable	On
opcache.enable_cli	Off

But that does not seem to change anything from my loal testing.

end of side note

So yeah, looks like we need Quibble to learn how to start one of Apache/lighttpd/nginx and add that as a dependency.

Would you be able to run the same benchmark for hhvm as well?

I don't have hhvm set up on my machine, but if I get it configured, then sure :)

So yeah, looks like we need Quibble to learn how to start one of Apache/lighttpd/nginx and add that as a dependency.

I started experimenting with this; I started with Apache then abandoned it because of the difficulties of running as non-root, then tried nginx but that has its own issues with kick off php-fpm and nginx from within Quibble.

Would it work to do something like a CLI argument for --webserver=http://apache:8080 in which case Quibble routes browser requests to http://apache:8080. In integration/config we would then need to do docker run for an image containing Apache + PHP-(version), and after Quibble is done we'd need to stop and remove that container.

In T225218#5253448, @kostajh wrote:

I started experimenting with this; I started with Apache then abandoned it because of the difficulties of running as non-root, then tried nginx but that has its own issues with kick off php-fpm and nginx from within Quibble.

Apache I could imagine you can not bind to tcp port 80 since that is only for root and I am not sure how that would work in an unprivileged container. That can be worked around with some linux capability, but Quibble / MediaWiki tests already support a port > 1024. So Apache should really be achievable.

Nginx, I have no idea, if you have hints I am willing to learn!

Thinking long, Quibble might well be able to detect/pick a httpd server depending on whats is available in the environment (eg Apache > nginx > lighttpd > php -s).

Would it work to do something like a CLI argument for --webserver=http://apache:8080 in which case Quibble routes browser requests to http://apache:8080. In integration/config we would then need to do docker run for an image containing Apache + PHP-(version), and after Quibble is done we'd need to stop and remove that container.

Then we will have to orchestrate containers and I am not really willing to do that with the current CI stack. The same issue I had with the patch to spawn Parsoid as a backend service :-\ Really that should be implemented with Helm and Kubernetes.

I am pretty sure we can hook Apache from Quibble :]

Change 516729 had a related patch set uploaded (by Kosta Harlan; owner: Kosta Harlan):
[integration/quibble@master] (wip) Add option for using Apache as server

https://gerrit.wikimedia.org/r/516729

gerritbot added a project: Patch-For-Review.Jun 13 2019, 2:33 AM

kostajh mentioned this in rQUIBBLEe9f93c7d3534: (wip) Add option for using Apache as server.Jun 13 2019, 2:36 AM

Krinkle mentioned this in T225730: Reduce runtime of MW shared gate Jenkins jobs to 5 min.Jun 13 2019, 3:29 PM

Preliminary test run with Apache was not that thrilling: the selenium page.js test passed in 300 seconds instead of 345 with PHP built in server. I’ll poke at it some more though.

Some benchmarks with quibble + this patch + docker on macOS.

Starting the container

cd ~/quibble-builds && rm src/LocalSettings.php; docker run -it --rm \
  --name quibble \
  --entrypoint=bash \
    -e ZUUL_PROJECT=mediawiki/core \
    -v /Users/kharlan/quibble-builds/cache:/cache \
    -v /Users/kharlan/quibble-builds/log:/workspace/log \
    -v /Users/kharlan/quibble-builds/ref:/srv/git:ro \
    -v /Users/kharlan/quibble-builds/src:/workspace/src \
    quibble-apache

built-in PHP + sqlite

quibble --skip-deps --skip-zuul --db sqlite --db-dir /workspace/src/cache/ --run selenium

run 1

15 passing (318.30s)
5 skipped

run 2

15 passing (299.40s)
5 skipped

run 3 (disabled page undoable test):

14 passing (302.40s)
6 skipped

apache2 + sqlite

quibble --skip-deps --skip-zuul --db sqlite --db-dir /workspace/src/cache/ --run selenium --webserver=apache2

run 1

14 passing (246.50s)
5 skipped
1 failing

run 2

14 passing (208.10s)
5 skipped
1 failing

run 3 (disabled page undoable test)

14 passing (263.40s)
6 skipped

This error occurred on both runs with --webserver=apache2, so I disabled it for the third test run:

1) Page should be undoable:
'beforeEach-content-0.8998667415652057-Iñtërnâtiônàlizætiøn' === 'beforeEach-content-0.0023315024241947757-Iñtërnâtiônàlizætiøn'

built-in PHP + MySQL

quibble --skip-deps --skip-zuul --db mysql --db-dir /workspace/src/cache/ --run selenium

run 1

14 passing (225.20s)
6 skipped

run 2

14 passing (264.80s)
6 skipped

apache2 + MySQL

n.b. For MySQL, I had to change the db directory to /workspace/src/cache.

quibble --skip-deps --skip-zuul --db mysql --db-dir /workspace/src/cache/ --run selenium --webserver=apache2

run 1

14 passing (191.90s)
6 skipped

run 2

14 passing (195.10s)
6 skipped

awight mentioned this in T226869: Run browser tests in parallel.Jun 28 2019, 8:58 PM

Change 516729 abandoned by Kosta Harlan:
Add option for using Apache as server

Reason:
IMO let's wait for local-charts :) Although if someone wants to pick this up, feel free

https://gerrit.wikimedia.org/r/516729

Maintenance_bot removed a project: Patch-For-Review.Jul 1 2019, 3:10 PM

kostajh mentioned this in T234902: quibble + local-charts?.Oct 8 2019, 11:13 AM

Krinkle awarded a token.Oct 12 2019, 4:21 PM

Krinkle added a parent task: T225730: Reduce runtime of MW shared gate Jenkins jobs to 5 min.

Change 516729 restored by Kosta Harlan:
Add option for using Apache as server

https://gerrit.wikimedia.org/r/516729

kostajh mentioned this in T238224: Provide docker-compose based development / CI environment in MediaWiki core.Dec 9 2019, 11:20 AM

Change 568931 had a related patch set uploaded (by Kosta Harlan; owner: Kosta Harlan):
[integration/quibble@master] (wip/poc/avert your eyes): quibble drives docker

https://gerrit.wikimedia.org/r/568931

gerritbot added a project: Patch-For-Review.Jan 30 2020, 9:58 AM

I think this is a killer feature, in combination with T226869: Run browser tests in parallel. The built-in server is single-threaded, so I expect parallel browser testing to bottleneck on PHP.

However, I should mention this interesting changelog for PHP 7.4:

You can configure the built-in webserver to fork multiple workers in order to test code that requires multiple concurrent requests to the built-in webserver. Set the PHP_CLI_SERVER_WORKERS environment variable to the number of desired workers before starting the server.

Jdforrester-WMF added a parent task: T218534: Run Parsoid service in quibble.Apr 14 2020, 5:29 PM

awight mentioned this in T216819: Using Quibble with different backend than gerrit.Apr 19 2020, 9:21 AM

awight awarded a token.Apr 19 2020, 5:15 PM

@kostajh We started discussing in code review, but I wanted to move here for visibility. What do you think about heading towards --webserver=none, nor driving docker from Quibble, but instead just letting the CI job spin up an apache container pointing at the source directory? We can keep the code for a PHP standalone backend in Quibble in case of emergency. But the ideal would be to avoid adding any more complexity to Quibble, and decouple it completely from the details of how to serve pages.

My thought was already too naive--we would need to containerize the mysql server as well.

Change 590651 had a related patch set uploaded (by Awight; owner: Awight):
[integration/quibble@master] Flag for "none" webserver

https://gerrit.wikimedia.org/r/590651

awight moved this task from Backlog to Enhancement on the Quibble board.Apr 20 2020, 9:03 AM

hashar removed a project: Continuous-Integration-Infrastructure.Apr 30 2020, 8:50 PM

Apache is definitely to be considered.

I also found php 7.4+ supports multiple workers and filed T259456 for that.

awight claimed this task.Aug 19 2020, 7:00 AM

awight added a project: User-awight.

awight moved this task from Backlog to Prototype on the User-awight board.

Change 591826 had a related patch set uploaded (by Awight; owner: Awight):
[integration/quibble@master] Configurable web host and port

https://gerrit.wikimedia.org/r/591826

Change 591826 merged by jenkins-bot:
[integration/quibble@master] Configurable web host and port

https://gerrit.wikimedia.org/r/591826

Change 590651 merged by jenkins-bot:
[integration/quibble@master] Flag for "external" webserver

https://gerrit.wikimedia.org/r/590651

Change 625636 had a related patch set uploaded (by Awight; owner: Awight):
[integration/quibble@master] [WIP] Manage database backend in separate step

https://gerrit.wikimedia.org/r/625636

Change 625638 had a related patch set uploaded (by Awight; owner: Awight):
[integration/quibble@master] [WIP] Extract web backend management

https://gerrit.wikimedia.org/r/625638

Change 625636 merged by jenkins-bot:
[integration/quibble@master] Manage database backend outside of commands

https://gerrit.wikimedia.org/r/625636

Change 625638 merged by jenkins-bot:
[integration/quibble@master] Manage web backend outside of commands

https://gerrit.wikimedia.org/r/625638

Change 628138 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/quibble@master] Release Quibble 0.0.45

https://gerrit.wikimedia.org/r/628138

Change 628138 merged by jenkins-bot:
[integration/quibble@master] Release Quibble 0.0.45

https://gerrit.wikimedia.org/r/628138

Change 628759 had a related patch set uploaded (by Awight; owner: Awight):
[integration/quibble@master] Implement webserver engines as subclasses

https://gerrit.wikimedia.org/r/628759

Change 630164 had a related patch set uploaded (by Awight; owner: Awight):
[integration/config@master] [WIP] Apache service for Quibble

https://gerrit.wikimedia.org/r/630164

Daimona subscribed.Oct 7 2020, 11:26 AM

Change 628759 merged by jenkins-bot:
[integration/quibble@master] Implement webserver engines as subclasses

https://gerrit.wikimedia.org/r/628759

Change 640432 had a related patch set uploaded (by Awight; owner: Awight):
[integration/quibble@master] Adapt API URL to the current webserver

https://gerrit.wikimedia.org/r/640432

Change 640432 merged by jenkins-bot:
[integration/quibble@master] Adapt API URL to the current webserver

https://gerrit.wikimedia.org/r/640432

Change 516729 merged by jenkins-bot:
[integration/quibble@master] Dockerfile launches external apache

https://gerrit.wikimedia.org/r/516729

Change 644485 had a related patch set uploaded (by Awight; owner: Awight):
[integration/config@master] [WIP] CI image for quibble serving from apache

https://gerrit.wikimedia.org/r/644485

Change 630164 abandoned by Awight:
[integration/config@master] [WIP] Apache service for Quibble

Reason:
This was a stale fork.

https://gerrit.wikimedia.org/r/630164

Change 644485 merged by jenkins-bot:
[integration/config@master] Image for quibble serving from apache

https://gerrit.wikimedia.org/r/644485

Change 647202 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] dockerfiles: quibble-apache missed php.ini file

https://gerrit.wikimedia.org/r/647202

Change 647202 merged by jenkins-bot:
[integration/config@master] dockerfiles: quibble-apache missed php.ini file

https://gerrit.wikimedia.org/r/647202

Mentioned in SAL (#wikimedia-releng) [2020-12-09T10:28:17Z] <hashar> Successfully tagged docker-registry.discovery.wmnet/releng/quibble-apache:0.0.1 T225218

Change 647199 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Quibble full run with Apache

https://gerrit.wikimedia.org/r/647199

Change 647704 had a related patch set uploaded (by Awight; owner: Awight):
[integration/quibble@master] Installer hardcodes paths

https://gerrit.wikimedia.org/r/647704

Change 647199 merged by jenkins-bot:
[integration/config@master] Quibble full run with Apache

https://gerrit.wikimedia.org/r/647199

Change 647704 merged by jenkins-bot:
[integration/quibble@master] Installer hardcodes paths

https://gerrit.wikimedia.org/r/647704

@awight @hashar the deadlock error seen on https://integration.wikimedia.org/ci/view/CI/job/integration-quibble-apache-fullrun/9/console looks to be the same as T199393. We could mark those particular API tests (tests/api-testing/action/UserContribs.js) as skipped or add some pause in between account creations. Or we could fix T199393 :)

Nice catch. Given the test is subject to a race condition, I guess it should be disabled entirely or indeed fixed but that seems to not be a trivial task. :-(

Daniel had a patch https://gerrit.wikimedia.org/r/c/mediawiki/tools/api-testing/+/530343/ . Seems the account creation happens in parallel so the order is not guaranteed. Maybe php -S kind of forced the serialization of the requests, but with Apache they are processed in parallel and we hit the bug. Fun one!

hashar mentioned this in T199393: Selenium tests sometimes fail due to deadlock in User::addToDatabase from api.php?action=createaccount.Dec 18 2020, 8:00 PM

hashar added a subtask: T199393: Selenium tests sometimes fail due to deadlock in User::addToDatabase from api.php?action=createaccount.

Change 654443 had a related patch set uploaded (by Awight; owner: Awight):
[mediawiki/core@master] Wait until the recent changes are updated

https://gerrit.wikimedia.org/r/654443

Change 654443 merged by jenkins-bot:
[mediawiki/core@master] Wait until the recent changes are updated

https://gerrit.wikimedia.org/r/654443

ReleaseTaggerBot added a project: MW-1.36-notes (1.36.0-wmf.26; 2021-01-12).Jan 5 2021, 8:00 PM

@hashar The experimental quibble-apache job passes now, so the obstacles are cleared to deploying a non-voting job, at your leisure!

The tests are still running in sequence at the moment, so I would have expected the apache+php-fpm job to have similar or slightly better performance. The results so far are ambiguous, all I can say is that we don't seem to be incurring any huge extra costs. (Unfairly,) directly comparing timings for the test suites (browser tests for core and Vector; API tests; excluding install time for both suites) from two arbitrary quibble-fullrun jobs, one with apache and the other with standalone PHP:

	browser tests	API tests
PHP-standalone	56s	152s
apache+php-fpm	72s	118s

Total run time for the apache job was 8m50s. Cherry-picking a metric, total execution time for the server-limited suites is 9% lower for the apache job. Please take my garbage data with a grain of salt, obviously these are just ballpark numbers until we have enough equivalent runs to look at the average. My only interpretation so far is that it shows nothing is going horribly wrong.

In T225218#6724222, @awight wrote:

@hashar The experimental quibble-apache job passes now, so the obstacles are cleared to deploying a non-voting job, at your leisure!

Just a caveat, there are going to be issues with api-testing and Selenium tests failing due to the absence of single-threaded processing. For example in this build https://integration.wikimedia.org/ci/view/CI/job/integration-quibble-apache-fullrun/12/console we can see some tests failing. I think it's OK to deploy as a non-voting job but we should probably create tasks (or a single tracking task?) to fix these as they come up.

The tests are still running in sequence at the moment, so I would have expected the apache+php-fpm job to have similar or slightly better performance. The results so far are ambiguous, all I can say is that we don't seem to be incurring any huge extra costs. (Unfairly,) directly comparing timings for the test suites (browser tests for core and Vector; API tests; excluding install time for both suites) from two arbitrary quibble-fullrun jobs, one with apache and the other with standalone PHP:

browser tests API tests

PHP-standalone 56s 152s

apache+php-fpm 72s 118s

Total run time for the apache job was 8m50s. Cherry-picking a metric, total execution time for the server-limited suites is 9% lower for the apache job. Please take my garbage data with a grain of salt, obviously these are just ballpark numbers until we have enough equivalent runs to look at the average. My only interpretation so far is that it shows nothing is going horribly wrong.

There is probably some optimization we could do for php-fpm/apache integration to speed things up. For example tweaking the opcache setting to cache PHP files indefinitely.

I thought I posted somewhere but I can't find it now; early on in this patch's history, I had experimented with benchmarking on a DigitalOcean box and didn't find any improvements with Apache over PHP's built-in server. Even so, the fact that we are surfacing race conditions in our tests, as well as triggering application errors like T199393, highlights the usefulness of using Apache with Quibble.

In T225218#6724831, @kostajh wrote:

[...] the fact that we are surfacing race conditions in our tests, as well as triggering application errors like T199393, highlights the usefulness of using Apache with Quibble.

This about sums it up for me as well. I feel like we've reached a point where quibble-apache went from being an investigation, to a goal I'd like to commit to. Running UI test suites concurrently is a bonus, but the robustness benefits and similarity to production are more compelling.

Change 654804 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/quibble@master] Release Quibble 0.0.46

https://gerrit.wikimedia.org/r/654804

Change 654804 merged by jenkins-bot:
[integration/quibble@master] Release Quibble 0.0.46

https://gerrit.wikimedia.org/r/654804

Mentioned in SAL (#wikimedia-releng) [2021-01-07T16:01:18Z] <hashar> Tag Quibble 0.0.46 @ df9e75329ab # T225218 T266441 T263500

In T225218#6724222, @awight wrote:

@hashar The experimental quibble-apache job passes now, so the obstacles are cleared to deploying a non-voting job, at your leisure!

I assume the non-voting job would be just to make sure nothing melts down, and then we'd make it voting and transition away from the php built-in server jobs. Once the non-voting job is added, and again when it becomes voting, we should probably send a heads up to wikitech-l to let folks know about race condition failures with Selenium and API testing tests, and have a single phab task for tracking patches that disable those tests as well as patches for making them more resilient.

Krinkle subscribed.Jan 14 2021, 1:36 AM

Jdforrester-WMF subscribed.Jan 14 2021, 5:24 PM