MultiHttpClient currently uses curl if it is available, with fallback to PhpHttpRequest if curl is not available (T139169). While functional, the fallback to PhpHttpRequest introduces an undesirable dependency on mediawiki code.
Convert MultiHttpClient to use Guzzle instead. Guzzle includes built-in support for both concurrency, and automatic fallback to php streams if curl is unavailable.
See T202110 for related information, and T202143 for a security review request for Guzzle.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | BPirkle | T137926 Support running MediaWiki without 'curl' PHP extension | |||
| Open | None | T202352 Convert MultiHttpClient to use Guzzle |
Change 454346 had a related patch set uploaded (by BPirkle; owner: BPirkle):
[mediawiki/core@master] Convert MultiHttpClient to use Guzzle
Failure of the above patch set is expected, as Guzzle is still awaiting security review. Just pushing what I have so far, in case anyone cares to look at where this is going.
Security review was completed in Dec. 2018. The patch should now pass tests and is awaiting code review.
Change 454346 merged by jenkins-bot:
[mediawiki/core@master] Convert MultiHttpClient to use Guzzle
Change 494663 had a related patch set uploaded (by Hashar; owner: Hashar):
[mediawiki/core@master] Revert "Convert MultiHttpClient to use Guzzle"
Sorry I had to revert the patch in mediawiki/core. That broke maintenance scripts on Beta-Cluster-Infrastructure. From T217733:
echo 'aawiki'; /usr/local/bin/mwscript update.php --wiki=aawiki --quick aawiki #!/usr/bin/env php Warning: Invalid argument: option: 6 in /srv/mediawiki-staging/php-master/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php on line 56 Warning: Invalid argument: option: 6 in /srv/mediawiki-staging/php-master/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php on line 56 Warning: Invalid argument: function: not string, closure, or array in /srv/mediawiki-staging/php-master/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php on line 108 ... Fatal error: Uncaught exception 'ConfigException' with message 'Failed to load configuration from etcd: cURL error 23: Failed writing header (see http://curl.haxx.se/libcurl/c/libcurl-errors.html) in /srv/mediawiki-staging/php-master/includes/config/EtcdConfig.php:200
Stack trace:
#0 /srv/mediawiki-staging/php-master/includes/config/EtcdConfig.php(123): EtcdConfig->load()
#1 /srv/mediawiki-staging/wmf-config/etcd.php(43): EtcdConfig->getModifiedIndex()
#2 /srv/mediawiki-staging/wmf-config/etcd.php(49): wmfSetupEtcd()
#3 /srv/mediawiki-staging/wmf-config/CommonSettings.php(134): wmfEtcdConfig()
#4 /srv/mediawiki-staging/php-master/LocalSettings.php(5): include()
#5 /srv/mediawiki-staging/php-master/includes/Setup.php(105): include()
#6 /srv/mediawiki-staging/php-master/maintenance/doMaintenance.php(81): include()
#7 /srv/mediawiki-staging/php-master/maintenance/update.php(248): include()
#8 /srv/mediawiki-staging/multiversion/MWScript.php(100): include()
#9 {main}To reproduce I was simply running any maintenance script:
ssh deployment-deploy01.deployment-prep.eqiad.wmflabs echo 1|mwscript eval.php --wiki=enwiki
Change 494663 merged by jenkins-bot:
[mediawiki/core@master] Revert "Convert MultiHttpClient to use Guzzle"
@hashar , sorry for the inconvenience, I'm glad reverting the change made things happy again.
I've tried a few things to investigate/reproduce/debug the issue, mostly tracing around in the mediawiki-config repo and EtcdConfig.php and trying semi-random things with eval.php. Thus far, I haven't made much progress, and I keep ending up in parts of our production environment that I'm not yet familiar with.
Does anyone have any suggestions on how I could approach figuring out why this failed?
Not specific to etcd: https://logstash-beta.wmflabs.org/goto/ada0e105ccdfa5bfbd83a38049431c5c
6 is CURLMOPT_MAXCONNECTS which is apparently getting passed to curl_setopt instead of curl_multi_setopt. So presumably triggered by https://gerrit.wikimedia.org/r/c/mediawiki/core/+/454346/16/includes/libs/MultiHttpClient.php#227. From a very superficial glance at the Guzzle code, it does not seem to support custom curl_multi options at all.
No worries. We just revert when something is broken, regardless of the quality or effort put in the original patch. It is a good way to easily come back to last good known state. Although it is annoying to have its patch revoked, it is at least not entirely erased and can be restored/polished back again.
That being said, I forgot to look at beta logstash to get actual traces (thank you @Tgr). So here are the traces for the few warnings all originating from the same request to https://en.wikipedia.beta.wmflabs/wiki/Main_Page?debug=true Note those traces got triggered by a deferred update from EventBus:
PHP Warning: Invalid argument: function: not string, closure, or array
#0 [internal function]: MWExceptionHandler::handleError(integer, string, string, integer, array, array)
#1 /srv/mediawiki/php-master/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php(108): curl_multi_exec(resource, integer)
#2 /srv/mediawiki/php-master/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php(125): GuzzleHttp\Handler\CurlMultiHandler->tick()
#3 /srv/mediawiki/php-master/vendor/guzzlehttp/promises/src/Promise.php(246): GuzzleHttp\Handler\CurlMultiHandler->execute(boolean)
#4 /srv/mediawiki/php-master/vendor/guzzlehttp/promises/src/Promise.php(223): GuzzleHttp\Promise\Promise->invokeWaitFn()
#5 /srv/mediawiki/php-master/vendor/guzzlehttp/promises/src/Promise.php(267): GuzzleHttp\Promise\Promise->waitIfPending()
#6 /srv/mediawiki/php-master/vendor/guzzlehttp/promises/src/Promise.php(225): GuzzleHttp\Promise\Promise->invokeWaitList()
#7 /srv/mediawiki/php-master/vendor/guzzlehttp/promises/src/Promise.php(62): GuzzleHttp\Promise\Promise->waitIfPending()
#8 /srv/mediawiki/php-master/vendor/guzzlehttp/promises/src/EachPromise.php(101): GuzzleHttp\Promise\Promise->wait()
#9 /srv/mediawiki/php-master/vendor/guzzlehttp/promises/src/Promise.php(246): Closure$GuzzleHttp\Promise\EachPromise::createPromise(boolean)
#10 /srv/mediawiki/php-master/vendor/guzzlehttp/promises/src/Promise.php(223): GuzzleHttp\Promise\Promise->invokeWaitFn()
#11 /srv/mediawiki/php-master/vendor/guzzlehttp/promises/src/Promise.php(267): GuzzleHttp\Promise\Promise->waitIfPending()
#12 /srv/mediawiki/php-master/vendor/guzzlehttp/promises/src/Promise.php(225): GuzzleHttp\Promise\Promise->invokeWaitList()
#13 /srv/mediawiki/php-master/vendor/guzzlehttp/promises/src/Promise.php(62): GuzzleHttp\Promise\Promise->waitIfPending()
#14 /srv/mediawiki/php-master/includes/libs/MultiHttpClient.php(281): GuzzleHttp\Promise\Promise->wait()
#15 /srv/mediawiki/php-master/includes/libs/MultiHttpClient.php(181): MultiHttpClient->runMultiGuzzle(array, array)
#16 /srv/mediawiki/php-master/includes/libs/MultiHttpClient.php(141): MultiHttpClient->runMulti(array, array)
#17 /srv/mediawiki/php-master/extensions/EventBus/includes/EventBus.php(151): MultiHttpClient->run(array, array)
#18 /srv/mediawiki/php-master/extensions/EventBus/includes/JobQueueEventBus.php(157): EventBus->send(array, integer)
#19 /srv/mediawiki/php-master/includes/deferred/MWCallableUpdate.php(34): Closure$JobQueueEventBus::doBatchPush()
#20 /srv/mediawiki/php-master/includes/deferred/DeferredUpdates.php(270): MWCallableUpdate->doUpdate()
#21 /srv/mediawiki/php-master/includes/deferred/DeferredUpdates.php(228): DeferredUpdates::runUpdate(MWCallableUpdate, Wikimedia\Rdbms\LBFactoryMulti, string, integer)
#22 /srv/mediawiki/php-master/includes/deferred/DeferredUpdates.php(140): DeferredUpdates::execute(array, string, integer)
#23 /srv/mediawiki/php-master/includes/MediaWiki.php(909): DeferredUpdates::doUpdates(string)
#24 /srv/mediawiki/php-master/includes/MediaWiki.php(733): MediaWiki->restInPeace(string, boolean)
#25 [internal function]: Closure$MediaWiki::doPostOutputShutdown()
#26 {main}And since the warning does not abort, the code keep proceeding emitting two PHP Warning: Invalid argument: option: 6 (maybe because two queries are made??). The traces:
#0 [internal function]: MWExceptionHandler::handleError(integer, string, string, integer, array, array)
#1 /srv/mediawiki/php-master/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php(56): curl_setopt_array(resource, array)
#2 /srv/mediawiki/php-master/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php(63): GuzzleHttp\Handler\CurlFactory->create(GuzzleHttp\Psr7\Request, array)
#3 /srv/mediawiki/php-master/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php(522): GuzzleHttp\Handler\CurlMultiHandler->__invoke(GuzzleHttp\Psr7\Request, array)
#4 /srv/mediawiki/php-master/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php(146): GuzzleHttp\Handler\CurlFactory::retryFailedRewind(GuzzleHttp\Handler\CurlMultiHandler, GuzzleHttp\Handler\EasyHandle, array)
#5 /srv/mediawiki/php-master/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php(102): GuzzleHttp\Handler\CurlFactory::finishError(GuzzleHttp\Handler\CurlMultiHandler, GuzzleHttp\Handler\EasyHandle, GuzzleHttp\Handler\CurlFactory)
#6 /srv/mediawiki/php-master/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php(183): GuzzleHttp\Handler\CurlFactory::finish(GuzzleHttp\Handler\CurlMultiHandler, GuzzleHttp\Handler\EasyHandle, GuzzleHttp\Handler\CurlFactory)
#7 /srv/mediawiki/php-master/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php(110): GuzzleHttp\Handler\CurlMultiHandler->processMessages()
#8 /srv/mediawiki/php-master/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php(125): GuzzleHttp\Handler\CurlMultiHandler->tick()
#9 /srv/mediawiki/php-master/vendor/guzzlehttp/promises/src/Promise.php(246): GuzzleHttp\Handler\CurlMultiHandler->execute(boolean)
#10 /srv/mediawiki/php-master/vendor/guzzlehttp/promises/src/Promise.php(223): GuzzleHttp\Promise\Promise->invokeWaitFn()
#11 /srv/mediawiki/php-master/vendor/guzzlehttp/promises/src/Promise.php(267): GuzzleHttp\Promise\Promise->waitIfPending()
#12 /srv/mediawiki/php-master/vendor/guzzlehttp/promises/src/Promise.php(225): GuzzleHttp\Promise\Promise->invokeWaitList()
#13 /srv/mediawiki/php-master/vendor/guzzlehttp/promises/src/Promise.php(62): GuzzleHttp\Promise\Promise->waitIfPending()
#14 /srv/mediawiki/php-master/vendor/guzzlehttp/promises/src/EachPromise.php(101): GuzzleHttp\Promise\Promise->wait()
#15 /srv/mediawiki/php-master/vendor/guzzlehttp/promises/src/Promise.php(246): Closure$GuzzleHttp\Promise\EachPromise::createPromise(boolean)
#16 /srv/mediawiki/php-master/vendor/guzzlehttp/promises/src/Promise.php(223): GuzzleHttp\Promise\Promise->invokeWaitFn()
#17 /srv/mediawiki/php-master/vendor/guzzlehttp/promises/src/Promise.php(267): GuzzleHttp\Promise\Promise->waitIfPending()
#18 /srv/mediawiki/php-master/vendor/guzzlehttp/promises/src/Promise.php(225): GuzzleHttp\Promise\Promise->invokeWaitList()
#19 /srv/mediawiki/php-master/vendor/guzzlehttp/promises/src/Promise.php(62): GuzzleHttp\Promise\Promise->waitIfPending()
#20 /srv/mediawiki/php-master/includes/libs/MultiHttpClient.php(281): GuzzleHttp\Promise\Promise->wait()
#21 /srv/mediawiki/php-master/includes/libs/MultiHttpClient.php(181): MultiHttpClient->runMultiGuzzle(array, array)
#22 /srv/mediawiki/php-master/includes/libs/MultiHttpClient.php(141): MultiHttpClient->runMulti(array, array)
#23 /srv/mediawiki/php-master/extensions/EventBus/includes/EventBus.php(151): MultiHttpClient->run(array, array)
#24 /srv/mediawiki/php-master/extensions/EventBus/includes/JobQueueEventBus.php(157): EventBus->send(array, integer)
#25 /srv/mediawiki/php-master/includes/deferred/MWCallableUpdate.php(34): Closure$JobQueueEventBus::doBatchPush()
#26 /srv/mediawiki/php-master/includes/deferred/DeferredUpdates.php(270): MWCallableUpdate->doUpdate()
#27 /srv/mediawiki/php-master/includes/deferred/DeferredUpdates.php(228): DeferredUpdates::runUpdate(MWCallableUpdate, Wikimedia\Rdbms\LBFactoryMulti, string, integer)
#28 /srv/mediawiki/php-master/includes/deferred/DeferredUpdates.php(140): DeferredUpdates::execute(array, string, integer)
#29 /srv/mediawiki/php-master/includes/MediaWiki.php(909): DeferredUpdates::doUpdates(string)
#30 /srv/mediawiki/php-master/includes/MediaWiki.php(733): MediaWiki->restInPeace(string, boolean)
#31 [internal function]: Closure$MediaWiki::doPostOutputShutdown()
#32 {main}#0 [internal function]: MWExceptionHandler::handleError(integer, string, string, integer, array, array)
#1 /srv/mediawiki/php-master/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php(56): curl_setopt_array(resource, array)
#2 /srv/mediawiki/php-master/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php(63): GuzzleHttp\Handler\CurlFactory->create(GuzzleHttp\Psr7\Request, array)
#3 /srv/mediawiki/php-master/vendor/guzzlehttp/guzzle/src/Handler/Proxy.php(27): GuzzleHttp\Handler\CurlMultiHandler->__invoke(GuzzleHttp\Psr7\Request, array)
#4 /srv/mediawiki/php-master/vendor/guzzlehttp/guzzle/src/Handler/Proxy.php(51): Closure$GuzzleHttp\Handler\Proxy::wrapSync(GuzzleHttp\Psr7\Request, array)
#5 /srv/mediawiki/php-master/vendor/guzzlehttp/guzzle/src/PrepareBodyMiddleware.php(37): Closure$GuzzleHttp\Handler\Proxy::wrapStreaming(GuzzleHttp\Psr7\Request, array)
#6 /srv/mediawiki/php-master/vendor/guzzlehttp/guzzle/src/Middleware.php(30): GuzzleHttp\PrepareBodyMiddleware->__invoke(GuzzleHttp\Psr7\Request, array)
#7 /srv/mediawiki/php-master/vendor/guzzlehttp/guzzle/src/RedirectMiddleware.php(70): Closure$GuzzleHttp\Middleware::cookies#2(GuzzleHttp\Psr7\Request, array)
#8 /srv/mediawiki/php-master/vendor/guzzlehttp/guzzle/src/Middleware.php(60): GuzzleHttp\RedirectMiddleware->__invoke(GuzzleHttp\Psr7\Request, array)
#9 /srv/mediawiki/php-master/vendor/guzzlehttp/guzzle/src/HandlerStack.php(67): Closure$GuzzleHttp\Middleware::httpErrors#2(GuzzleHttp\Psr7\Request, array)
#10 /srv/mediawiki/php-master/vendor/guzzlehttp/guzzle/src/Client.php(277): GuzzleHttp\HandlerStack->__invoke(GuzzleHttp\Psr7\Request, array)
#11 /srv/mediawiki/php-master/vendor/guzzlehttp/guzzle/src/Client.php(125): GuzzleHttp\Client->transfer(GuzzleHttp\Psr7\Request, array)
#12 /srv/mediawiki/php-master/includes/libs/MultiHttpClient.php(278): GuzzleHttp\Client->requestAsync(string, GuzzleHttp\Psr7\Uri, array)
#13 /srv/mediawiki/php-master/includes/libs/MultiHttpClient.php(181): MultiHttpClient->runMultiGuzzle(array, array)
#14 /srv/mediawiki/php-master/includes/libs/MultiHttpClient.php(141): MultiHttpClient->runMulti(array, array)
#15 /srv/mediawiki/php-master/extensions/EventBus/includes/EventBus.php(151): MultiHttpClient->run(array, array)
#16 /srv/mediawiki/php-master/extensions/EventBus/includes/JobQueueEventBus.php(157): EventBus->send(array, integer)
#17 /srv/mediawiki/php-master/includes/deferred/MWCallableUpdate.php(34): Closure$JobQueueEventBus::doBatchPush()
#18 /srv/mediawiki/php-master/includes/deferred/DeferredUpdates.php(270): MWCallableUpdate->doUpdate()
#19 /srv/mediawiki/php-master/includes/deferred/DeferredUpdates.php(228): DeferredUpdates::runUpdate(MWCallableUpdate, Wikimedia\Rdbms\LBFactoryMulti, string, integer)
#20 /srv/mediawiki/php-master/includes/deferred/DeferredUpdates.php(140): DeferredUpdates::execute(array, string, integer)
#21 /srv/mediawiki/php-master/includes/MediaWiki.php(909): DeferredUpdates::doUpdates(string)
#22 /srv/mediawiki/php-master/includes/MediaWiki.php(733): MediaWiki->restInPeace(string, boolean)
#23 [internal function]: Closure$MediaWiki::doPostOutputShutdown()
#24 {main}And then:
/srv/mediawiki/php-master/vendor/guzzlehttp/guzzle/src/Client.php:310
PHP Warning: Invalid argument: formdata: (need Array or Object)
#0 [internal function]: MWExceptionHandler::handleError(integer, string, string, integer, array, array)
#1 /srv/mediawiki/php-master/vendor/guzzlehttp/guzzle/src/Client.php(310): http_build_query(string, string, string)
#2 /srv/mediawiki/php-master/vendor/guzzlehttp/guzzle/src/Client.php(273): GuzzleHttp\Client->applyOptions(GuzzleHttp\Psr7\Request, array)
#3 /srv/mediawiki/php-master/vendor/guzzlehttp/guzzle/src/Client.php(125): GuzzleHttp\Client->transfer(GuzzleHttp\Psr7\Request, array)
#4 /srv/mediawiki/php-master/includes/libs/MultiHttpClient.php(278): GuzzleHttp\Client->requestAsync(string, GuzzleHttp\Psr7\Uri, array)
#5 /srv/mediawiki/php-master/includes/libs/MultiHttpClient.php(181): MultiHttpClient->runMultiGuzzle(array, array)
#6 /srv/mediawiki/php-master/includes/libs/MultiHttpClient.php(141): MultiHttpClient->runMulti(array, array)
#7 /srv/mediawiki/php-master/extensions/EventBus/includes/EventBus.php(151): MultiHttpClient->run(array, array)
#8 /srv/mediawiki/php-master/extensions/EventBus/includes/JobQueueEventBus.php(157): EventBus->send(array, integer)
#9 /srv/mediawiki/php-master/includes/deferred/MWCallableUpdate.php(34): Closure$JobQueueEventBus::doBatchPush()
#10 /srv/mediawiki/php-master/includes/deferred/DeferredUpdates.php(270): MWCallableUpdate->doUpdate()
#11 /srv/mediawiki/php-master/includes/deferred/DeferredUpdates.php(228): DeferredUpdates::runUpdate(MWCallableUpdate, Wikimedia\Rdbms\LBFactoryMulti, string, integer)
#12 /srv/mediawiki/php-master/includes/deferred/DeferredUpdates.php(140): DeferredUpdates::execute(array, string, integer)
#13 /srv/mediawiki/php-master/includes/MediaWiki.php(909): DeferredUpdates::doUpdates(string)
#14 /srv/mediawiki/php-master/includes/MediaWiki.php(733): MediaWiki->restInPeace(string, boolean)
#15 [internal function]: Closure$MediaWiki::doPostOutputShutdown()
#16 {main}And finally EventBus yields Unable to deliver all events: cURL error 23: Failed writing header (see http://curl.haxx.se/libcurl/c/libcurl-errors.html) with events:
{
"sha1": "4765f63ee6e9681009adeffa8a3a75368cba904d",
"database": "enwiki",
"mediawiki_signature": "06d751a81fd34674a0e5e1abcf63478ec7aa33889426fddaf3cd916c508d15f0",
"meta": {
"dt": "2019-03-06T09:24:47+00:00",
"domain": "en.wikipedia.beta.wmflabs.org",
"topic": "mediawiki.job.wikibase-addUsagesForPage",
"id": "af2aec20-3ff1-11e9-8d4e-fa163ed6f6de",
"uri": "https://en.wikipedia.beta.wmflabs.org/wiki/Main_Page",
"request_id": "XH@R3qwQBHcAAD-G9BwAAAAD"
},
"page_title": "Main_Page",
"type": "wikibase-addUsagesForPage",
"params": {
"usages": {
"Q371142#S": {
"aspect": "S",
"modifier": null,
"entityId": "Q371142"
}
},
"pageId": 1
},
"page_namespace": 0
}There are some more warnings/error but they look similar :)
Note that MultiHttpClient is used by EtcdConfig, which is one of very few classes that can be interacted with before MediaWiki is initialised.
Specifically, from entry point (e.g. index.php) -> WebStart -> Setup -> LocalSettings -> wmf-config/CommonSettings -> wmf-config/etcd.
I don't know if this explains the issue, but it's relatively uncommon for classes to be using during init. (It was mentioned in the trace of T217733)
Hence MultiHttpClient mustn't use any globals or singletons. (I don't know if it does, but if so, that'd likely be a problem.)
Hope that helps.
Change 495742 had a related patch set uploaded (by BPirkle; owner: BPirkle):
[mediawiki/core@master] Testing code for T202352. Not to be merged. Will be abandoned.
Change 552527 had a related patch set uploaded (by Physikerwelt; owner: Physikerwelt):
[mediawiki/core@master] WIP: MWHttpRequest regression demo
Change 495742 abandoned by BPirkle:
Testing code for T202352. Not to be merged. Will be abandoned.
Reason:
Abandoning, this has grown extremely stale.
This seems like a lot of work just to avoid a one line "sudo apt install php-curl" command.
I'd rather just document curl as being required and remove runMultiHttp().
We're revisiting this, but we'd like to take a different approach considering that the previous attempt wasn't successful.
The idea is that we add a new class with the Guzzle implementation to live alongside the old MultiHttpClient. This prevents us from addressing too many things at the same time. For the HttpRequestFactory, we'd have a flag which can be switched to true for the factory to return the new implementation which should help with a more gradual rollout. The whole process would have to take multiple steps in order for us to do it right and prevent "doing too much":
We will also be changing the way that we supply data to it. GuzzleMultiHttpClient would require requests, previously assoc. arrays, to be instances of the PSR RequestInterface, and will return instances of ResponseInterface (as the requests themselves are only returned when using runMulti specifically, doesn't seem to be used past the point where the response is actually received either).
There are instances where MultiHttpClient is only used in order not to pull in the entirety of Mediawiki, not because of its concurrency option. In this case we should instead opt for pulling in PSR ClientInterface.