Page MenuHomePhabricator

Convert MultiHttpClient to use Guzzle
Open, Needs TriagePublic

Description

MultiHttpClient currently uses curl if it is available, with fallback to PhpHttpRequest if curl is not available (T139169). While functional, the fallback to PhpHttpRequest introduces an undesirable dependency on mediawiki code.

Convert MultiHttpClient to use Guzzle instead. Guzzle includes built-in support for both concurrency, and automatic fallback to php streams if curl is unavailable.

See T202110 for related information, and T202143 for a security review request for Guzzle.

Event Timeline

BPirkle created this task.Aug 21 2018, 12:51 AM

Change 454346 had a related patch set uploaded (by BPirkle; owner: BPirkle):
[mediawiki/core@master] Convert MultiHttpClient to use Guzzle

https://gerrit.wikimedia.org/r/454346

Failure of the above patch set is expected, as Guzzle is still awaiting security review. Just pushing what I have so far, in case anyone cares to look at where this is going.

Paladox added a subscriber: Paladox.Nov 8 2018, 3:35 AM

Security review was completed in Dec. 2018. The patch should now pass tests and is awaiting code review.

Change 454346 merged by jenkins-bot:
[mediawiki/core@master] Convert MultiHttpClient to use Guzzle

https://gerrit.wikimedia.org/r/454346

Krinkle closed this task as Resolved.Mar 5 2019, 11:34 PM
Krinkle removed a project: Patch-For-Review.
Krinkle awarded a token.

Change 494663 had a related patch set uploaded (by Hashar; owner: Hashar):
[mediawiki/core@master] Revert "Convert MultiHttpClient to use Guzzle"

https://gerrit.wikimedia.org/r/494663

hashar reopened this task as Open.Mar 6 2019, 8:57 AM
hashar added a subscriber: hashar.

Sorry I had to revert the patch in mediawiki/core. That broke maintenance scripts on Beta-Cluster-Infrastructure. From T217733:

Message
echo 'aawiki'; /usr/local/bin/mwscript update.php --wiki=aawiki --quick
aawiki
#!/usr/bin/env php
Warning: Invalid argument: option: 6 in /srv/mediawiki-staging/php-master/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php on line 56
Warning: Invalid argument: option: 6 in /srv/mediawiki-staging/php-master/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php on line 56
Warning: Invalid argument: function: not string, closure, or array in /srv/mediawiki-staging/php-master/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php on line 108
...

Fatal error: Uncaught exception 'ConfigException'
with message 'Failed to load configuration from etcd: cURL error 23: Failed writing header (see http://curl.haxx.se/libcurl/c/libcurl-errors.html)
in /srv/mediawiki-staging/php-master/includes/config/EtcdConfig.php:200
trace
Stack trace:
#0 /srv/mediawiki-staging/php-master/includes/config/EtcdConfig.php(123): EtcdConfig->load()
#1 /srv/mediawiki-staging/wmf-config/etcd.php(43): EtcdConfig->getModifiedIndex()
#2 /srv/mediawiki-staging/wmf-config/etcd.php(49): wmfSetupEtcd()
#3 /srv/mediawiki-staging/wmf-config/CommonSettings.php(134): wmfEtcdConfig()
#4 /srv/mediawiki-staging/php-master/LocalSettings.php(5): include()
#5 /srv/mediawiki-staging/php-master/includes/Setup.php(105): include()
#6 /srv/mediawiki-staging/php-master/maintenance/doMaintenance.php(81): include()
#7 /srv/mediawiki-staging/php-master/maintenance/update.php(248): include()
#8 /srv/mediawiki-staging/multiversion/MWScript.php(100): include()
#9 {main}

To reproduce I was simply running any maintenance script:

ssh deployment-deploy01.deployment-prep.eqiad.wmflabs
echo 1|mwscript eval.php --wiki=enwiki

Change 494663 merged by jenkins-bot:
[mediawiki/core@master] Revert "Convert MultiHttpClient to use Guzzle"

https://gerrit.wikimedia.org/r/494663

@hashar , sorry for the inconvenience, I'm glad reverting the change made things happy again.

I've tried a few things to investigate/reproduce/debug the issue, mostly tracing around in the mediawiki-config repo and EtcdConfig.php and trying semi-random things with eval.php. Thus far, I haven't made much progress, and I keep ending up in parts of our production environment that I'm not yet familiar with.

Does anyone have any suggestions on how I could approach figuring out why this failed?

Tgr added a comment.Mar 8 2019, 7:23 AM

Not specific to etcd: https://logstash-beta.wmflabs.org/goto/ada0e105ccdfa5bfbd83a38049431c5c
6 is CURLMOPT_MAXCONNECTS which is apparently getting passed to curl_setopt instead of curl_multi_setopt. So presumably triggered by https://gerrit.wikimedia.org/r/c/mediawiki/core/+/454346/16/includes/libs/MultiHttpClient.php#227. From a very superficial glance at the Guzzle code, it does not seem to support custom curl_multi options at all.

hashar added a comment.Mar 8 2019, 9:28 AM

@hashar , sorry for the inconvenience, I'm glad reverting the change made things happy again.

No worries. We just revert when something is broken, regardless of the quality or effort put in the original patch. It is a good way to easily come back to last good known state. Although it is annoying to have its patch revoked, it is at least not entirely erased and can be restored/polished back again.

That being said, I forgot to look at beta logstash to get actual traces (thank you @Tgr). So here are the traces for the few warnings all originating from the same request to https://en.wikipedia.beta.wmflabs/wiki/Main_Page?debug=true Note those traces got triggered by a deferred update from EventBus:

PHP Warning: Invalid argument: function: not string, closure, or array
	#0 [internal function]: MWExceptionHandler::handleError(integer, string, string, integer, array, array)
#1 /srv/mediawiki/php-master/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php(108): curl_multi_exec(resource, integer)
#2 /srv/mediawiki/php-master/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php(125): GuzzleHttp\Handler\CurlMultiHandler->tick()
#3 /srv/mediawiki/php-master/vendor/guzzlehttp/promises/src/Promise.php(246): GuzzleHttp\Handler\CurlMultiHandler->execute(boolean)
#4 /srv/mediawiki/php-master/vendor/guzzlehttp/promises/src/Promise.php(223): GuzzleHttp\Promise\Promise->invokeWaitFn()
#5 /srv/mediawiki/php-master/vendor/guzzlehttp/promises/src/Promise.php(267): GuzzleHttp\Promise\Promise->waitIfPending()
#6 /srv/mediawiki/php-master/vendor/guzzlehttp/promises/src/Promise.php(225): GuzzleHttp\Promise\Promise->invokeWaitList()
#7 /srv/mediawiki/php-master/vendor/guzzlehttp/promises/src/Promise.php(62): GuzzleHttp\Promise\Promise->waitIfPending()
#8 /srv/mediawiki/php-master/vendor/guzzlehttp/promises/src/EachPromise.php(101): GuzzleHttp\Promise\Promise->wait()
#9 /srv/mediawiki/php-master/vendor/guzzlehttp/promises/src/Promise.php(246): Closure$GuzzleHttp\Promise\EachPromise::createPromise(boolean)
#10 /srv/mediawiki/php-master/vendor/guzzlehttp/promises/src/Promise.php(223): GuzzleHttp\Promise\Promise->invokeWaitFn()
#11 /srv/mediawiki/php-master/vendor/guzzlehttp/promises/src/Promise.php(267): GuzzleHttp\Promise\Promise->waitIfPending()
#12 /srv/mediawiki/php-master/vendor/guzzlehttp/promises/src/Promise.php(225): GuzzleHttp\Promise\Promise->invokeWaitList()
#13 /srv/mediawiki/php-master/vendor/guzzlehttp/promises/src/Promise.php(62): GuzzleHttp\Promise\Promise->waitIfPending()
#14 /srv/mediawiki/php-master/includes/libs/MultiHttpClient.php(281): GuzzleHttp\Promise\Promise->wait()
#15 /srv/mediawiki/php-master/includes/libs/MultiHttpClient.php(181): MultiHttpClient->runMultiGuzzle(array, array)
#16 /srv/mediawiki/php-master/includes/libs/MultiHttpClient.php(141): MultiHttpClient->runMulti(array, array)
#17 /srv/mediawiki/php-master/extensions/EventBus/includes/EventBus.php(151): MultiHttpClient->run(array, array)
#18 /srv/mediawiki/php-master/extensions/EventBus/includes/JobQueueEventBus.php(157): EventBus->send(array, integer)
#19 /srv/mediawiki/php-master/includes/deferred/MWCallableUpdate.php(34): Closure$JobQueueEventBus::doBatchPush()
#20 /srv/mediawiki/php-master/includes/deferred/DeferredUpdates.php(270): MWCallableUpdate->doUpdate()
#21 /srv/mediawiki/php-master/includes/deferred/DeferredUpdates.php(228): DeferredUpdates::runUpdate(MWCallableUpdate, Wikimedia\Rdbms\LBFactoryMulti, string, integer)
#22 /srv/mediawiki/php-master/includes/deferred/DeferredUpdates.php(140): DeferredUpdates::execute(array, string, integer)
#23 /srv/mediawiki/php-master/includes/MediaWiki.php(909): DeferredUpdates::doUpdates(string)
#24 /srv/mediawiki/php-master/includes/MediaWiki.php(733): MediaWiki->restInPeace(string, boolean)
#25 [internal function]: Closure$MediaWiki::doPostOutputShutdown()
#26 {main}

And since the warning does not abort, the code keep proceeding emitting two PHP Warning: Invalid argument: option: 6 (maybe because two queries are made??). The traces:

#0 [internal function]: MWExceptionHandler::handleError(integer, string, string, integer, array, array)
#1 /srv/mediawiki/php-master/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php(56): curl_setopt_array(resource, array)
#2 /srv/mediawiki/php-master/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php(63): GuzzleHttp\Handler\CurlFactory->create(GuzzleHttp\Psr7\Request, array)
#3 /srv/mediawiki/php-master/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php(522): GuzzleHttp\Handler\CurlMultiHandler->__invoke(GuzzleHttp\Psr7\Request, array)
#4 /srv/mediawiki/php-master/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php(146): GuzzleHttp\Handler\CurlFactory::retryFailedRewind(GuzzleHttp\Handler\CurlMultiHandler, GuzzleHttp\Handler\EasyHandle, array)
#5 /srv/mediawiki/php-master/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php(102): GuzzleHttp\Handler\CurlFactory::finishError(GuzzleHttp\Handler\CurlMultiHandler, GuzzleHttp\Handler\EasyHandle, GuzzleHttp\Handler\CurlFactory)
#6 /srv/mediawiki/php-master/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php(183): GuzzleHttp\Handler\CurlFactory::finish(GuzzleHttp\Handler\CurlMultiHandler, GuzzleHttp\Handler\EasyHandle, GuzzleHttp\Handler\CurlFactory)
#7 /srv/mediawiki/php-master/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php(110): GuzzleHttp\Handler\CurlMultiHandler->processMessages()
#8 /srv/mediawiki/php-master/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php(125): GuzzleHttp\Handler\CurlMultiHandler->tick()
#9 /srv/mediawiki/php-master/vendor/guzzlehttp/promises/src/Promise.php(246): GuzzleHttp\Handler\CurlMultiHandler->execute(boolean)
#10 /srv/mediawiki/php-master/vendor/guzzlehttp/promises/src/Promise.php(223): GuzzleHttp\Promise\Promise->invokeWaitFn()
#11 /srv/mediawiki/php-master/vendor/guzzlehttp/promises/src/Promise.php(267): GuzzleHttp\Promise\Promise->waitIfPending()
#12 /srv/mediawiki/php-master/vendor/guzzlehttp/promises/src/Promise.php(225): GuzzleHttp\Promise\Promise->invokeWaitList()
#13 /srv/mediawiki/php-master/vendor/guzzlehttp/promises/src/Promise.php(62): GuzzleHttp\Promise\Promise->waitIfPending()
#14 /srv/mediawiki/php-master/vendor/guzzlehttp/promises/src/EachPromise.php(101): GuzzleHttp\Promise\Promise->wait()
#15 /srv/mediawiki/php-master/vendor/guzzlehttp/promises/src/Promise.php(246): Closure$GuzzleHttp\Promise\EachPromise::createPromise(boolean)
#16 /srv/mediawiki/php-master/vendor/guzzlehttp/promises/src/Promise.php(223): GuzzleHttp\Promise\Promise->invokeWaitFn()
#17 /srv/mediawiki/php-master/vendor/guzzlehttp/promises/src/Promise.php(267): GuzzleHttp\Promise\Promise->waitIfPending()
#18 /srv/mediawiki/php-master/vendor/guzzlehttp/promises/src/Promise.php(225): GuzzleHttp\Promise\Promise->invokeWaitList()
#19 /srv/mediawiki/php-master/vendor/guzzlehttp/promises/src/Promise.php(62): GuzzleHttp\Promise\Promise->waitIfPending()
#20 /srv/mediawiki/php-master/includes/libs/MultiHttpClient.php(281): GuzzleHttp\Promise\Promise->wait()
#21 /srv/mediawiki/php-master/includes/libs/MultiHttpClient.php(181): MultiHttpClient->runMultiGuzzle(array, array)
#22 /srv/mediawiki/php-master/includes/libs/MultiHttpClient.php(141): MultiHttpClient->runMulti(array, array)
#23 /srv/mediawiki/php-master/extensions/EventBus/includes/EventBus.php(151): MultiHttpClient->run(array, array)
#24 /srv/mediawiki/php-master/extensions/EventBus/includes/JobQueueEventBus.php(157): EventBus->send(array, integer)
#25 /srv/mediawiki/php-master/includes/deferred/MWCallableUpdate.php(34): Closure$JobQueueEventBus::doBatchPush()
#26 /srv/mediawiki/php-master/includes/deferred/DeferredUpdates.php(270): MWCallableUpdate->doUpdate()
#27 /srv/mediawiki/php-master/includes/deferred/DeferredUpdates.php(228): DeferredUpdates::runUpdate(MWCallableUpdate, Wikimedia\Rdbms\LBFactoryMulti, string, integer)
#28 /srv/mediawiki/php-master/includes/deferred/DeferredUpdates.php(140): DeferredUpdates::execute(array, string, integer)
#29 /srv/mediawiki/php-master/includes/MediaWiki.php(909): DeferredUpdates::doUpdates(string)
#30 /srv/mediawiki/php-master/includes/MediaWiki.php(733): MediaWiki->restInPeace(string, boolean)
#31 [internal function]: Closure$MediaWiki::doPostOutputShutdown()
#32 {main}
#0 [internal function]: MWExceptionHandler::handleError(integer, string, string, integer, array, array)
#1 /srv/mediawiki/php-master/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php(56): curl_setopt_array(resource, array)
#2 /srv/mediawiki/php-master/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php(63): GuzzleHttp\Handler\CurlFactory->create(GuzzleHttp\Psr7\Request, array)
#3 /srv/mediawiki/php-master/vendor/guzzlehttp/guzzle/src/Handler/Proxy.php(27): GuzzleHttp\Handler\CurlMultiHandler->__invoke(GuzzleHttp\Psr7\Request, array)
#4 /srv/mediawiki/php-master/vendor/guzzlehttp/guzzle/src/Handler/Proxy.php(51): Closure$GuzzleHttp\Handler\Proxy::wrapSync(GuzzleHttp\Psr7\Request, array)
#5 /srv/mediawiki/php-master/vendor/guzzlehttp/guzzle/src/PrepareBodyMiddleware.php(37): Closure$GuzzleHttp\Handler\Proxy::wrapStreaming(GuzzleHttp\Psr7\Request, array)
#6 /srv/mediawiki/php-master/vendor/guzzlehttp/guzzle/src/Middleware.php(30): GuzzleHttp\PrepareBodyMiddleware->__invoke(GuzzleHttp\Psr7\Request, array)
#7 /srv/mediawiki/php-master/vendor/guzzlehttp/guzzle/src/RedirectMiddleware.php(70): Closure$GuzzleHttp\Middleware::cookies#2(GuzzleHttp\Psr7\Request, array)
#8 /srv/mediawiki/php-master/vendor/guzzlehttp/guzzle/src/Middleware.php(60): GuzzleHttp\RedirectMiddleware->__invoke(GuzzleHttp\Psr7\Request, array)
#9 /srv/mediawiki/php-master/vendor/guzzlehttp/guzzle/src/HandlerStack.php(67): Closure$GuzzleHttp\Middleware::httpErrors#2(GuzzleHttp\Psr7\Request, array)
#10 /srv/mediawiki/php-master/vendor/guzzlehttp/guzzle/src/Client.php(277): GuzzleHttp\HandlerStack->__invoke(GuzzleHttp\Psr7\Request, array)
#11 /srv/mediawiki/php-master/vendor/guzzlehttp/guzzle/src/Client.php(125): GuzzleHttp\Client->transfer(GuzzleHttp\Psr7\Request, array)
#12 /srv/mediawiki/php-master/includes/libs/MultiHttpClient.php(278): GuzzleHttp\Client->requestAsync(string, GuzzleHttp\Psr7\Uri, array)
#13 /srv/mediawiki/php-master/includes/libs/MultiHttpClient.php(181): MultiHttpClient->runMultiGuzzle(array, array)
#14 /srv/mediawiki/php-master/includes/libs/MultiHttpClient.php(141): MultiHttpClient->runMulti(array, array)
#15 /srv/mediawiki/php-master/extensions/EventBus/includes/EventBus.php(151): MultiHttpClient->run(array, array)
#16 /srv/mediawiki/php-master/extensions/EventBus/includes/JobQueueEventBus.php(157): EventBus->send(array, integer)
#17 /srv/mediawiki/php-master/includes/deferred/MWCallableUpdate.php(34): Closure$JobQueueEventBus::doBatchPush()
#18 /srv/mediawiki/php-master/includes/deferred/DeferredUpdates.php(270): MWCallableUpdate->doUpdate()
#19 /srv/mediawiki/php-master/includes/deferred/DeferredUpdates.php(228): DeferredUpdates::runUpdate(MWCallableUpdate, Wikimedia\Rdbms\LBFactoryMulti, string, integer)
#20 /srv/mediawiki/php-master/includes/deferred/DeferredUpdates.php(140): DeferredUpdates::execute(array, string, integer)
#21 /srv/mediawiki/php-master/includes/MediaWiki.php(909): DeferredUpdates::doUpdates(string)
#22 /srv/mediawiki/php-master/includes/MediaWiki.php(733): MediaWiki->restInPeace(string, boolean)
#23 [internal function]: Closure$MediaWiki::doPostOutputShutdown()
#24 {main}

And then:

/srv/mediawiki/php-master/vendor/guzzlehttp/guzzle/src/Client.php:310
PHP Warning: Invalid argument: formdata: (need Array or Object)

#0 [internal function]: MWExceptionHandler::handleError(integer, string, string, integer, array, array)
#1 /srv/mediawiki/php-master/vendor/guzzlehttp/guzzle/src/Client.php(310): http_build_query(string, string, string)
#2 /srv/mediawiki/php-master/vendor/guzzlehttp/guzzle/src/Client.php(273): GuzzleHttp\Client->applyOptions(GuzzleHttp\Psr7\Request, array)
#3 /srv/mediawiki/php-master/vendor/guzzlehttp/guzzle/src/Client.php(125): GuzzleHttp\Client->transfer(GuzzleHttp\Psr7\Request, array)
#4 /srv/mediawiki/php-master/includes/libs/MultiHttpClient.php(278): GuzzleHttp\Client->requestAsync(string, GuzzleHttp\Psr7\Uri, array)
#5 /srv/mediawiki/php-master/includes/libs/MultiHttpClient.php(181): MultiHttpClient->runMultiGuzzle(array, array)
#6 /srv/mediawiki/php-master/includes/libs/MultiHttpClient.php(141): MultiHttpClient->runMulti(array, array)
#7 /srv/mediawiki/php-master/extensions/EventBus/includes/EventBus.php(151): MultiHttpClient->run(array, array)
#8 /srv/mediawiki/php-master/extensions/EventBus/includes/JobQueueEventBus.php(157): EventBus->send(array, integer)
#9 /srv/mediawiki/php-master/includes/deferred/MWCallableUpdate.php(34): Closure$JobQueueEventBus::doBatchPush()
#10 /srv/mediawiki/php-master/includes/deferred/DeferredUpdates.php(270): MWCallableUpdate->doUpdate()
#11 /srv/mediawiki/php-master/includes/deferred/DeferredUpdates.php(228): DeferredUpdates::runUpdate(MWCallableUpdate, Wikimedia\Rdbms\LBFactoryMulti, string, integer)
#12 /srv/mediawiki/php-master/includes/deferred/DeferredUpdates.php(140): DeferredUpdates::execute(array, string, integer)
#13 /srv/mediawiki/php-master/includes/MediaWiki.php(909): DeferredUpdates::doUpdates(string)
#14 /srv/mediawiki/php-master/includes/MediaWiki.php(733): MediaWiki->restInPeace(string, boolean)
#15 [internal function]: Closure$MediaWiki::doPostOutputShutdown()
#16 {main}

And finally EventBus yields Unable to deliver all events: cURL error 23: Failed writing header (see http://curl.haxx.se/libcurl/c/libcurl-errors.html) with events:

 	{
  "sha1": "4765f63ee6e9681009adeffa8a3a75368cba904d",
  "database": "enwiki",
  "mediawiki_signature": "06d751a81fd34674a0e5e1abcf63478ec7aa33889426fddaf3cd916c508d15f0",
  "meta": {
    "dt": "2019-03-06T09:24:47+00:00",
    "domain": "en.wikipedia.beta.wmflabs.org",
    "topic": "mediawiki.job.wikibase-addUsagesForPage",
    "id": "af2aec20-3ff1-11e9-8d4e-fa163ed6f6de",
    "uri": "https://en.wikipedia.beta.wmflabs.org/wiki/Main_Page",
    "request_id": "XH@R3qwQBHcAAD-G9BwAAAAD"
  },
  "page_title": "Main_Page",
  "type": "wikibase-addUsagesForPage",
  "params": {
    "usages": {
      "Q371142#S": {
        "aspect": "S",
        "modifier": null,
        "entityId": "Q371142"
      }
    },
    "pageId": 1
  },
  "page_namespace": 0
}

There are some more warnings/error but they look similar :)

Krinkle added a comment.EditedMar 8 2019, 5:55 PM

Note that MultiHttpClient is used by EtcdConfig, which is one of very few classes that can be interacted with before MediaWiki is initialised.

Specifically, from entry point (e.g. index.php) -> WebStart -> Setup -> LocalSettings -> wmf-config/CommonSettings -> wmf-config/etcd.

I don't know if this explains the issue, but it's relatively uncommon for classes to be using during init. (It was mentioned in the trace of T217733)

Hence MultiHttpClient mustn't use any globals or singletons. (I don't know if it does, but if so, that'd likely be a problem.)

Hope that helps.

Change 495742 had a related patch set uploaded (by BPirkle; owner: BPirkle):
[mediawiki/core@master] Testing code for T202352. Not to be merged. Will be abandoned.

https://gerrit.wikimedia.org/r/495742