Possibly related to, or a duplicate of T211806: Passing in the "body" request option as an array to send a POST request has been deprecated, so adding the same people: @Reedy @Legoktm @BPirkle

It’s definitely not a dupe, I guess it’s related to adding guzzle though...

Will look in a bit

Agreed. Not a dupe, but appears related. I don't have an immediate answer for what's happening, but I'm looking as well.

From fatal.log:

2018-12-13 08:11:10 [XBIUGQpAEDYAAAJ4RBkAAACM] mw1289 commonswiki 1.33.0-wmf.8 fatal ERROR: [XBIUGQpAEDYAAAJ4RBkAAACM] /w/api.php   PHP Fatal Error from line 97 of /srv/mediawiki/php-1.33.0-wmf.8/vendor/guzzlehttp/psr7/src/Stream.php: request has exceeded memory limit {"fatal_exception":{"class":"ErrorException","message":"PHP Fatal Error: request has exceeded memory limit","code":16777217,"file":"/srv/mediawiki/php-1.33.0-wmf.8/vendor/guzzlehttp/psr7/src/Stream.php","line":97,"trace":"#0 /srv/mediawiki/php-1.33.0-wmf.8/vendor/guzzlehttp/psr7/src/Stream.php(97): stream_get_contents()
#1 /srv/mediawiki/php-1.33.0-wmf.8/includes/http/GuzzleHttpRequest.php(127): GuzzleHttp\\Psr7\\Stream->getContents()
#2 /srv/mediawiki/php-1.33.0-wmf.8/includes/upload/UploadFromUrl.php(276): GuzzleHttpRequest->execute()
#3 /srv/mediawiki/php-1.33.0-wmf.8/includes/upload/UploadFromUrl.php(195): UploadFromUrl->reallyFetchFile(array)
#4 /srv/mediawiki/php-1.33.0-wmf.8/includes/api/ApiUpload.php(71): UploadFromUrl->fetchFile()
#5 /srv/mediawiki/php-1.33.0-wmf.8/includes/api/ApiMain.php(1595): ApiUpload->execute()
#6 /srv/mediawiki/php-1.33.0-wmf.8/includes/api/ApiMain.php(531): ApiMain->executeAction()
#7 /srv/mediawiki/php-1.33.0-wmf.8/includes/api/ApiMain.php(502): ApiMain->executeActionWithErrorHandling()
#8 /srv/mediawiki/php-1.33.0-wmf.8/api.php(87): ApiMain->execute()
#9 /srv/mediawiki/w/api.php(3): include(string)
#10 {main}"},"exception_id":"XBIUGQpAEDYAAAJ4RBkAAACM","exception_url":"/w/api.php","caught_by":"mwe_handler"}

Doesn’t surprise me that it’s an upload from url request

Might be something we need to file upstream if it’s poor memory handling

I wonder if it’s worth pushing that request type back to php/curl in the meantime if it’s more than just an edge case

I think this is my error and not Guzzle's.

Guzzle uses streams rather than CURLOPT_WRITEFUNCTION style callbacks (http://docs.guzzlephp.org/en/stable/request-options.html#sink). Internally (assuming curl is available) Guzzle uses CURLOPT_WRITEFUNCTION and its own callback to write to the sink. See vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php line 379.

While I like this model, it is a mismatch for our current codebase. I'd pushed this complexity to callers (see my class comment on GuzzleHttpRequest). This was one of my early tasks after joining WMF and I under-appreciated the implications.

At this point, I'd suggest changing the HttpRequestFactory default back to curl (if available) or php (otherwise) and only using Guzzle if explicitly requested. This will effective disable Guzzle for the moment, and solve the production issues. I can then revise my implementation outside of an "Unbreak Now!" context and reactivate use of Guzzle when that's done. Presumably, that won't go live until after the holidays at this point. Guzzle should happily ride along as a vendor library in the meantime - no need to roll back anything there.

Patch incoming.

I wonder if we should change beta back to using guzzle in mw-config, so we have some amount of testing in the WMF wild?

Http::$httpEngine = 'guzzle';

Change 479490 had a related patch set uploaded (by BPirkle; owner: BPirkle):
[mediawiki/core@master] Temporarily remove guzzle as default for HttpRequestFactory

https://gerrit.wikimedia.org/r/479490

Change 479493 had a related patch set uploaded (by Reedy; owner: BPirkle):
[mediawiki/core@wmf/1.33.0-wmf.8] Temporarily remove guzzle as default for HttpRequestFactory

https://gerrit.wikimedia.org/r/479493

In T211886#4820599, @Reedy wrote:

Doesn’t surprise me that it’s an upload from url request

...which made me wonder if https://phabricator.wikimedia.org/T211900 could be related (though reported a bit later than this task)

In T211886#4821205, @Aklapper wrote:

In T211886#4820599, @Reedy wrote:

Doesn’t surprise me that it’s an upload from url request

...which made me wonder if https://phabricator.wikimedia.org/T211900 could be related (though reported a bit later than this task)

It's possible if there's some other edge cases in the implementation. Worth getting them to try again when we have this patch merged and deployed

Change 479490 merged by jenkins-bot:
[mediawiki/core@master] Temporarily remove guzzle as default for HttpRequestFactory

https://gerrit.wikimedia.org/r/479490

Change 479493 merged by jenkins-bot:
[mediawiki/core@wmf/1.33.0-wmf.8] Temporarily remove guzzle as default for HttpRequestFactory

https://gerrit.wikimedia.org/r/479493

Mentioned in SAL (#wikimedia-operations) [2018-12-13T22:31:15Z] <reedy@deploy1001> Synchronized php-1.33.0-wmf.8/includes/http/HttpRequestFactory.php: T211886 (duration: 00m 44s)

Looks good. Upload of a large file to test.wikipedia.org, which failed earlier today with the the "exceeded memory limit" error, now succeeds.

I agree this is possibly related to T211900.

Yes, I will create a separate task for fixing the Guzzle implementation issues.