Page MenuHomePhabricator

Fatal error: request has exceeded memory limit in /srv/mediawiki/php-1.33.0-wmf.8/vendor/guzzlehttp/psr7/src/Stream.php on line 97
Closed, ResolvedPublic

Description

Noticed in Fatal-Monitor after deploying wmf.8 to all wikis:

Fatal error: request has exceeded memory limit in /srv/mediawiki/php-1.33.0-wmf.8/vendor/guzzlehttp/psr7/src/Stream.php on line 97
[{exception_id}] {exception_url} PHP Fatal Error from line 97 of /srv/mediawiki/php-1.33.0-wmf.8/vendor/guzzlehttp/psr7/src/Stream.php: request has exceeded memory limit

Event Timeline

zeljkofilipin triaged this task as Unbreak Now! priority.Dec 13 2018, 3:03 PM
zeljkofilipin created this task.
Restricted Application changed the subtype of this task from "Release" to "Task". · View Herald TranscriptDec 13 2018, 3:03 PM
Restricted Application added subscribers: Liuxinyu970226, TerraCodes, Aklapper. · View Herald Transcript
Reedy added a comment.Dec 13 2018, 3:16 PM

It’s definitely not a dupe, I guess it’s related to adding guzzle though...

Will look in a bit

Agreed. Not a dupe, but appears related. I don't have an immediate answer for what's happening, but I'm looking as well.

From fatal.log:

2018-12-13 08:11:10 [XBIUGQpAEDYAAAJ4RBkAAACM] mw1289 commonswiki 1.33.0-wmf.8 fatal ERROR: [XBIUGQpAEDYAAAJ4RBkAAACM] /w/api.php   PHP Fatal Error from line 97 of /srv/mediawiki/php-1.33.0-wmf.8/vendor/guzzlehttp/psr7/src/Stream.php: request has exceeded memory limit {"fatal_exception":{"class":"ErrorException","message":"PHP Fatal Error: request has exceeded memory limit","code":16777217,"file":"/srv/mediawiki/php-1.33.0-wmf.8/vendor/guzzlehttp/psr7/src/Stream.php","line":97,"trace":"#0 /srv/mediawiki/php-1.33.0-wmf.8/vendor/guzzlehttp/psr7/src/Stream.php(97): stream_get_contents()
#1 /srv/mediawiki/php-1.33.0-wmf.8/includes/http/GuzzleHttpRequest.php(127): GuzzleHttp\\Psr7\\Stream->getContents()
#2 /srv/mediawiki/php-1.33.0-wmf.8/includes/upload/UploadFromUrl.php(276): GuzzleHttpRequest->execute()
#3 /srv/mediawiki/php-1.33.0-wmf.8/includes/upload/UploadFromUrl.php(195): UploadFromUrl->reallyFetchFile(array)
#4 /srv/mediawiki/php-1.33.0-wmf.8/includes/api/ApiUpload.php(71): UploadFromUrl->fetchFile()
#5 /srv/mediawiki/php-1.33.0-wmf.8/includes/api/ApiMain.php(1595): ApiUpload->execute()
#6 /srv/mediawiki/php-1.33.0-wmf.8/includes/api/ApiMain.php(531): ApiMain->executeAction()
#7 /srv/mediawiki/php-1.33.0-wmf.8/includes/api/ApiMain.php(502): ApiMain->executeActionWithErrorHandling()
#8 /srv/mediawiki/php-1.33.0-wmf.8/api.php(87): ApiMain->execute()
#9 /srv/mediawiki/w/api.php(3): include(string)
#10 {main}"},"exception_id":"XBIUGQpAEDYAAAJ4RBkAAACM","exception_url":"/w/api.php","caught_by":"mwe_handler"}
Reedy added a comment.Dec 13 2018, 3:57 PM

Doesn’t surprise me that it’s an upload from url request

Might be something we need to file upstream if it’s poor memory handling

I wonder if it’s worth pushing that request type back to php/curl in the meantime if it’s more than just an edge case

I think this is my error and not Guzzle's.

Guzzle uses streams rather than CURLOPT_WRITEFUNCTION style callbacks (http://docs.guzzlephp.org/en/stable/request-options.html#sink). Internally (assuming curl is available) Guzzle uses CURLOPT_WRITEFUNCTION and its own callback to write to the sink. See vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php line 379.

While I like this model, it is a mismatch for our current codebase. I'd pushed this complexity to callers (see my class comment on GuzzleHttpRequest). This was one of my early tasks after joining WMF and I under-appreciated the implications.

At this point, I'd suggest changing the HttpRequestFactory default back to curl (if available) or php (otherwise) and only using Guzzle if explicitly requested. This will effective disable Guzzle for the moment, and solve the production issues. I can then revise my implementation outside of an "Unbreak Now!" context and reactivate use of Guzzle when that's done. Presumably, that won't go live until after the holidays at this point. Guzzle should happily ride along as a vendor library in the meantime - no need to roll back anything there.

Patch incoming.

Reedy added a comment.Dec 13 2018, 4:55 PM

I wonder if we should change beta back to using guzzle in mw-config, so we have some amount of testing in the WMF wild?

Http::$httpEngine = 'guzzle';

Change 479490 had a related patch set uploaded (by BPirkle; owner: BPirkle):
[mediawiki/core@master] Temporarily remove guzzle as default for HttpRequestFactory

https://gerrit.wikimedia.org/r/479490

Change 479493 had a related patch set uploaded (by Reedy; owner: BPirkle):
[mediawiki/core@wmf/1.33.0-wmf.8] Temporarily remove guzzle as default for HttpRequestFactory

https://gerrit.wikimedia.org/r/479493

Doesn’t surprise me that it’s an upload from url request

...which made me wonder if https://phabricator.wikimedia.org/T211900 could be related (though reported a bit later than this task)

Reedy added a comment.Dec 13 2018, 6:09 PM

Doesn’t surprise me that it’s an upload from url request

...which made me wonder if https://phabricator.wikimedia.org/T211900 could be related (though reported a bit later than this task)

It's possible if there's some other edge cases in the implementation. Worth getting them to try again when we have this patch merged and deployed

Change 479490 merged by jenkins-bot:
[mediawiki/core@master] Temporarily remove guzzle as default for HttpRequestFactory

https://gerrit.wikimedia.org/r/479490

Change 479493 merged by jenkins-bot:
[mediawiki/core@wmf/1.33.0-wmf.8] Temporarily remove guzzle as default for HttpRequestFactory

https://gerrit.wikimedia.org/r/479493

Mentioned in SAL (#wikimedia-operations) [2018-12-13T22:31:15Z] <reedy@deploy1001> Synchronized php-1.33.0-wmf.8/includes/http/HttpRequestFactory.php: T211886 (duration: 00m 44s)

Reedy lowered the priority of this task from Unbreak Now! to Normal.Dec 13 2018, 10:33 PM
Reedy removed a project: Patch-For-Review.

Doesn't block the train anymore

@BPirkle want to use a separate bug for fixing the issues in the Guzzle implementation in MW?

Looks good. Upload of a large file to test.wikipedia.org, which failed earlier today with the the "exceeded memory limit" error, now succeeds.

I agree this is possibly related to T211900.

Yes, I will create a separate task for fixing the Guzzle implementation issues.

Reedy closed this task as Resolved.Dec 13 2018, 10:40 PM
Reedy assigned this task to BPirkle.
Reedy added subscribers: Yann, Framawiki.