Page MenuHomePhabricator

internal_api_error_Exception: [22e05a83] Exception Caught: wfDiff(): popen() failed errors on English Wikipedia
Closed, ResolvedPublicPRODUCTION ERROR

Description

Keep getting MediaWiki api exceptions (output from Pywikibot):

Working on Henry Hall Dixon
Got redirected to http://viaf.org/viaf/240853956/
@@ -11 +11 @@

  • {{Authority control|VIAF=61103002}}

+ {{Authority control|VIAF=240853956}}

Following viaf redirect from 61103002 -> 240853956
ERROR: Detected MediaWiki API exception Exception; raising
<snip>
pywikibot.exceptions.OtherPageSaveError: Edit to page [[Henry Hall Dixon]] failed:
internal_api_error_Exception: [0d4c19b2] Exception Caught: wfDiff(): popen() failed
<class 'pywikibot.exceptions.OtherPageSaveError'>

Following viaf redirect from 14547908 -> 33094492
ERROR: Detected MediaWiki API exception Exception; raising
<snip>
pywikibot.exceptions.OtherPageSaveError: Edit to page [[Angelo Donati]] failed:
internal_api_error_Exception: [22e05a83] Exception Caught: wfDiff(): popen() failed
<class 'pywikibot.exceptions.OtherPageSaveError'>

Event Timeline

Multichill raised the priority of this task from to High.
Multichill updated the task description. (Show Details)
Multichill added a project: MediaWiki-API.
Multichill added a subscriber: Multichill.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 24 2015, 4:53 PM
Anomie added a subscriber: Anomie.Apr 24 2015, 4:55 PM

Exception is:

2015-04-24 13:31:54 mw1125 enwiki exception INFO: [0d4c19b2] [no req]   Exception from line 3141 of /srv/mediawiki/php-1.26wmf2/includes/GlobalFunctions.php: wfDiff(): popen() failed
#0 /srv/mediawiki/php-1.26wmf2/extensions/Echo/includes/DiffParser.php(93): wfDiff()
#1 /srv/mediawiki/php-1.26wmf2/extensions/Echo/includes/DiscussionParser.php(576): EchoDiffParser->getChangeSet()
#2 /srv/mediawiki/php-1.26wmf2/extensions/Echo/includes/DiscussionParser.php(265): EchoDiscussionParser::getMachineReadableDiff()
#3 /srv/mediawiki/php-1.26wmf2/extensions/Echo/includes/DiscussionParser.php(18): EchoDiscussionParser::getChangeInterpretationForRevision()
#4 /srv/mediawiki/php-1.26wmf2/extensions/Echo/Hooks.php(446): EchoDiscussionParser::generateEventsForRevision()
#5 /srv/mediawiki/php-1.26wmf2/includes/Hooks.php(209): EchoHooks::onArticleSaved()
#6 /srv/mediawiki/php-1.26wmf2/includes/content/ContentHandler.php(1161): Hooks::run()
#7 /srv/mediawiki/php-1.26wmf2/includes/page/WikiPage.php(1979): ContentHandler::runLegacyHooks()
#8 /srv/mediawiki/php-1.26wmf2/includes/page/Article.php(1994): WikiPage->doEditContent()
#9 /srv/mediawiki/php-1.26wmf2/includes/EditPage.php(1895): Article->__call()
#10 /srv/mediawiki/php-1.26wmf2/includes/EditPage.php(1288): EditPage->internalAttemptSave()
#11 /srv/mediawiki/php-1.26wmf2/includes/api/ApiEditPage.php(414): EditPage->attemptSave()
#12 /srv/mediawiki/php-1.26wmf2/includes/api/ApiMain.php(1043): ApiEditPage->execute()
#13 /srv/mediawiki/php-1.26wmf2/includes/api/ApiMain.php(394): ApiMain->executeAction()
#14 /srv/mediawiki/php-1.26wmf2/includes/api/ApiMain.php(367): ApiMain->executeActionWithErrorHandling()
#15 /srv/mediawiki/php-1.26wmf2/api.php(90): ApiMain->execute()
#16 /srv/mediawiki/w/api.php(3): include()
#17 {main} {"private":false}
aude added a subscriber: aude.Apr 24 2015, 4:56 PM
Exception from line 3141 of /srv/mediawiki/php-1.26wmf2/includes/GlobalFunctions.php: wfDiff(): popen() failed
#0 /srv/mediawiki/php-1.26wmf2/extensions/Echo/includes/DiffParser.php(93): wfDiff()
#1 /srv/mediawiki/php-1.26wmf2/extensions/Echo/includes/DiscussionParser.php(576): EchoDiffParser->getChangeSet()
--
#5 /srv/mediawiki/php-1.26wmf2/includes/specialpage/SpecialPage.php(384): SpecialWhatLinksHere->execute()
#6 /srv/mediawiki/php-1.26wmf2/includes/specialpage/SpecialPageFactory.php(581): SpecialPage->run()
#7 /srv/mediawiki/php-1.26wmf2/includes/MediaWiki.php(267): SpecialPageFactory::executePath()
#8 /srv/mediawiki/php-1.26wmf2/includes/MediaWiki.php(566): MediaWiki->performRequest()
#9 /srv/mediawiki/php-1.26wmf2/includes/MediaWiki.php(414): MediaWiki->main()
#10 /srv/mediawiki/php-1.26wmf2/index.php(46): MediaWiki->run()
#11 /srv/mediawiki/w/index.php(3): include()

Seems to correspond to messages like this in hhvm.log:

Apr 24 17:03:55 mw1125:  #012Warning: popen(/usr/bin/diff -u '/tmp/merge-old-IGG07I' '/tmp/merge-your-jO9Ah4',r): Invalid argument in /srv/mediawiki/php-1.26wmf2/includes/GlobalFunctions.php on line 3137

And another one:

pywikibot.exceptions.OtherPageSaveError: Edit to page [[Liza Featherstone]] failed:
internal_api_error_Exception: [4e41a70e] Exception Caught: wfDiff(): popen() failed
<class 'pywikibot.exceptions.OtherPageSaveError'>

XXN added a subscriber: XXN.Apr 24 2015, 6:51 PM

I saw about 30 of those between 1:27:26 PM GMT and 7:39:21 PM GMT.

The last one after using VisualFileChange:

API request failed (internal_api_error_Exception): [8bb13855] Exception Caught: wfDiff(): popen() failed <i>at Fri, 24 Apr 2015 19:39:21 GMT</i> <u>served by mw1125</u>
Anomie edited projects, added acl*sre-team; removed MediaWiki-API.Apr 25 2015, 11:42 AM
Anomie set Security to None.

Reviewing the logs again, this particular instance of the problem seems like it may have accidentally been fixed at around 20:50 UTC yesterday, my guess would be due to an HHVM restart associated with the deploy of Gerrit change 206440.

For the record, the problem in this instance seems to have been mostly on mw1125 (some instances on mw1146), with the first log message at 13:26:20 and the last at 20:49:32 (UTC) on 2015-04-24.

Aklapper added a subscriber: greg.Apr 27 2015, 2:43 PM

@greg: Probably fixed as per anomie's last comment but still something to keep an eye on.

BBlack added a subscriber: BBlack.Apr 29 2015, 3:17 PM

Any further updates on whether this looks fixed or not?

I'm guessing it was some sort of memory or other resource issue with HHVM being unable to create the subprocess for the popen, but that's just a guess.

At any rate, I don't see any log entries in the past few days, so we could probably close this at least until it happens again.

+1 on closing. Haven't seen this for a while.

Gone from the exception logs so closing.

ArielGlenn closed this task as Resolved.May 12 2015, 2:38 PM
XXN removed a subscriber: XXN.Jun 13 2015, 10:08 PM
mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:11 PM
Restricted Application added a subscriber: Liuxinyu970226. · View Herald TranscriptAug 28 2019, 11:11 PM
Aklapper removed a subscriber: Anomie.Fri, Oct 16, 5:41 PM