internal_api_error_Exception: [22e05a83] Exception Caught: wfDiff(): popen() failed errors on English Wikipedia
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Assigned To

None

Authored By

	Multichill
	Apr 24 2015, 4:53 PM

Description

Keep getting MediaWiki api exceptions (output from Pywikibot):

Working on Henry Hall Dixon
Got redirected to http://viaf.org/viaf/240853956/
@@ -11 +11 @@

{{Authority control|VIAF=61103002}}

+ {{Authority control|VIAF=240853956}}

Following viaf redirect from 61103002 -> 240853956
ERROR: Detected MediaWiki API exception Exception; raising
<snip>
pywikibot.exceptions.OtherPageSaveError: Edit to page [[Henry Hall Dixon]] failed:
internal_api_error_Exception: [0d4c19b2] Exception Caught: wfDiff(): popen() failed
<class 'pywikibot.exceptions.OtherPageSaveError'>

Following viaf redirect from 14547908 -> 33094492
ERROR: Detected MediaWiki API exception Exception; raising
<snip>
pywikibot.exceptions.OtherPageSaveError: Edit to page [[Angelo Donati]] failed:
internal_api_error_Exception: [22e05a83] Exception Caught: wfDiff(): popen() failed
<class 'pywikibot.exceptions.OtherPageSaveError'>

Related Objects

Mentioned Here: T93597: wfDiff() fails to open resource
T93625: Don't use wfDiff() in Echo

Event Timeline

Multichill created this task.Apr 24 2015, 4:53 PM

Multichill raised the priority of this task from to High.

Multichill updated the task description. (Show Details)

Multichill added a project: MediaWiki-Action-API.

Multichill subscribed.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 24 2015, 4:53 PM

Exception is:

2015-04-24 13:31:54 mw1125 enwiki exception INFO: [0d4c19b2] [no req]   Exception from line 3141 of /srv/mediawiki/php-1.26wmf2/includes/GlobalFunctions.php: wfDiff(): popen() failed
#0 /srv/mediawiki/php-1.26wmf2/extensions/Echo/includes/DiffParser.php(93): wfDiff()
#1 /srv/mediawiki/php-1.26wmf2/extensions/Echo/includes/DiscussionParser.php(576): EchoDiffParser->getChangeSet()
#2 /srv/mediawiki/php-1.26wmf2/extensions/Echo/includes/DiscussionParser.php(265): EchoDiscussionParser::getMachineReadableDiff()
#3 /srv/mediawiki/php-1.26wmf2/extensions/Echo/includes/DiscussionParser.php(18): EchoDiscussionParser::getChangeInterpretationForRevision()
#4 /srv/mediawiki/php-1.26wmf2/extensions/Echo/Hooks.php(446): EchoDiscussionParser::generateEventsForRevision()
#5 /srv/mediawiki/php-1.26wmf2/includes/Hooks.php(209): EchoHooks::onArticleSaved()
#6 /srv/mediawiki/php-1.26wmf2/includes/content/ContentHandler.php(1161): Hooks::run()
#7 /srv/mediawiki/php-1.26wmf2/includes/page/WikiPage.php(1979): ContentHandler::runLegacyHooks()
#8 /srv/mediawiki/php-1.26wmf2/includes/page/Article.php(1994): WikiPage->doEditContent()
#9 /srv/mediawiki/php-1.26wmf2/includes/EditPage.php(1895): Article->__call()
#10 /srv/mediawiki/php-1.26wmf2/includes/EditPage.php(1288): EditPage->internalAttemptSave()
#11 /srv/mediawiki/php-1.26wmf2/includes/api/ApiEditPage.php(414): EditPage->attemptSave()
#12 /srv/mediawiki/php-1.26wmf2/includes/api/ApiMain.php(1043): ApiEditPage->execute()
#13 /srv/mediawiki/php-1.26wmf2/includes/api/ApiMain.php(394): ApiMain->executeAction()
#14 /srv/mediawiki/php-1.26wmf2/includes/api/ApiMain.php(367): ApiMain->executeActionWithErrorHandling()
#15 /srv/mediawiki/php-1.26wmf2/api.php(90): ApiMain->execute()
#16 /srv/mediawiki/w/api.php(3): include()
#17 {main} {"private":false}

Exception from line 3141 of /srv/mediawiki/php-1.26wmf2/includes/GlobalFunctions.php: wfDiff(): popen() failed
#0 /srv/mediawiki/php-1.26wmf2/extensions/Echo/includes/DiffParser.php(93): wfDiff()
#1 /srv/mediawiki/php-1.26wmf2/extensions/Echo/includes/DiscussionParser.php(576): EchoDiffParser->getChangeSet()
--
#5 /srv/mediawiki/php-1.26wmf2/includes/specialpage/SpecialPage.php(384): SpecialWhatLinksHere->execute()
#6 /srv/mediawiki/php-1.26wmf2/includes/specialpage/SpecialPageFactory.php(581): SpecialPage->run()
#7 /srv/mediawiki/php-1.26wmf2/includes/MediaWiki.php(267): SpecialPageFactory::executePath()
#8 /srv/mediawiki/php-1.26wmf2/includes/MediaWiki.php(566): MediaWiki->performRequest()
#9 /srv/mediawiki/php-1.26wmf2/includes/MediaWiki.php(414): MediaWiki->main()
#10 /srv/mediawiki/php-1.26wmf2/index.php(46): MediaWiki->run()
#11 /srv/mediawiki/w/index.php(3): include()

See also T93597 and T93625

Seems to correspond to messages like this in hhvm.log:

Apr 24 17:03:55 mw1125:  #012Warning: popen(/usr/bin/diff -u '/tmp/merge-old-IGG07I' '/tmp/merge-your-jO9Ah4',r): Invalid argument in /srv/mediawiki/php-1.26wmf2/includes/GlobalFunctions.php on line 3137

And another one:

pywikibot.exceptions.OtherPageSaveError: Edit to page [[Liza Featherstone]] failed:
internal_api_error_Exception: [4e41a70e] Exception Caught: wfDiff(): popen() failed
<class 'pywikibot.exceptions.OtherPageSaveError'>

XXN subscribed.Apr 24 2015, 6:51 PM

I saw about 30 of those between 1:27:26 PM GMT and 7:39:21 PM GMT.

The last one after using VisualFileChange:

API request failed (internal_api_error_Exception): [8bb13855] Exception Caught: wfDiff(): popen() failed <i>at Fri, 24 Apr 2015 19:39:21 GMT</i> <u>served by mw1125</u>

Anomie edited projects, added acl*sre-team; removed MediaWiki-Action-API.Apr 25 2015, 11:42 AM

Anomie set Security to None.

Reviewing the logs again, this particular instance of the problem seems like it may have accidentally been fixed at around 20:50 UTC yesterday, my guess would be due to an HHVM restart associated with the deploy of Gerrit change 206440.

For the record, the problem in this instance seems to have been mostly on mw1125 (some instances on mw1146), with the first log message at 13:26:20 and the last at 20:49:32 (UTC) on 2015-04-24.

Krenair added a project: Wikimedia-production-error.Apr 25 2015, 8:17 PM

Krenair subscribed.

@greg: Probably fixed as per anomie's last comment but still something to keep an eye on.

Any further updates on whether this looks fixed or not?

I'm guessing it was some sort of memory or other resource issue with HHVM being unable to create the subprocess for the popen, but that's just a guess.

At any rate, I don't see any log entries in the past few days, so we could probably close this at least until it happens again.

+1 on closing. Haven't seen this for a while.

Umherirrender unsubscribed.May 11 2015, 6:57 PM

Gone from the exception logs so closing.

ArielGlenn closed this task as Resolved.May 12 2015, 2:38 PM

• demon moved this task from Untriaged to Resolved on the Wikimedia-production-error board.Jun 12 2015, 6:34 PM

XXN unsubscribed.Jun 13 2015, 10:08 PM

• mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:11 PM

Restricted Application added a subscriber: Liuxinyu970226. · View Herald TranscriptAug 28 2019, 11:11 PM

Aklapper removed a subscriber: Anomie.Oct 16 2020, 5:41 PM

internal_api_error_Exception: [22e05a83] Exception Caught: wfDiff(): popen() failed errors on English WikipediaClosed, ResolvedPublicPRODUCTION ERRORActions

Description

Related Objects

Event Timeline

internal_api_error_Exception: [22e05a83] Exception Caught: wfDiff(): popen() failed errors on English Wikipedia
Closed, ResolvedPublicPRODUCTION ERROR
Actions