Page MenuHomePhabricator

ORES extension failing to parse scoring response
Closed, ResolvedPublicPRODUCTION ERROR

Description

Notice: Undefined index: enwiki in /srv/mediawiki/php-1.31.0-wmf.5/extensions/ORES/includes/Cache.php on line 52
Warning: Invalid argument supplied for foreach() in /srv/mediawiki/php-1.31.0-wmf.5/extensions/ORES/includes/Cache.php on line 56

My first guess is that we're requesting from the ORES v1 endpoint, and trying to parse it as a v3 response. @Ladsgroup may have already patched this?

Event Timeline

I have a feeling that the patch is not deployed yet.

My guess must have been wrong. Your patches for T178962 are deployed, and we should no longer be hitting the v1 API from the extension. Now I'm thinking these might be error responses?

Halfak triaged this task as High priority.Nov 1 2017, 4:55 PM
Halfak moved this task from Unsorted to Maintenance/cleanup on the Machine-Learning-Team board.
demon raised the priority of this task from High to Unbreak Now!.Nov 1 2017, 6:15 PM
demon subscribed.

This is really really spammy in production, raising to UBN!

Here are the MW loglines when this happens:

Expectation (masterConns <= 0) by MediaWiki::restInPeace not met (actual: 1):
[connect to 10.64.16.77 (enwiki)]
#0 /srv/mediawiki/php-1.31.0-wmf.5/includes/libs/rdbms/TransactionProfiler.php(164): Wikimedia\Rdbms\TransactionProfiler->reportExpectationViolated()
#1 /srv/mediawiki/php-1.31.0-wmf.5/includes/libs/rdbms/loadbalancer/LoadBalancer.php(678): Wikimedia\Rdbms\TransactionProfiler->recordConnection()
#2 /srv/mediawiki/php-1.31.0-wmf.5/includes/GlobalFunctions.php(2865): Wikimedia\Rdbms\LoadBalancer->getConnection()
#3 /srv/mediawiki/php-1.31.0-wmf.5/extensions/ORES/includes/Cache.php(58): wfGetDB()
#4 /srv/mediawiki/php-1.31.0-wmf.5/extensions/ORES/includes/Hooks/ApiHooksHandler.php(323): ORES\Cache->storeScores()
#5 /srv/mediawiki/php-1.31.0-wmf.5/includes/deferred/MWCallableUpdate.php(30): Closure$ORES\Hooks\ApiHooksHandler::loadScoresForRevisions#2()
#6 /srv/mediawiki/php-1.31.0-wmf.5/includes/deferred/DeferredUpdates.php(259): MWCallableUpdate->doUpdate()
#7 /srv/mediawiki/php-1.31.0-wmf.5/includes/deferred/DeferredUpdates.php(210): DeferredUpdates::runUpdate()
#8 /srv/mediawiki/php-1.31.0-wmf.5/includes/deferred/DeferredUpdates.php(131): DeferredUpdates::execute()
#9 /srv/mediawiki/php-1.31.0-wmf.5/includes/MediaWiki.php(897): DeferredUpdates::doUpdates()
#10 /srv/mediawiki/php-1.31.0-wmf.5/includes/MediaWiki.php(719): MediaWiki->restInPeace()
#11 (): Closure$MediaWiki::doPostOutputShutdown()
#12 {main}

[Wed Nov  1 18:19:41 2017] [hphp] [35544:7f69877ff700:399745:000002] [] \nWarning: Invalid argument supplied for foreach() in /srv/mediawiki/php-1.31.0-wmf.5/extensions/ORES/includes/Cache.php on line 56

Warning: Invalid argument supplied for foreach() in /srv/mediawiki/php-1.31.0-wmf.5/extensions/ORES/includes/Cache.php on line 56

[Wed Nov  1 18:19:41 2017] [hphp] [35544:7f69877ff700:399745:000001] [] \nNotice: Undefined index: enwiki in /srv/mediawiki/php-1.31.0-wmf.5/extensions/ORES/includes/Cache.php on line 52

[WfoQOgpAMDoAAHrMGg0AAAAE] /w/api.php?action=query&format=json&prop=revisions&rvlimit=29&rvprop=ids|user|userid|timestamp|size|oresscores|flags&titles=Neural%20tube   ErrorException from line 56 of /srv/mediawiki/php-1.31.0-wmf.5/extensions/ORES/includes/Cache.php: PHP Warning: Invalid argument supplied for foreach()

Notice: Undefined index: enwiki in /srv/mediawiki/php-1.31.0-wmf.5/extensions/ORES/includes/Cache.php on line 52

[WfoQOgpAMDoAAHrMGg0AAAAE] /w/api.php?action=query&format=json&prop=revisions&rvlimit=29&rvprop=ids|user|userid|timestamp|size|oresscores|flags&titles=Neural%20tube   ErrorException from line 52 of /srv/mediawiki/php-1.31.0-wmf.5/extensions/ORES/includes/Cache.php: PHP Notice: Undefined index: enwiki

Change 387847 had a related patch set uploaded (by Awight; owner: Awight):
[mediawiki/extensions/ORES@master] Always assume a v3 API response.

https://gerrit.wikimedia.org/r/387847

Change 387847 merged by jenkins-bot:
[mediawiki/extensions/ORES@master] Always assume a v3 API response.

https://gerrit.wikimedia.org/r/387847

Change 387851 had a related patch set uploaded (by Awight; owner: Awight):
[mediawiki/extensions/ORES@wmf/1.31.0-wmf.5] Always assume a v3 API response.

https://gerrit.wikimedia.org/r/387851

Change 387852 had a related patch set uploaded (by Awight; owner: Awight):
[mediawiki/extensions/ORES@wmf/1.31.0-wmf.6] Always assume a v3 API response.

https://gerrit.wikimedia.org/r/387852

Change 387851 merged by jenkins-bot:
[mediawiki/extensions/ORES@wmf/1.31.0-wmf.5] Always assume a v3 API response.

https://gerrit.wikimedia.org/r/387851

Change 387852 merged by jenkins-bot:
[mediawiki/extensions/ORES@wmf/1.31.0-wmf.6] Always assume a v3 API response.

https://gerrit.wikimedia.org/r/387852

Mentioned in SAL (#wikimedia-operations) [2017-11-01T19:28:30Z] <awight@tin> Synchronized php-1.31.0-wmf.6/extensions/ORES: Fix for API=oresscores, T179430 (duration: 00m 52s)

Mentioned in SAL (#wikimedia-operations) [2017-11-01T19:31:56Z] <awight@tin> Synchronized php-1.31.0-wmf.5/extensions/ORES: Fix for API=oresscores, T179430 (duration: 00m 50s)

I deployed a patch that I thought would have fixed the issue.

mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:09 PM