FetchScoreJob is trying to update scores for nonexistent models
Closed, ResolvedPublic
Actions

Assigned To

None

Authored By

	awight
	Oct 11 2017, 5:11 PM

Description

I haven't verified yet whether this affects production, but Beta has a long list of these errors.

2017-10-10 06:51:25 [WdxRygpEFhUAAG6BZfwAAAAT] deployment-videoscaler01 wikidatawiki 1.31.0-alpha exception ERROR: [WdxRygpEFhUAAG6BZfwAAAAT] /rpc/RunJobs.php?wiki=wikidatawiki&type=ORESFetchScoreJob&maxtime=30&maxmem=300M   RuntimeException from line 172 of /srv/mediawiki/php-master/extensions/ORES/includes/Cache.php: No model available for [goodfaith] {"exception_id":"WdxRygpEFhUAAG6BZfwAAAAT","exception_url":"/rpc/RunJobs.php?wiki=wikidatawiki&type=ORESFetchScoreJob&maxtime=30&maxmem=300M","caught_by":"mwe_handler"} 
[Exception RuntimeException] (/srv/mediawiki/php-master/extensions/ORES/includes/Cache.php:172) No model available for [goodfaith]
  #0 /srv/mediawiki/php-master/extensions/ORES/includes/Cache.php(207): ORES\Cache->getModelId(string)
  #1 /srv/mediawiki/php-master/extensions/ORES/includes/Cache.php(49): ORES\Cache->processRevision(array, integer, array)
  #2 /srv/mediawiki/php-master/extensions/ORES/includes/FetchScoreJob.php(67): ORES\Cache->storeScores(array)
  #3 /srv/mediawiki/php-master/includes/jobqueue/JobRunner.php(295): ORES\FetchScoreJob->run()
  #4 /srv/mediawiki/php-master/includes/jobqueue/JobRunner.php(193): JobRunner->executeJob(ORES\FetchScoreJob, Wikimedia\Rdbms\LBFactoryMulti, BufferingStatsdDataFactory, integer)
  #5 /srv/mediawiki/rpc/RunJobs.php(47): JobRunner->run(array)
  #6 {main}

2017-10-10 06:52:07 [WdxgBQpEFhUAAFV3Cl0AAAAH] deployment-jobrunner02 wikidatawiki 1.31.0-alpha exception ERROR: [WdxgBQpEFhUAAFV3Cl0AAAAH] /rpc/RunJobs.php?wiki=wikidatawiki&type=ORESFetchScoreJob&maxtime=60&maxmem=300M   RuntimeException from line 172 of /srv/mediawiki/php-master/extensions/ORES/includes/Cache.php: No model available for [goodfaith] {"exception_id":"WdxgBQpEFhUAAFV3Cl0AAAAH","exception_url":"/rpc/RunJobs.php?wiki=wikidatawiki&type=ORESFetchScoreJob&maxtime=60&maxmem=300M","caught_by":"mwe_handler"} 
[Exception RuntimeException] (/srv/mediawiki/php-master/extensions/ORES/includes/Cache.php:172) No model available for [goodfaith]
  #0 /srv/mediawiki/php-master/extensions/ORES/includes/Cache.php(207): ORES\Cache->getModelId(string)
  #1 /srv/mediawiki/php-master/extensions/ORES/includes/Cache.php(49): ORES\Cache->processRevision(array, integer, array)
  #2 /srv/mediawiki/php-master/extensions/ORES/includes/FetchScoreJob.php(72): ORES\Cache->storeScores(array)
  #3 /srv/mediawiki/php-master/includes/jobqueue/JobRunner.php(295): ORES\FetchScoreJob->run()
  #4 /srv/mediawiki/php-master/includes/jobqueue/JobRunner.php(193): JobRunner->executeJob(ORES\FetchScoreJob, Wikimedia\Rdbms\LBFactoryMulti, BufferingStatsdDataFactory, integer)
  #5 /srv/mediawiki/rpc/RunJobs.php(47): JobRunner->run(array)
  #6 {main}

Event Timeline

awight created this task.Oct 11 2017, 5:11 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 11 2017, 5:11 PM

Probably unrelated, Beta Redis needs some more configuration?

2017-10-11 06:52:30 [fdf9ae2aefff4a3a78b47ac5] deployment-videoscaler01 wikidatawiki 1.31.0-alpha error WARNING: [fdf9ae2aefff4a3a78b47ac5] /rpc/RunJobs.php?wiki=wikidatawiki&type=ORESFetchScoreJob&maxtime=60&maxmem=300M   ErrorException from line 0 of : PHP Warning: unable to connect to unix:///var/run/nutcracker/redis_eqiad.sock [2]: No such file or directory {"exception_id":"fdf9ae2aefff4a3a78b47ac5","exception_url":"/rpc/RunJobs.php?wiki=wikidatawiki&type=ORESFetchScoreJob&maxtime=60&maxmem=300M","caught_by":"mwe_handler"} 
[Exception ErrorException] (:0) PHP Warning: unable to connect to unix:///var/run/nutcracker/redis_eqiad.sock [2]: No such file or directory
  #0 [internal function]: MWExceptionHandler::handleError(integer, string, string, integer, array, array)
  #1 [internal function]: Redis->doConnect(string, integer, integer, string, integer, boolean)
  #2 /srv/mediawiki/php-master/includes/libs/redis/RedisConnectionPool.php(238): Redis->connect(string, integer, integer)
  #3 /srv/mediawiki/php-master/includes/libs/objectcache/RedisBagOStuff.php(354): RedisConnectionPool->getConnection(string, Monolog\Logger)
  #4 /srv/mediawiki/php-master/includes/libs/objectcache/RedisBagOStuff.php(91): RedisBagOStuff->getConnection(string)
  #5 /srv/mediawiki/php-master/includes/libs/objectcache/CachedBagOStuff.php(56): RedisBagOStuff->doGet(string, integer)
  #6 /srv/mediawiki/php-master/includes/libs/objectcache/BagOStuff.php(185): CachedBagOStuff->doGet(string, integer)
  #7 /srv/mediawiki/php-master/includes/session/SessionManager.php(938): BagOStuff->get(string)
  #8 /srv/mediawiki/php-master/includes/session/SessionInfo.php(150): MediaWiki\Session\SessionManager->generateSessionId()
  #9 /srv/mediawiki/php-master/includes/session/SessionProvider.php(176): MediaWiki\Session\SessionInfo->__construct(integer, array)
  #10 /srv/mediawiki/php-master/includes/session/SessionManager.php(269): MediaWiki\Session\SessionProvider->newSessionInfo(NULL)
  #11 /srv/mediawiki/php-master/includes/session/SessionManager.php(243): MediaWiki\Session\SessionManager->getEmptySessionInternal(WebRequest)
  #12 /srv/mediawiki/php-master/includes/session/SessionManager.php(193): MediaWiki\Session\SessionManager->getEmptySession(WebRequest)
  #13 /srv/mediawiki/php-master/includes/WebRequest.php(735): MediaWiki\Session\SessionManager->getSessionForRequest(WebRequest)
  #14 /srv/mediawiki/php-master/includes/session/SessionManager.php(129): WebRequest->getSession()
  #15 /srv/mediawiki/php-master/includes/Setup.php(762): MediaWiki\Session\SessionManager::getGlobalSession()
  #16 /srv/mediawiki/php-master/includes/WebStart.php(114): include(string)
  #17 /srv/mediawiki/rpc/RunJobs.php(31): include(string)
  #18 {main}

@Ladsgroup said he knows about a related bug. Maybe share it here. :)

Halfak triaged this task as Medium priority.Oct 26 2017, 9:52 PM

Halfak moved this task from Unsorted to Maintenance/cleanup on the Machine-Learning-Team board.

Halfak added a project: Beta-Cluster-Infrastructure.

awight raised the priority of this task from Medium to Needs Triage.Oct 26 2017, 9:53 PM

awight added a project: Beta-Cluster-reproducible.

I think I know what's going on here, these are being overwritten by prod settings. I will get this handled ASAP.

This has been resolved very long time ago.

Ladsgroup closed this task as Resolved.Apr 14 2018, 5:19 PM

FetchScoreJob is trying to update scores for nonexistent modelsClosed, ResolvedPublicActions

Description

Event Timeline

FetchScoreJob is trying to update scores for nonexistent models
Closed, ResolvedPublic
Actions