Page MenuHomePhabricator

Replicate RC/WL failures in Beta
Closed, ResolvedPublic

Description

Version of ORES prod deployed: 5084251

Event Timeline

Halfak created this task.Nov 22 2017, 4:39 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 22 2017, 4:39 PM

Mentioned in SAL (#wikimedia-releng) [2017-11-22T16:58:46Z] <halfak> deploying ores-prod-deploy:5084251 T181168

I might need to clear thresholds. I'm running P6366 for "damaging" and "goodfaith" on ruwiki.

Still seems to all be working.

FYI, here's what I ran:

$ mwscript eval.php --wiki ruwiki
> $cache = MediaWiki\MediaWikiServices::getInstance()->getMainWANObjectCache();
> $key = $cache->makeKey( 'ORES', 'threshold_statistics', 'goodfaith', 1 );
> $cache->delete($key);
> $key = $cache->makeKey( 'ORES', 'threshold_statistics', 'damaging', 1 );
> $cache->delete($key);
> ^D

With the help of @awight and @Ladsgroup, I figured out that the deploy failed. The deploy has now succeeded and we're working on ruwiki's configuration so that it will even *try* to use ORES.

Change 392876 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[operations/mediawiki-config@master] labs: Enable ORES extension in ruwiki

https://gerrit.wikimedia.org/r/392876

Change 392876 merged by jenkins-bot:
[operations/mediawiki-config@master] labs: Enable ORES extension in ruwiki

https://gerrit.wikimedia.org/r/392876

Confirmed that ORES is showing up in Special:Recentchanges.

Checking on key values:

halfak@deployment-tin:/srv/deployment/ores/deploy$ mwscript eval.php --wiki ruwiki
> $cache = MediaWiki\MediaWikiServices::getInstance()->getMainWANObjectCache();
> $key = $cache->makeKey( 'ORES', 'threshold_statistics', 'goodfaith', 1 );
> $cache->get($key)
> $key = $cache->makeKey( 'ORES', 'threshold_statistics', 'damaging', 1 );
> $cache->get($key)

We noticed that the $wgOresModels['default']['goodfaith'] model is disabled on the beta cluster for all wikis but enwiki. @Ladsgroup is patching InitialiseSettings-labs.php momentarily.

Change 392880 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[operations/mediawiki-config@master] labs: Enable goodfaith in ruwiki

https://gerrit.wikimedia.org/r/392880

Change 392880 merged by jenkins-bot:
[operations/mediawiki-config@master] labs: Enable goodfaith in ruwiki

https://gerrit.wikimedia.org/r/392880

Got it!

From https://ru.wikipedia.beta.wmflabs.org/wiki/%D0%A1%D0%BB%D1%83%D0%B6%D0%B5%D0%B1%D0%BD%D0%B0%D1%8F:%D0%A1%D0%B2%D0%B5%D0%B6%D0%B8%D0%B5_%D0%BF%D1%80%D0%B0%D0%B2%D0%BA%D0%B8?hidebots=1&hidecategorization=1&hideWikibase=1&limit=50&days=7&urlversion=2&uselang=en, I can see:

[WhXRtwpEE4AAAGaglTAAAAAO] /wiki/%D0%A1%D0%BB%D1%83%D0%B6%D0%B5%D0%B1%D0%BD%D0%B0%D1%8F:%D0%A1%D0%B2%D0%B5%D0%B6%D0%B8%D0%B5_%D0%BF%D1%80%D0%B0%D0%B2%D0%BA%D0%B8?hidebots=1&hidecategorization=1&hideWikibase=1&limit=50&days=7&urlversion=2&uselang=en RuntimeException from line 285 of /srv/mediawiki/php-master/extensions/ORES/includes/Stats.php: Unable to parse threshold: {"levelName":"verylikelybad","levelConfig":"maximum recall @ precision >= 0.75","bound":"max","statsData":{"false":{"maximum recall @ precision >= 0.15":{"!f1":0.923,"!precision":0.995,"!recall":0.861,"accuracy":0.86,"f1":0.256,"filter_rate":0.841,"fpr":0.139,"match_rate":0.159,"precision":0.151,"recall":0.842,"threshold":0.252},"maximum recall @ precision >= 0.45":{"!f1":0.985,"!precision":0.977,"!recall":0.993,"accuracy":0.97,"f1":0.269,"filter_rate":0.988,"fpr":0.007,"match_rate":0.012,"precision":0.452,"recall":0.192,"threshold":0.797},"maximum recall @ precision >= 0.75":null},"true":{"maximum recall @ precision >= 0.995":{"!f1":0.254,"!precision":0.149,"!recall":0.854,"accuracy":0.856,"f1":0.921,"filter_rate":0.164,"fpr":0.146,"match_rate":0.836,"precision":0.995,"recall":0.856,"threshold":0.766}}}}

Backtrace:

#0 /srv/mediawiki/php-master/extensions/ORES/includes/Stats.php(249): ORES\Stats->extractBoundValue(string, string, string, array)
#1 /srv/mediawiki/php-master/extensions/ORES/includes/Stats.php(58): ORES\Stats->parseThresholds(array, string)
#2 /srv/mediawiki/php-master/extensions/ORES/includes/Hooks/ChangesListHooksHandler.php(233): ORES\Stats->getThresholds(string)
#3 /srv/mediawiki/php-master/includes/Hooks.php(177): ORES\Hooks\ChangesListHooksHandler::onChangesListSpecialPageStructuredFilters(SpecialRecentChanges)
#4 /srv/mediawiki/php-master/includes/Hooks.php(205): Hooks::callHook(string, array, array, NULL)
#5 /srv/mediawiki/php-master/includes/specialpage/ChangesListSpecialPage.php(882): Hooks::run(string, array)
#6 /srv/mediawiki/php-master/includes/specials/SpecialRecentchanges.php(184): ChangesListSpecialPage->registerFilters()
#7 /srv/mediawiki/php-master/includes/specialpage/ChangesListSpecialPage.php(1023): SpecialRecentChanges->registerFilters()
#8 /srv/mediawiki/php-master/includes/specialpage/ChangesListSpecialPage.php(843): ChangesListSpecialPage->setup(NULL)
#9 /srv/mediawiki/php-master/includes/specialpage/ChangesListSpecialPage.php(621): ChangesListSpecialPage->getOptions()
#10 /srv/mediawiki/php-master/includes/specialpage/ChangesListSpecialPage.php(544): ChangesListSpecialPage->considerActionsForDefaultSavedQuery()
#11 /srv/mediawiki/php-master/includes/specials/SpecialRecentchanges.php(166): ChangesListSpecialPage->execute(NULL)
#12 /srv/mediawiki/php-master/includes/specialpage/SpecialPage.php(522): SpecialRecentChanges->execute(NULL)
#13 /srv/mediawiki/php-master/includes/specialpage/SpecialPageFactory.php(578): SpecialPage->run(NULL)
#14 /srv/mediawiki/php-master/includes/MediaWiki.php(287): SpecialPageFactory::executePath(Title, RequestContext)
#15 /srv/mediawiki/php-master/includes/MediaWiki.php(851): MediaWiki->performRequest()
#16 /srv/mediawiki/php-master/includes/MediaWiki.php(523): MediaWiki->main()
#17 /srv/mediawiki/php-master/index.php(43): MediaWiki->run()
#18 /srv/mediawiki/w/index.php(3): include(string)
#19 {main}
MBH added a subscriber: MBH.Nov 26 2017, 3:50 PM
Halfak closed this task as Resolved.Jan 30 2018, 8:31 PM