Replicate RC/WL failures in Beta
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Halfak
	Nov 22 2017, 4:39 PM

Description

Version of ORES prod deployed: 5084251

Details

	Subject	Repo	Branch	Lines +/-
	labs: Enable goodfaith in ruwiki	operations/mediawiki-config	master	+7 -0
	labs: Enable ORES extension in ruwiki	operations/mediawiki-config	master	+3 -1

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		awight	T181010 [Spike] Write reports about why Ext:ORES is helping cause server 500s and write tasks to fix
		Resolved		Halfak	T181168 Replicate RC/WL failures in Beta

Event Timeline

Halfak created this task.Nov 22 2017, 4:39 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 22 2017, 4:39 PM

Mentioned in SAL (#wikimedia-releng) [2017-11-22T16:58:46Z] <halfak> deploying ores-prod-deploy:5084251 T181168

I deployed https://phabricator.wikimedia.org/source/ores-deploy/ @ 5084251

Problem does not exist in Beta at https://ru.wikipedia.beta.wmflabs.org/wiki/Служебная:Свежие_правки or https://ru.wikipedia.beta.wmflabs.org/wiki/Служебная:Список_наблюдения.

I might need to clear thresholds. I'm running P6366 for "damaging" and "goodfaith" on ruwiki.

Still seems to all be working.

FYI, here's what I ran:

$ mwscript eval.php --wiki ruwiki
> $cache = MediaWiki\MediaWikiServices::getInstance()->getMainWANObjectCache();
> $key = $cache->makeKey( 'ORES', 'threshold_statistics', 'goodfaith', 1 );
> $cache->delete($key);
> $key = $cache->makeKey( 'ORES', 'threshold_statistics', 'damaging', 1 );
> $cache->delete($key);
> ^D

With the help of @awight and @Ladsgroup, I figured out that the deploy failed. The deploy has now succeeded and we're working on ruwiki's configuration so that it will even *try* to use ORES.

Change 392876 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[operations/mediawiki-config@master] labs: Enable ORES extension in ruwiki

https://gerrit.wikimedia.org/r/392876

gerritbot added a project: Patch-For-Review.Nov 22 2017, 5:59 PM

Change 392876 merged by jenkins-bot:
[operations/mediawiki-config@master] labs: Enable ORES extension in ruwiki

https://gerrit.wikimedia.org/r/392876

Confirmed that ORES is showing up in Special:Recentchanges.

Checking on key values:

halfak@deployment-tin:/srv/deployment/ores/deploy$ mwscript eval.php --wiki ruwiki
> $cache = MediaWiki\MediaWikiServices::getInstance()->getMainWANObjectCache();
> $key = $cache->makeKey( 'ORES', 'threshold_statistics', 'goodfaith', 1 );
> $cache->get($key)
> $key = $cache->makeKey( 'ORES', 'threshold_statistics', 'damaging', 1 );
> $cache->get($key)

We noticed that the $wgOresModels['default']['goodfaith'] model is disabled on the beta cluster for all wikis but enwiki. @Ladsgroup is patching InitialiseSettings-labs.php momentarily.

Change 392880 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[operations/mediawiki-config@master] labs: Enable goodfaith in ruwiki

https://gerrit.wikimedia.org/r/392880

Change 392880 merged by jenkins-bot:
[operations/mediawiki-config@master] labs: Enable goodfaith in ruwiki

https://gerrit.wikimedia.org/r/392880

Got it!

From https://ru.wikipedia.beta.wmflabs.org/wiki/%D0%A1%D0%BB%D1%83%D0%B6%D0%B5%D0%B1%D0%BD%D0%B0%D1%8F:%D0%A1%D0%B2%D0%B5%D0%B6%D0%B8%D0%B5_%D0%BF%D1%80%D0%B0%D0%B2%D0%BA%D0%B8?hidebots=1&hidecategorization=1&hideWikibase=1&limit=50&days=7&urlversion=2&uselang=en, I can see:

[WhXRtwpEE4AAAGaglTAAAAAO] /wiki/%D0%A1%D0%BB%D1%83%D0%B6%D0%B5%D0%B1%D0%BD%D0%B0%D1%8F:%D0%A1%D0%B2%D0%B5%D0%B6%D0%B8%D0%B5_%D0%BF%D1%80%D0%B0%D0%B2%D0%BA%D0%B8?hidebots=1&hidecategorization=1&hideWikibase=1&limit=50&days=7&urlversion=2&uselang=en RuntimeException from line 285 of /srv/mediawiki/php-master/extensions/ORES/includes/Stats.php: Unable to parse threshold: {"levelName":"verylikelybad","levelConfig":"maximum recall @ precision >= 0.75","bound":"max","statsData":{"false":{"maximum recall @ precision >= 0.15":{"!f1":0.923,"!precision":0.995,"!recall":0.861,"accuracy":0.86,"f1":0.256,"filter_rate":0.841,"fpr":0.139,"match_rate":0.159,"precision":0.151,"recall":0.842,"threshold":0.252},"maximum recall @ precision >= 0.45":{"!f1":0.985,"!precision":0.977,"!recall":0.993,"accuracy":0.97,"f1":0.269,"filter_rate":0.988,"fpr":0.007,"match_rate":0.012,"precision":0.452,"recall":0.192,"threshold":0.797},"maximum recall @ precision >= 0.75":null},"true":{"maximum recall @ precision >= 0.995":{"!f1":0.254,"!precision":0.149,"!recall":0.854,"accuracy":0.856,"f1":0.921,"filter_rate":0.164,"fpr":0.146,"match_rate":0.836,"precision":0.995,"recall":0.856,"threshold":0.766}}}}

Backtrace:

#0 /srv/mediawiki/php-master/extensions/ORES/includes/Stats.php(249): ORES\Stats->extractBoundValue(string, string, string, array)
#1 /srv/mediawiki/php-master/extensions/ORES/includes/Stats.php(58): ORES\Stats->parseThresholds(array, string)
#2 /srv/mediawiki/php-master/extensions/ORES/includes/Hooks/ChangesListHooksHandler.php(233): ORES\Stats->getThresholds(string)
#3 /srv/mediawiki/php-master/includes/Hooks.php(177): ORES\Hooks\ChangesListHooksHandler::onChangesListSpecialPageStructuredFilters(SpecialRecentChanges)
#4 /srv/mediawiki/php-master/includes/Hooks.php(205): Hooks::callHook(string, array, array, NULL)
#5 /srv/mediawiki/php-master/includes/specialpage/ChangesListSpecialPage.php(882): Hooks::run(string, array)
#6 /srv/mediawiki/php-master/includes/specials/SpecialRecentchanges.php(184): ChangesListSpecialPage->registerFilters()
#7 /srv/mediawiki/php-master/includes/specialpage/ChangesListSpecialPage.php(1023): SpecialRecentChanges->registerFilters()
#8 /srv/mediawiki/php-master/includes/specialpage/ChangesListSpecialPage.php(843): ChangesListSpecialPage->setup(NULL)
#9 /srv/mediawiki/php-master/includes/specialpage/ChangesListSpecialPage.php(621): ChangesListSpecialPage->getOptions()
#10 /srv/mediawiki/php-master/includes/specialpage/ChangesListSpecialPage.php(544): ChangesListSpecialPage->considerActionsForDefaultSavedQuery()
#11 /srv/mediawiki/php-master/includes/specials/SpecialRecentchanges.php(166): ChangesListSpecialPage->execute(NULL)
#12 /srv/mediawiki/php-master/includes/specialpage/SpecialPage.php(522): SpecialRecentChanges->execute(NULL)
#13 /srv/mediawiki/php-master/includes/specialpage/SpecialPageFactory.php(578): SpecialPage->run(NULL)
#14 /srv/mediawiki/php-master/includes/MediaWiki.php(287): SpecialPageFactory::executePath(Title, RequestContext)
#15 /srv/mediawiki/php-master/includes/MediaWiki.php(851): MediaWiki->performRequest()
#16 /srv/mediawiki/php-master/includes/MediaWiki.php(523): MediaWiki->main()
#17 /srv/mediawiki/php-master/index.php(43): MediaWiki->run()
#18 /srv/mediawiki/w/index.php(3): include(string)
#19 {main}

Halfak claimed this task.Nov 22 2017, 7:48 PM

Halfak moved this task from Parked to Completed on the Machine-Learning-Team (Active Tasks) board.

Halfak added a parent task: T181010: [Spike] Write reports about why Ext:ORES is helping cause server 500s and write tasks to fix.