MWException: Default '"soft"' is invalid for preference oresDamagingPref of most users
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Assigned To

None

Authored By

	Krinkle
	Aug 30 2023, 10:40 PM

Description

Error

mwversion: 1.41.0-wmf.23
reqId: d21f2e3d-ab2c-457b-bfa3-f05d5a1bcc3d
Find reqId in Logstash

normalized_message

[{reqId}] {exception_url}   MWException: Default '"soft"' is invalid for preference oresDamagingPref of user ***

exception.trace

from /srv/mediawiki/php-1.41.0-wmf.23/includes/preferences/DefaultPreferencesFactory.php(355)
#0 /srv/mediawiki/php-1.41.0-wmf.23/includes/preferences/DefaultPreferencesFactory.php(264): MediaWiki\Preferences\DefaultPreferencesFactory->loadPreferenceValues(User, DerivativeContext, array)
#1 /srv/mediawiki/php-1.41.0-wmf.23/extensions/GlobalPreferences/includes/GlobalPreferencesFactory.php(117): MediaWiki\Preferences\DefaultPreferencesFactory->getFormDescriptor(User, DerivativeContext)
#2 /srv/mediawiki/php-1.41.0-wmf.23/includes/api/ApiOptions.php(214): GlobalPreferences\GlobalPreferencesFactory->getFormDescriptor(User, DerivativeContext)
#3 /srv/mediawiki/php-1.41.0-wmf.23/includes/api/ApiOptions.php(116): ApiOptions->getPreferences()
#4 /srv/mediawiki/php-1.41.0-wmf.23/includes/api/ApiMain.php(1924): ApiOptions->execute()
#5 /srv/mediawiki/php-1.41.0-wmf.23/includes/api/ApiMain.php(901): ApiMain->executeAction()
#6 /srv/mediawiki/php-1.41.0-wmf.23/includes/api/ApiMain.php(872): ApiMain->executeActionWithErrorHandling()
#7 /srv/mediawiki/php-1.41.0-wmf.23/api.php(95): ApiMain->execute()
#8 /srv/mediawiki/php-1.41.0-wmf.23/api.php(48): wfApiMain()
#9 /srv/mediawiki/w/api.php(3): require(string)
#10 {main}

Impact

This is fatally breaking a range of user actions and API entrypoints, resulting in HTTP 500 Internal Server Error, including:

opening Special:Preferences.
interactions with Special:ContentTranslation.
making Action API requests.

Notes

Search "oresDamagingPref" on https://logstash.wikimedia.org/app/dashboards#/view/mediawiki-errors for an overview.

Screenshot 2023-08-30 at 23.33.24.png (1×2 px, 363 KB)

Seems to have started yesterday.

Details

Request URL: https://ar.wikipedia.org/w/api.php, https://ar.wikipedia.org/wiki/Special:Preferences

	Subject	Repo	Branch	Lines +/-
	ores-extension: fix arwiki likelybad threshold	operations/mediawiki-config	master	+2 -2

Customize query in gerrit

Related Objects

Duplicates Merged Here: T345320: Fatal Exception (MWException) in arwiki when opening prefrences

Event Timeline

Krinkle created this task.Aug 30 2023, 10:40 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 30 2023, 10:40 PM

@isarantopoulos I think the threshold for arwiki is wrong

Ammarpad triaged this task as Unbreak Now! priority.Aug 31 2023, 7:34 AM

Ammarpad merged a task: T345320: Fatal Exception (MWException) in arwiki when opening prefrences.

Ammarpad added subscribers: hubaishan, alaa.

I am investigating and will report back in a bit

This Bug is not only for user نعمان حمداوي, it is for most users, and only appears today

soft is likelybad. And it's disabled in arwiki (and it's default I think). I think that might be breaking it.

hubaishan renamed this task from MWException: Default '"soft"' is invalid for preference oresDamagingPref of user نعمان حمداوي to MWException: Default '"soft"' is invalid for preference oresDamagingPref of most users.Aug 31 2023, 7:53 AM

hubaishan changed Request URL from https://ar.wikipedia.org/w/api.php to https://ar.wikipedia.org/w/api.php, https://ar.wikipedia.org/wiki/Special:Preferences.

You are right, it is the one that was returning null and we set it to false

this is the request that returns null.

I suggest one of the following

Revert likelybad to 'likelybad' => [ 'min' => 'maximum recall @ precision >= 0.45', 'max' => 1 ], and then check the thresholds that existed in production and use them as the current request doesn't seem to return anything meaningful.
Change the threshold to something really close to the existing one that returns a proper response: https://ores.wikimedia.org/v3/scores/arwiki?models=damaging&model_info=statistics.thresholds.true."''maximum recall @ precision >= 0.43''". using this we would put likelybad in the space 0.496 - 1.

I suggest to go with the latter, it will resolve the issue with no impact. @Ladsgroup wdyt?

yeah, let's go with the latter.

Change 953962 had a related patch set uploaded (by Ilias Sarantopoulos; author: Ilias Sarantopoulos):

[operations/mediawiki-config@master] ores-extension: fix arwiki likelybad threshold

https://gerrit.wikimedia.org/r/953962

gerritbot added a project: Patch-For-Review.Aug 31 2023, 8:16 AM

Change 953962 merged by jenkins-bot:

[operations/mediawiki-config@master] ores-extension: fix arwiki likelybad threshold

https://gerrit.wikimedia.org/r/953962

Mentioned in SAL (#wikimedia-operations) [2023-08-31T08:28:37Z] <ladsgroup@deploy1002> Started scap: Backport for [[gerrit:953962|ores-extension: fix arwiki likelybad threshold (T345305)]]

Maintenance_bot removed a project: Patch-For-Review.Aug 31 2023, 8:30 AM

Mentioned in SAL (#wikimedia-operations) [2023-08-31T08:36:12Z] <ladsgroup@deploy1002> ladsgroup and isaranto: Backport for [[gerrit:953962|ores-extension: fix arwiki likelybad threshold (T345305)]] synced to the testservers mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option)

it is OK at test server

yeah, I'm deploying but k8s cluster is being rebooted and thus deployments are quite slow right now.

Mentioned in SAL (#wikimedia-operations) [2023-08-31T09:37:34Z] <ladsgroup@deploy1002> Finished scap: Backport for [[gerrit:953962|ores-extension: fix arwiki likelybad threshold (T345305)]] (duration: 68m 57s)

Ladsgroup closed this task as Resolved.Aug 31 2023, 9:52 AM

isarantopoulos lowered the priority of this task from Unbreak Now! to Needs Triage.Nov 20 2023, 11:36 AM

isarantopoulos moved this task from Unsorted to 2023-2024 Q3 Done on the Machine-Learning-Team board.

	F37643515: Screenshot 2023-08-30 at 23.33.24.png
	Aug 30 2023, 10:40 PM

MWException: Default '"soft"' is invalid for preference oresDamagingPref of most usersClosed, ResolvedPublicPRODUCTION ERRORActions