Page MenuHomePhabricator
Create Task
Maniphest T204961

ORES requests for wikidatawiki models=damaging end up with HTTP request timed out
Closed, ResolvedPublic
Actions

Description

Since promoting group2 to 1.32.0-wmf.22

message.
[{exception_id}] {exception_url} RuntimeException from line 95 of /srv/mediawiki/php-1.32.0-wmf.22/extensions/ORES/includes/ORESService.php: Failed to make ORES request to [http://ores.discovery.wmnet:8081/v3/scores/wikidatawiki/?models=damaging%7Cgoodf

With HTTP request timed out.

stacktrace
#0 /srv/mediawiki/php-1.32.0-wmf.22/extensions/ORES/includes/ScoreFetcher.php(55): ORES\ORESService->request(array, NULL)
#1 /srv/mediawiki/php-1.32.0-wmf.22/extensions/ORES/includes/Hooks/ApiHooksHandler.php(273): ORES\ScoreFetcher->getScores(array)
#2 /srv/mediawiki/php-1.32.0-wmf.22/extensions/ORES/includes/Hooks/ApiHooksHandler.php(227): ORES\Hooks\ApiHooksHandler::loadScoresForRevisions(array)
#3 /srv/mediawiki/php-1.32.0-wmf.22/includes/Hooks.php(174): ORES\Hooks\ApiHooksHandler::onApiQueryBaseAfterQuery(ApiQueryRecentChanges, Wikimedia\Rdbms\ResultWrapper, array)
#4 /srv/mediawiki/php-1.32.0-wmf.22/includes/Hooks.php(202): Hooks::callHook(string, array, array, NULL)
#5 /srv/mediawiki/php-1.32.0-wmf.22/includes/api/ApiQueryBase.php(381): Hooks::run(string, array)
#6 /srv/mediawiki/php-1.32.0-wmf.22/includes/api/ApiQueryRecentChanges.php(426): ApiQueryBase->select(string, array, array)
#7 /srv/mediawiki/php-1.32.0-wmf.22/includes/api/ApiQueryRecentChanges.php(133): ApiQueryRecentChanges->run()
#8 /srv/mediawiki/php-1.32.0-wmf.22/includes/api/ApiQuery.php(249): ApiQueryRecentChanges->execute()
#9 /srv/mediawiki/php-1.32.0-wmf.22/includes/api/ApiMain.php(1587): ApiQuery->execute()
#10 /srv/mediawiki/php-1.32.0-wmf.22/includes/api/ApiMain.php(531): ApiMain->executeAction()
#11 /srv/mediawiki/php-1.32.0-wmf.22/includes/api/ApiMain.php(502): ApiMain->executeActionWithErrorHandling()
#12 /srv/mediawiki/php-1.32.0-wmf.22/api.php(87): ApiMain->execute()
#13 /srv/mediawiki/w/api.php(3): include(string)
#14 {main}

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Catch and gracefully handle when service is not responding properly to ApiHooksHandlermediawiki/extensions/ORESmaster+7 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

hashar created this task.Sep 20 2018, 2:53 PM
Restricted Application added a project: Machine-Learning-Team. · View Herald TranscriptSep 20 2018, 2:53 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
hashar added a parent task: T191068: 1.32.0-wmf.22 deployment blockers.Sep 20 2018, 2:54 PM

Though if I try a reported URL manually, it seems to work. Tried on deploy1001, deploy2001 and mw2219

hashar added a comment.Sep 20 2018, 3:03 PM

Based on Graphana, seems some ORES server got overloaded and some requests did time out. That was a short break though and apparently everything works fine now. To be investigated though.

Ladsgroup subscribed.Sep 20 2018, 3:13 PM

Yes, basically for each deployment we get an overload error spike: https://grafana.wikimedia.org/dashboard/db/ores?refresh=1m&panelId=9&fullscreen&orgId=1&from=now-2d&to=now-1m
The reason is that all restarts basically happen at the same time, is there a way around that in scap?

Ladsgroup added a comment.Sep 20 2018, 3:19 PM

I don't think it should be a blocker to the train as it's due to changes happening to the service (T160692: Use poolcounter to limit number of connections to ores uwsgi) but it should be pretty high priority (and I will get it fixed)

Ladsgroup mentioned this in T204753: Failed executing job: ORESFetchScoreJob.Sep 20 2018, 4:43 PM
Ladsgroup claimed this task.Sep 20 2018, 8:00 PM
Ladsgroup edited projects, added Machine-Learning-Team (Active Tasks); removed Machine-Learning-Team.
Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptSep 20 2018, 8:00 PM
hashar mentioned this in T191068: 1.32.0-wmf.22 deployment blockers.Sep 20 2018, 9:15 PM
hashar added a comment.Sep 21 2018, 10:53 AM

That is not a blocker to the train per see. I have filled it as a subtask because the train triggers the issue. Seems that is transient and self resolve after some short amount of time.

So surely, we should get it fixed, but that is not a blocker per see :]

greg triaged this task as High priority.Sep 24 2018, 4:18 PM
greg subscribed.
In T204961#4602109, @Ladsgroup wrote:

I don't think it should be a blocker to the train as it's due to changes happening to the service (T160692: Use poolcounter to limit number of connections to ores uwsgi) but it should be pretty high priority (and I will get it fixed)

ETA?

Ladsgroup added a comment.Sep 24 2018, 8:10 PM

There are three ways to handle this, I'm planning to do all to make sure it won't happen again. Each of them might take one day or two to implement and deploy.

hashar added a project: Wikimedia-Incident.Sep 25 2018, 10:23 AM
hashar moved this task from Active investigation to Follow-up prevention on the Wikimedia-Incident board.
gerritbot subscribed.Sep 25 2018, 5:48 PM

Change 462779 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[mediawiki/extensions/ORES@master] Catch and gracefully handle when service is not responding properly to ApiHooksHandler

https://gerrit.wikimedia.org/r/462779

gerritbot added a project: Patch-For-Review.Sep 25 2018, 5:48 PM
Ladsgroup moved this task from Parked to Review on the Machine-Learning-Team (Active Tasks) board.Sep 26 2018, 11:34 AM
gerritbot added a comment.Oct 1 2018, 3:55 PM

Change 462779 merged by jenkins-bot:
[mediawiki/extensions/ORES@master] Catch and gracefully handle when service is not responding properly to ApiHooksHandler

https://gerrit.wikimedia.org/r/462779

Ladsgroup moved this task from Review to Pending deployment on the Machine-Learning-Team (Active Tasks) board.Oct 1 2018, 3:57 PM
ReleaseTaggerBot added a project: MW-1.32-notes (WMF-deploy-2018-10-02 (1.32.0-wmf.24)).Oct 1 2018, 4:01 PM
Ladsgroup moved this task from Incoming to In progress on the User-Ladsgroup board.Oct 2 2018, 8:19 PM
Ladsgroup moved this task from Pending deployment to Completed on the Machine-Learning-Team (Active Tasks) board.Oct 8 2018, 3:29 PM
Ladsgroup closed this task as Resolved.Nov 1 2018, 8:00 PM
hashar added a comment.Nov 5 2018, 2:00 PM

Thank you @Ladsgroup to have followed on that incident task. Any spam log we can drop makes deployment of MediaWiki easier!

Krinkle edited projects, added Sustainability (Incident Followup); removed Wikimedia-Incident.Apr 28 2020, 9:50 PM
Maintenance_bot removed a project: Patch-For-Review.Apr 28 2020, 10:13 PM
Maintenance_bot moved this task from In progress to Done on the User-Ladsgroup board.Apr 28 2020, 10:15 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits