Page MenuHomePhabricator
Create Task
Maniphest T313973

Exception: Invalid JSON response for page: Espejo
Open, HighPublicPRODUCTION ERROR
Actions

Assigned To
None
Authored By
Sgs
Wed, Jul 27, 6:50 PM
Tags
Referenced Files
F35391705: Screenshot Capture - 2022-08-03 - 10-51-38.png
Wed, Aug 3, 6:35 PM
F35391703: Screenshot Capture - 2022-08-03 - 10-53-22.png
Wed, Aug 3, 6:35 PM
Subscribers
Aklapper
CBogen
• dcausse
EBernhardson
Etonkovidova
hnowlan
kostajh
View All 13 Subscribers

Description

Error
normalized_message
[{reqId}] {exception_url}   Exception: Invalid JSON response for page: Espejo
exception.trace
from /srv/mediawiki/php-1.39.0-wmf.21/extensions/GrowthExperiments/includes/Util.php(181)
#0 /srv/mediawiki/php-1.39.0-wmf.21/extensions/GrowthExperiments/includes/HomepageHooks.php(429): GrowthExperiments\Util::logStatus(StatusValue)
#1 /srv/mediawiki/php-1.39.0-wmf.21/includes/HookContainer/HookContainer.php(160): GrowthExperiments\HomepageHooks->onBeforePageDisplay(OutputPage, MediaWiki\Minerva\Skins\SkinMinerva)
#2 /srv/mediawiki/php-1.39.0-wmf.21/includes/HookContainer/HookRunner.php(942): MediaWiki\HookContainer\HookContainer->run(string, array, array)
#3 /srv/mediawiki/php-1.39.0-wmf.21/includes/OutputPage.php(2838): MediaWiki\HookContainer\HookRunner->onBeforePageDisplay(OutputPage, MediaWiki\Minerva\Skins\SkinMinerva)
#4 /srv/mediawiki/php-1.39.0-wmf.21/includes/MediaWiki.php(925): OutputPage->output(boolean)
#5 /srv/mediawiki/php-1.39.0-wmf.21/includes/MediaWiki.php(938): MediaWiki::{closure}()
#6 /srv/mediawiki/php-1.39.0-wmf.21/includes/MediaWiki.php(568): MediaWiki->main()
#7 /srv/mediawiki/php-1.39.0-wmf.21/index.php(50): MediaWiki->run()
#8 /srv/mediawiki/php-1.39.0-wmf.21/index.php(46): wfIndexMain()
#9 /srv/mediawiki/w/index.php(3): require(string)
#10 {main}
Impact
Notes

Details

Request URL
https://tr.wikipedia.org/w/index.php?title=*&getasktype=*&gesuggestededit=*&geclickid=*&genewcomertasktoken=*&veaction=*&section=*
ProjectBranchLines +/-Subject
mediawiki/extensions/GrowthExperimentsmaster+5 -4Parametrize title when logging Add Image API errors
mediawiki/extensions/GrowthExperimentswmf/1.39.0-wmf.22+6 -2ServiceImageRecommendationProvider: Add extra logging when no JSON response received
mediawiki/extensions/GrowthExperimentsmaster+6 -2ServiceImageRecommendationProvider: Add extra logging when no JSON response received
Customize query in gerrit

Related Objects

Event Timeline

Sgs created this task.Wed, Jul 27, 6:50 PM
Restricted Application added subscribers: Masumrezarock100, Aklapper. · View Herald TranscriptWed, Jul 27, 6:50 PM
kostajh edited projects, added Growth-Team (Current Sprint); removed Growth-Team.Thu, Jul 28, 1:56 PM
kostajh added a subscriber: kostajh.

Happening for image recommendation tasks, 629 of these in the last 30 days.

Probably the first step is to improve the logging so we can see what the server response is.

kostajh triaged this task as High priority.Thu, Jul 28, 1:56 PM
kostajh moved this task from Incoming to Ready for Development on the Growth-Team (Current Sprint) board.
thcipriani added a subscriber: thcipriani.Thu, Jul 28, 3:42 PM

Notable that these errors seem to have started with 1.39.0-wmf.19—I'm not seeing any errors of this kind prior to that version.

gerritbot added a comment.Fri, Jul 29, 1:31 PM

Change 818461 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/GrowthExperiments@master] ServiceImageRecommendationProvider: Add extra logging when no JSON response received

https://gerrit.wikimedia.org/r/818461

gerritbot added a project: Patch-For-Review.Fri, Jul 29, 1:31 PM
Krinkle moved this task from Untriaged to Jul 2022 on the Wikimedia-production-error board.Fri, Jul 29, 10:56 PM
Etonkovidova added a subscriber: Etonkovidova.Mon, Aug 1, 5:57 PM
In T313973#8111925, @kostajh wrote:

Happening for image recommendation tasks, 629 of these in the last 30 days.

Probably the first step is to improve the logging so we can see what the server response is.

742 errors from July01-Aug01/2022

gerritbot added a comment.Mon, Aug 1, 6:14 PM

Change 818461 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] ServiceImageRecommendationProvider: Add extra logging when no JSON response received

https://gerrit.wikimedia.org/r/818461

Maintenance_bot removed a project: Patch-For-Review.Mon, Aug 1, 6:30 PM
ReleaseTaggerBot added a project: MW-1.39-notes (1.39.0-wmf.23; 2022-08-01).Mon, Aug 1, 7:00 PM
gerritbot added a comment.Mon, Aug 1, 7:39 PM

Change 819075 had a related patch set uploaded (by Urbanecm; author: Kosta Harlan):

[mediawiki/extensions/GrowthExperiments@wmf/1.39.0-wmf.22] ServiceImageRecommendationProvider: Add extra logging when no JSON response received

https://gerrit.wikimedia.org/r/819075

gerritbot added a project: Patch-For-Review.Mon, Aug 1, 7:39 PM
gerritbot added a comment.Wed, Aug 3, 3:31 PM

Change 819075 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@wmf/1.39.0-wmf.22] ServiceImageRecommendationProvider: Add extra logging when no JSON response received

https://gerrit.wikimedia.org/r/819075

Stashbot added a comment.Wed, Aug 3, 3:36 PM

Mentioned in SAL (#wikimedia-operations) [2022-08-03T15:36:22Z] <urbanecm@deploy1002> Synchronized php-1.39.0-wmf.22/extensions/GrowthExperiments/includes/NewcomerTasks/AddImage/ServiceImageRecommendationProvider.php: 4438957e78e0012aff646e52dc16a4fb796cfd6b: ServiceImageRecommendationProvider: Add extra logging when no JSON response received (T313973) (duration: 03m 04s)

Urbanecm_WMF added a subscriber: Urbanecm_WMF.Wed, Aug 3, 3:51 PM
In T313973#8111925, @kostajh wrote:

[...]
Probably the first step is to improve the logging so we can see what the server response is.

I went ahead and backported the logging patch. Logs now should tell us what the server response is. And indeed they do:

2022-08-03 15:48:57 [fa724915-151c-4349-8d12-38d486d451fb] mw1434 ptwiki 1.39.0-wmf.22 GrowthExperiments ERROR: GrowthExperiments\NewcomerTasks\AddImage\ServiceImageRecommendationProvider::get: Unable to decode JSON response for page: WhatsUp_Gold upstream connect error or disconnect/reset before headers. reset reason: connection termination {"response":"upstream connect error or disconnect/reset before headers. reset reason: connection termination"}

Looks like an error on the service's end (or SRE's end).

ReleaseTaggerBot edited projects, added MW-1.39-notes (1.39.0-wmf.22; 2022-07-25); removed MW-1.39-notes (1.39.0-wmf.23; 2022-08-01).Wed, Aug 3, 4:00 PM
Seddon added projects: Discovery-Search (Current work), Structured-Data-Backlog (Current Work).Wed, Aug 3, 4:23 PM
Seddon added subscribers: mfossati, EBernhardson, • dcausse.
Seddon added a subscriber: Tgr.
Maintenance_bot removed a project: Patch-For-Review.Wed, Aug 3, 4:30 PM
CBogen added a project: Platform Engineering.Wed, Aug 3, 4:45 PM
Tgr added a subscriber: hnowlan.Wed, Aug 3, 6:35 PM

Occurrences in the last 30 days:

Screenshot Capture - 2022-08-03 - 10-53-22.png (286×662 px, 20 KB)

Image recommendation stats in the last 90 days:
Screenshot Capture - 2022-08-03 - 10-51-38.png (807×1 px, 202 KB)

so this doesn't seem to have a visible impact on KPIs (but probably does result in some extra errors shown to users).

According to @hnowlan the error message is from Envoy - the image suggestions API is unavailable.

CBogen removed a project: Structured-Data-Backlog (Current Work).Thu, Aug 4, 3:52 PM
CBogen added a subscriber: CBogen.

Removing the Structured Data tag since this seems to be on the API Platform/Growth side only.

CBogen moved this task from To Do to Growth: Add an Image on the Image-Suggestions board.Thu, Aug 4, 6:32 PM
TJones edited projects, added Discovery-Search; removed Discovery-Search (Current work).Mon, Aug 8, 3:29 PM
Gehel removed a project: Discovery-Search.Mon, Aug 8, 3:35 PM
gerritbot added a comment.Thu, Aug 11, 2:20 AM

Change 822202 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[mediawiki/extensions/GrowthExperiments@master] Parametrize title when logging Add Image API errors

https://gerrit.wikimedia.org/r/822202

gerritbot added a project: Patch-For-Review.Thu, Aug 11, 2:20 AM
gerritbot added a comment.Thu, Aug 11, 12:28 PM

Change 822202 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] Parametrize title when logging Add Image API errors

https://gerrit.wikimedia.org/r/822202

Maintenance_bot removed a project: Patch-For-Review.Thu, Aug 11, 12:29 PM
ReleaseTaggerBot edited projects, added MW-1.39-notes (1.39.0-wmf.25; 2022-08-15); removed MW-1.39-notes (1.39.0-wmf.22; 2022-07-25).Thu, Aug 11, 1:00 PM
Sgs mentioned this in T315013: Exception: GrowthExperiments\NewcomerTasks\AddImage\ServiceImageRecommendationProvider::get: Unable to decode JSON response for page: *.Fri, Aug 12, 4:33 PM
WDoranWMF added a project: API Platform.Mon, Aug 15, 4:36 PM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL