Page MenuHomePhabricator

Exception: Invalid JSON response for page: Espejo
Open, HighPublicPRODUCTION ERROR

Description

Error
normalized_message
[{reqId}] {exception_url}   Exception: Invalid JSON response for page: Espejo
exception.trace
from /srv/mediawiki/php-1.39.0-wmf.21/extensions/GrowthExperiments/includes/Util.php(181)
#0 /srv/mediawiki/php-1.39.0-wmf.21/extensions/GrowthExperiments/includes/HomepageHooks.php(429): GrowthExperiments\Util::logStatus(StatusValue)
#1 /srv/mediawiki/php-1.39.0-wmf.21/includes/HookContainer/HookContainer.php(160): GrowthExperiments\HomepageHooks->onBeforePageDisplay(OutputPage, MediaWiki\Minerva\Skins\SkinMinerva)
#2 /srv/mediawiki/php-1.39.0-wmf.21/includes/HookContainer/HookRunner.php(942): MediaWiki\HookContainer\HookContainer->run(string, array, array)
#3 /srv/mediawiki/php-1.39.0-wmf.21/includes/OutputPage.php(2838): MediaWiki\HookContainer\HookRunner->onBeforePageDisplay(OutputPage, MediaWiki\Minerva\Skins\SkinMinerva)
#4 /srv/mediawiki/php-1.39.0-wmf.21/includes/MediaWiki.php(925): OutputPage->output(boolean)
#5 /srv/mediawiki/php-1.39.0-wmf.21/includes/MediaWiki.php(938): MediaWiki::{closure}()
#6 /srv/mediawiki/php-1.39.0-wmf.21/includes/MediaWiki.php(568): MediaWiki->main()
#7 /srv/mediawiki/php-1.39.0-wmf.21/index.php(50): MediaWiki->run()
#8 /srv/mediawiki/php-1.39.0-wmf.21/index.php(46): wfIndexMain()
#9 /srv/mediawiki/w/index.php(3): require(string)
#10 {main}
Impact
Notes

Event Timeline

kostajh added a subscriber: kostajh.

Happening for image recommendation tasks, 629 of these in the last 30 days.

Probably the first step is to improve the logging so we can see what the server response is.

Notable that these errors seem to have started with 1.39.0-wmf.19—I'm not seeing any errors of this kind prior to that version.

Change 818461 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/GrowthExperiments@master] ServiceImageRecommendationProvider: Add extra logging when no JSON response received

https://gerrit.wikimedia.org/r/818461

Happening for image recommendation tasks, 629 of these in the last 30 days.

Probably the first step is to improve the logging so we can see what the server response is.

742 errors from July01-Aug01/2022

Change 818461 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] ServiceImageRecommendationProvider: Add extra logging when no JSON response received

https://gerrit.wikimedia.org/r/818461

Change 819075 had a related patch set uploaded (by Urbanecm; author: Kosta Harlan):

[mediawiki/extensions/GrowthExperiments@wmf/1.39.0-wmf.22] ServiceImageRecommendationProvider: Add extra logging when no JSON response received

https://gerrit.wikimedia.org/r/819075

Change 819075 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@wmf/1.39.0-wmf.22] ServiceImageRecommendationProvider: Add extra logging when no JSON response received

https://gerrit.wikimedia.org/r/819075

Mentioned in SAL (#wikimedia-operations) [2022-08-03T15:36:22Z] <urbanecm@deploy1002> Synchronized php-1.39.0-wmf.22/extensions/GrowthExperiments/includes/NewcomerTasks/AddImage/ServiceImageRecommendationProvider.php: 4438957e78e0012aff646e52dc16a4fb796cfd6b: ServiceImageRecommendationProvider: Add extra logging when no JSON response received (T313973) (duration: 03m 04s)

[...]
Probably the first step is to improve the logging so we can see what the server response is.

I went ahead and backported the logging patch. Logs now should tell us what the server response is. And indeed they do:

2022-08-03 15:48:57 [fa724915-151c-4349-8d12-38d486d451fb] mw1434 ptwiki 1.39.0-wmf.22 GrowthExperiments ERROR: GrowthExperiments\NewcomerTasks\AddImage\ServiceImageRecommendationProvider::get: Unable to decode JSON response for page: WhatsUp_Gold upstream connect error or disconnect/reset before headers. reset reason: connection termination {"response":"upstream connect error or disconnect/reset before headers. reset reason: connection termination"}

Looks like an error on the service's end (or SRE's end).

Occurrences in the last 30 days:

Screenshot Capture - 2022-08-03 - 10-53-22.png (286×662 px, 20 KB)

Image recommendation stats in the last 90 days:
Screenshot Capture - 2022-08-03 - 10-51-38.png (807×1 px, 202 KB)

so this doesn't seem to have a visible impact on KPIs (but probably does result in some extra errors shown to users).

According to @hnowlan the error message is from Envoy - the image suggestions API is unavailable.

CBogen added a subscriber: CBogen.

Removing the Structured Data tag since this seems to be on the API Platform/Growth side only.

Change 822202 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[mediawiki/extensions/GrowthExperiments@master] Parametrize title when logging Add Image API errors

https://gerrit.wikimedia.org/r/822202

Change 822202 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] Parametrize title when logging Add Image API errors

https://gerrit.wikimedia.org/r/822202