[M] FormatMetadata: PHP Notice: Array to string conversion
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Assigned To

Authored By

	thcipriani
	Dec 9 2021, 4:14 PM

Description

Error

mwversion: 1.38.0-wmf.12
reqId: 26e624a1-c848-48fe-a70d-86a71eff22da
Find reqId in Logstash
Find normalized_message in Logstash

normalized_message

[{reqId}] {exception_url}   PHP Notice: Array to string conversion

exception.trace

from /srv/mediawiki/php-1.38.0-wmf.12/includes/media/FormatMetadata.php(1257)
#0 [internal function]: MWExceptionHandler::handleError(integer, string, string, integer, array)
#1 /srv/mediawiki/php-1.38.0-wmf.12/includes/media/FormatMetadata.php(1257): implode(string, array)
#2 /srv/mediawiki/php-1.38.0-wmf.12/includes/media/ExifBitmapHandler.php(73): FormatMetadata->flattenArrayReal(array, string, boolean)
#3 /srv/mediawiki/php-1.38.0-wmf.12/includes/filerepo/file/File.php(799): ExifBitmapHandler->convertMetadataVersion(array, integer)
#4 /srv/mediawiki/php-1.38.0-wmf.12/includes/api/ApiQueryImageInfo.php(606): File->convertMetadataVersion(array, string)
#5 /srv/mediawiki/php-1.38.0-wmf.12/includes/api/ApiQueryImageInfo.php(219): ApiQueryImageInfo::getInfo(LocalFile, array, ApiResult, NULL, array)
#6 /srv/mediawiki/php-1.38.0-wmf.12/includes/api/ApiQuery.php(629): ApiQueryImageInfo->execute()
#7 /srv/mediawiki/php-1.38.0-wmf.12/includes/api/ApiMain.php(1889): ApiQuery->execute()
#8 /srv/mediawiki/php-1.38.0-wmf.12/includes/api/ApiMain.php(868): ApiMain->executeAction()
#9 /srv/mediawiki/php-1.38.0-wmf.12/includes/api/ApiMain.php(839): ApiMain->executeActionWithErrorHandling()
#10 /srv/mediawiki/php-1.38.0-wmf.12/api.php(90): ApiMain->execute()
#11 /srv/mediawiki/php-1.38.0-wmf.12/api.php(45): wfApiMain()
#12 /srv/mediawiki/w/api.php(3): require(string)
#13 {main}

Impact

Notes

All happening on commons
All via the api
One big spike in the past 24 hours

Details

Request URL: https://commons.wikimedia.org/w/api.php?action=query&prop=*&coprop=*&coprimary=*&colimit=*&inprop=*&cllimit=*&iiprop=*&rvprop=*&format=*&rvslots=*&maxlag=*

	Subject	Repo	Branch	Lines +/-
	FormatMetadata: PHP Notice: Array to string conversion	mediawiki/core	master	+6 -0
	Filter out "LocationShown" from image metadata	mediawiki/core	master	+5 -0

Customize query in gerrit

Related Objects

Mentioned In: T307020: PHP Notice: Array to string conversion

Event Timeline

thcipriani created this task.Dec 9 2021, 4:14 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 9 2021, 4:14 PM

thcipriani moved this task from Untriaged to Dec 2021 on the Wikimedia-production-error board.Dec 9 2021, 4:15 PM

CBogen edited projects, added Structured-Data-Backlog (Current Work); removed Structured-Data-Backlog.Dec 13 2021, 5:38 PM

CBogen moved this task from Incoming to Ready for Estimation on the Structured-Data-Backlog (Current Work) board.

CBogen renamed this task from FormatMetadata: PHP Notice: Array to string conversion to [M] FormatMetadata: PHP Notice: Array to string conversion.Dec 15 2021, 5:24 PM

CBogen moved this task from Ready for Estimation to Ready for Development on the Structured-Data-Backlog (Current Work) board.Dec 15 2021, 5:37 PM

The error is present to several wikis other than commons wiki, e.g. wmf.12 lists dewiki, ruwiki, and meta.wikimedia - https://logstash.wikimedia.org/goto/ad8176e21fba1cd69a75f8f44b7a147c

SimoneThisDot claimed this task.Jan 20 2022, 9:12 AM

SimoneThisDot moved this task from Ready for Development to Doing on the Structured-Data-Backlog (Current Work) board.

@CBogen I have investigated the above issue and found it to be related to specific metadata that is not handled in the API.

The metadata is called "locationShown", and its value (expected to be a single array, is actually an array of arrays (image shown). The issue is actually in MediaWiki and not in the MediaSearch repository, so I am not sure if we are ok to continue working on it, or if we should delegate.

There are different solutions to the problem, and I would like to be suggested which one fit best in previous handling of this issues:

I can NOT show a metadata if it includes nested array information
I could parse this information to make a string of the Key value, it will create something like this:

LocationShown: GPSAltitude: 9579/100,
                           GPSLatitude: 53.106510321
                           GPSLongitude: 10.40

@MarkTraceur thoughts on the above?

I mean, my inclination is to always show information if we have it available. So if this is a relatively common case, let's parse the information. It sounds to me like this is a bug in how we consume an API more than anything.

@MarkTraceur The main worry I have is that the change is going to happen in MediaWiki, in a method that is used by more methods and extensions than just commons.

If I do make the change for this nested metadata to work, it will mean that ALL other extensions will also inherit this and I am not sure if their UI will accommodate the change, and even if it is something that they are expecting.

I have noticed that in a different part of the system "MediaHandler", SVG Handler, the metadata mentioned above (and more) are actually removed:

Oh, I understand now. In that case, yeah, let's mimic MediaHandler and toss it out for now. It would be nice to investigate briefly to find out why this field is not being handled correctly in some instances, however. Possibly something to check on now(ish), possibly something to just file a follow-up task about.

ok great.. I will move forward and mimic MediaHandler.
The reason for the miss-handling, is because this is the first "nested" array within the images metadata. So far all metadata were one level deep only, but with this metadata, information area actually nested array<Array>

SimoneThisDot changed the task status from Open to In Progress.Jan 31 2022, 11:52 AM

Change 758453 had a related patch set uploaded (by Simone Cuomo; author: Simone Cuomo):

[mediawiki/core@master] Filter out Arrays Of Array when formatting Images metadata

https://gerrit.wikimedia.org/r/758453

gerritbot added a project: Patch-For-Review.Jan 31 2022, 12:09 PM

SimoneThisDot moved this task from Doing to Code Review on the Structured-Data-Backlog (Current Work) board.Jan 31 2022, 12:09 PM

matthiasmullie moved this task from Code Review to Needs QA on the Structured-Data-Backlog (Current Work) board.Feb 28 2022, 1:05 PM

Change 758453 merged by jenkins-bot:

[mediawiki/core@master] Filter out \"LocationShown\" from image metadata

https://gerrit.wikimedia.org/r/758453

ReleaseTaggerBot added a project: MW-1.38-notes (1.38.0-wmf.24; 2022-02-28).Feb 28 2022, 2:00 PM

Maintenance_bot removed a project: Patch-For-Review.Feb 28 2022, 2:10 PM

Etonkovidova moved this task from Needs QA to Verify on Production on the Structured-Data-Backlog (Current Work) board.Mar 2 2022, 1:21 AM

Checked the normalized_message in Logstash - the error is not present in wmf.24.

Umherirrender mentioned this in T307020: PHP Notice: Array to string conversion.Apr 27 2022, 8:09 PM

Still happening in 1.39.0-wmf.9

I added this as a blocker for 1.39.0-wmf.10.

@SimoneThisDot Can you take another look at this? The "PHP Notice: Array to string conversion" from FormatMetadata.php has been persistent for the last week or so.

UBN! as a train blocker, moving back to incoming to ensure this is seen after this was re-opened.

Hi guys, I am investigating the issue now.
It seems like it is the same error, but a different cause (before it was the existence of a complex value LocationShown), but this does not seem to be the case in this case.

I will keep you updated as I find more

Thanks @SimoneThisDot !

Hi all , after some investigation and lots of help from @matthiasmullie, we have found out that this is not a train blocker as the issue was not due to any code changes but a single malformed Image.

Finding

The errors were actually triggered by an automatic service that was hitting a malformed image and triggering the errors above. The errors stopped this morning at 8 am

Further investigation showed a log for an image deletion with "bad metadata" https://commons.wikimedia.org/wiki/Commons:Deletion_requests/Files_uploaded_by_Rujee_Kitta.

1. Follow up actions
Make our metadata filtering more reliable and less prone to issues in metadata
Increase logging to provide more information when malformed metadata is provided

Summary

The issue should not be a trainblocker (as there has been no regression or nothing changed to prevent the release)
The issue will be worked on to make the application more reliable for future occurrences of this issue

cc: @Majavah @hashar

CBogen moved this task from Incoming to Doing on the Structured-Data-Backlog (Current Work) board.May 3 2022, 1:13 PM

dancy removed a parent task: T305216: 1.39.0-wmf.10 deployment blockers.May 3 2022, 2:54 PM

Awesome, thank you so much for the detailed summary of your investigations @SimoneThisDot and @matthiasmullie

Krinkle lowered the priority of this task from Unbreak Now! to High.May 9 2022, 7:58 PM

Seddon reassigned this task from SimoneThisDot to MarkThisDot.May 24 2022, 7:30 AM

Seddon added a subscriber: SimoneThisDot.

Change 803986 had a related patch set uploaded (by MarkThisDot; author: MarkThisDot):

[mediawiki/core@master] FormatMetadata: PHP Notice: Array to string conversion

https://gerrit.wikimedia.org/r/803986

gerritbot added a project: Patch-For-Review.Jun 8 2022, 8:49 PM

MarkThisDot moved this task from Doing to Code Review on the Structured-Data-Backlog (Current Work) board.Jun 10 2022, 9:56 PM

Umherirrender unsubscribed.Jun 19 2022, 7:33 PM

Change 803986 merged by jenkins-bot:

[mediawiki/core@master] FormatMetadata: PHP Notice: Array to string conversion

https://gerrit.wikimedia.org/r/803986

Maintenance_bot removed a project: Patch-For-Review.Jul 5 2022, 3:31 PM

dancy unsubscribed.Jul 5 2022, 3:31 PM

Seddon moved this task from Code Review to Needs QA on the Structured-Data-Backlog (Current Work) board.Jul 5 2022, 3:54 PM

Etonkovidova moved this task from Needs QA to Verify on Production on the Structured-Data-Backlog (Current Work) board.Jul 6 2022, 1:10 AM

Completed and handover to Simone

ReleaseTaggerBot added a project: MW-1.39-notes (1.39.0-wmf.21; 2022-07-18).Jul 12 2022, 2:01 AM

Checked wmf.21. The error (see below) was occurring in the beginning of May (as it was explained in https://phabricator.wikimedia.org/T297403#7899326):

normalized_message:[{reqId}] {exception_url} PHP Notice: Array to string conversion type:mediawiki http_method:POST monolog_level:400 referrer: - mwversion:1.39.0-wmf.9 caught_by:mwe_handler logsource:mw1428 url:/w/api.php exception_url:/w/api.php channel:error shard:s4 wiki:commonswiki servergroup:api_appserver

Screen Shot 2022-07-21 at 2.26.44 PM.png (634×2 px, 173 KB)

Two most recent errors (July 18/2022 and June 27/2022 (https://logstash.wikimedia.org/goto/1ff2ba1021f1c2bd5947b255930285a4) happened before wmf.21 and have a different source:

normalized_message:[{reqId}] {exception_url} PHP Notice: Array to string conversion reqId:55d2ff13e7924ab53ee98c2a monolog_level:400 @version:1 channel:error program:mediawiki phpversion:

	F35328933: Screen Shot 2022-07-21 at 2.26.44 PM.png
	Jul 21 2022, 9:36 PM

	F35096399: image.png
	May 3 2022, 1:00 PM

	F34930653: image.png
	Jan 25 2022, 9:31 PM

	F34925807: image.png
	Jan 21 2022, 2:24 PM

[M] FormatMetadata: PHP Notice: Array to string conversionClosed, ResolvedPublicPRODUCTION ERRORActions