A more useful test url is this; the problem is that whatever metadata that djvu file is trying to return is too large to be returned ever.

Addendum: actually I think it's impossible to continue the query at all, even if the bot understands the error. The only way to get around it is to remove the item from the query and to start over.

Is this specific to the imageinfo API or have you seen this behavior at other places?

It looks like it is not specific to the Imageinfo API. Some files seem to be affected on Commons. Example: https://commons.wikimedia.org/wiki/File:Dictionary_of_Greek_and_Roman_Geography_Volume_II.djvu

Have some memory limits in PHP been changed in the past few days on Wikimedia cluster? Has the version of DjVu libre been updated (maybe as part as the transition to Ubuntu 14.04)?

A similar issue has affected Wikimedia cluster two years ago when DjVu Libre as been updated as part of the migration to Ubuntu 12.04 (see this change: https://gerrit.wikimedia.org/r/#/c/36632 ).

In T86611#1036611, @Tpt wrote:

It looks like it is not specific to the Imageinfo API.

The problem raised here (metadata too large to fit in the API result) may be a different issue from whatever is going on there, even if it has the same trigger in the file having excessively-large data.

@Magog_the_Ogre: can you give more information on the impact of this bug? How often does it cause problems for botmasters? How hard is it to work around?

@Tgr,

I will try here.

For example, yesterday's run for my bot had a lot of djvu files with lots of metadata attached. I set my bot to ailimit=500 (in order to get around T92653). I still continually got this error message:

This result was truncated because it would otherwise be larger than the limit of 12582912 bytes

Unless there is a bug in my bot (admittedly possible), it seems that the API was truncating not just the metadata though, but the entire query. So I wasn't getting back other information like page categories, which led my bot to some strange bot behavior.

What behavior did you observe, specifically?

• Another infinite loop. That's this bug, no need to repeat.
• The query did not return the full limit's worth of results, but correctly continuing did return (or would have returned) the rest of the results without issue. That's expected behavior.
• The query did not return the full limit's worth of results, and continuing correctly did not return (or would not have returned) the rest of the results, e.g. some were (or would have been) skipped. That seems likely to be a different bug, and links to the specific queries would be very helpful here.
• Something else?

@Magog_the_Ogre: Could you reply to anomie's questions, please?

Anomie et al.: I'm very sorry. I've been sloppy and turned this bug report into a mess.

Any chance we can go back to looking at just the infinite loop?

@Anomie. It would be useful to get some progress on this bug. You mention above to @Tpt that this may or may not be the cause of his other concern.

At the Wikisources we are seeing a specific problem which can be seen at

https://en.wikisource.org/wiki/Index:EB1911_-_Volume_26.djvu

in that we pull the information for the number of pages in a djvu file, to help us to create the page index. <pagelist /> There is something failing in the output from the api where it presents something other than numbers and results in "Error: Numeric value expected".

A proper working example is

https://en.wikisource.org/wiki/Index:EB1911_-_Volume_25.djvu

This completely affects our work on transcriptions where the error exists to the point that a work pretty well ceases to progress. I would hope to see some movement here, and there is a clear bug here needing attention.

Thanks for whatever can be done.

This is still occurring, it seems. Sorry for being in confusing in my earlier messages.

https://commons.wikimedia.org/w/api.php?list=allimages&ailimit=max&action=query&continue=-||&aistart=20160211000000&aiend=20160211235959&aisort=timestamp&aidir=newer&aiprop=user|sha1|size|dimensions|mime|timestamp|comment|metadata&aicontinue=20160211235610|Boiste_-_Dictionnaire_universel,_1851.djvu

[list] => allimages
[ailimit] => max
[action] => query
[continue] => -||
[aistart] => 20160211000000
[aiend] => 20160211235959
[aisort] => timestamp
[aidir] => newer
[aiprop] => user|sha1|size|dimensions|mime|timestamp|comment|metadata
[aicontinue] => 20160211235610|Boiste_-_Dictionnaire_universel,_1851.djvu

{

"batchcomplete": "",
"continue": {
    "aicontinue": "20160211235610|Boiste_-_Dictionnaire_universel,_1851.djvu",
    "continue": "-||"
},
"warnings": {
    "result": {
        "*": "This result was truncated because it would otherwise  be larger than the limit of 12,582,912 bytes"
    }
},
"limits": {
    "allimages": 5000
},
"query": {
    "allimages": []
}

}

This is causing two of my bot's tasks to fail, as they can't pull in the recent upload data.

Is this 12MB+ of metadata in any way useful? The file page in Commons doesn't show any trace of it: https://commons.wikimedia.org/wiki/File:Boiste_-_Dictionnaire_universel,_1851.djvu Maybe its corruption either in the original file or when it's imported into Commons?

Running exiftool on the file gives:

ExifTool Version Number         : 10.00
File Name                       : Boiste_-_Dictionnaire_universel,_1851.djvu
Directory                       : .
File Size                       : 96 MB
File Modification Date/Time     : 2016:04:20 19:09:17+10:00
File Access Date/Time           : 2016:04:20 19:07:02+10:00
File Inode Change Date/Time     : 2016:04:20 19:09:17+10:00
File Permissions                : rw-rw-r--
File Type                       : DJVU (multi-page)
File Type Extension             : djvu
MIME Type                       : image/vnd.djvu
Subfile Type                    : Single-page image
Image Width                     : 2133
Image Height                    : 2845
DjVu Version                    : 0.24
Spatial Resolution              : 150
Gamma                           : 2.2
Orientation                     : Horizontal (normal)
Image Size                      : 2133x2845
Megapixels                      : 6.1

In T86611#2222110, @Ghouston wrote:

Is this 12MB+ of metadata in any way useful? The file page in Commons doesn't show any trace of it

I advise to read T101400 instead, which is way clearer.

It seems that the entire text of a document may be embedded in djvu or pdf metadata. Apart from the problem of requests failing due to the limit, it's wasteful to need to request so much data if you are interested in a few specific fields.

Here's a request for a pdf example that fits within the limit, but which is still megabytes of data (it doesn't crash my browser, but no guarantees):

https://commons.wikimedia.org/w/api.php?action=query&prop=imageinfo&iiprop=metadata&titles=File:Congressional%20Record%20Volume%2081%20Part%201.pdf

Requesting metadata simultaneously from the four similar files in the category https://commons.wikimedia.org/wiki/Category:United_States_Congressional_Record_Volume_81 does exceed the limit, which is a problem for bots that process multiple files for efficiency.

Maybe there are tweaks that could be made to the API to work around part of the problem in the short term, e.g., take a list of field names to be returned, or allow a maximum field data size to be specified. It wouldn't help if you actually wanted to get all the metadata for a file with more than 12MB of it.

I think my problems may go away if I request commonmetadata instead of metadata.

v4svpn

@V4switch this is not a security issue.

Getting this error in https://github.com/openzim/mwoffliner/issues/2401 trying to parse https://de.wikipedia.org/wiki/Liste_der_Schlangenarten using Parsoid:
https://de.wikipedia.org/w/api.php?action=parse&format=json&prop=text&parsoid=1&page=Liste_der_Schlangenarten

Meanwhile the same request using the legacy parser is working fine:
https://de.wikipedia.org/w/api.php?action=parse&format=json&prop=text&page=Liste_der_Schlangenarten

This indicates to me that this issue will get a lot more common when using action=parse with Parsoid due to all the metadata included as data attributes for template uses.

In T86611#1251373, @Magog_the_Ogre wrote:

Anomie et al.: I'm very sorry. I've been sloppy and turned this bug report into a mess.

Any chance we can go back to looking at just the infinite loop?

So, to be clear, the main issue here is that using the 'continue' response yields the same result next time?

@MSantos tagging and adding Content-Transform-Team due to the callout related to parsoid. This is a bigger issue, but it seems like y'all might want to be aware of the parsoid vs legacy parsing changes, as well.

Although there are action API endpoints that serve Parsoid HTML, Parsoid HTML is also served via a REST endpoint that does not apparently suffer from this problem.